Tutorial: Evaluation of Adaptive Systems

Given by Stephan Weibelzahl, Alexandros Paramythis and Judith Masthoff

Adaptive systems are by definition interactive systems. As such they stand to benefit considerably from a development lifecycle that ensures user involvement from the early design stages, and embraces evaluation, in both formative and summative forms. Evaluating an adaptive system typically entails a number of specific problems and potential pitfalls that need to be addressed by the selection of specific evaluation approaches and instruments.

This tutorial aims to raise the awareness of participants regarding adequate evaluation methods and techniques, and reliable assessment criteria and metrics, in order to foster an increase in the quantity and quality of evaluation studies of adaptive systems, thus contributing to improvements in the quality and usability of the next generations of adaptive systems.

This tutorial will address the following learning objectives:

  • Awareness of the specific problems involved in the evaluation of adaptive systems that differentiate them from their non-adaptive counterparts, and of ways to solve or circumvent these problems
  • Understanding of, and capacity to apply, the principles of layered evaluation of adaptive systems
  • Acquisition of the prerequisite knowledge for designing a targeted evaluation study for an adaptive system (e.g., addressing a given layer and set of criteria) by selecting appropriate methods and criteria