Citation Context

... as a model. We argue that state-of-the art evaluation of adaptive search systems should follow a “layered evaluation” approach that is an active focus of research in the area of useradaptive systems =-=[2]-=-. The core idea behind layered evaluation is that specific sub-components or layers of any user-adaptive system should be understood and evaluated independently. Layered evaluation can overcome shortc...

Citation Context

...ses, an experiment is set up to evaluate an adaptive system by comparing it to a standard system. While recent work started to look at the systems’ performance and the validity of their design (e.g., =-=[3]-=- and [10]), we propose to look at yet another facet of a system-oriented evaluation, in particular the design of the underlying AH architecture with its impact on the maintenance of an application. V....

Citation Context

...e to collect implicit feedback; and vii) deciding how to filter the collected implicit feedback. 3. PROPOSED METHODOLOGY It is essential that the correct methods are used when evaluating AIRS systems =-=[6]-=-. In order to sufficiently evaluate both the adaptive functionality and the retrieval performance of these systems a hybrid approach is proposed which combines IR, AH and Simulation-based evaluation m...

Citation Context

...eep text comprehension [8, 15]. TERENCE [18] is a European ICT multidisciplinary project, for the technology enhanced learning (TEL) area, that is developing the first adaptive learning systems (ALS, =-=[6, 7]-=-) for improving the reading comprehension of 8–10 year old poor comprehenders. The learning material of TERENCE is made of stories and games, written in the two languages of the project, Italian and E...

Citation Context

...mprehenders, which could be delivered by an adaptive learning system (ALS), that is, a suite of functionalities designed to deliver, track, report on and manage learning content for specific learners =-=[8,9]-=-. TERENCE [19] is a EU project that aims at delivering the first ALS for enhancing the reading comprehension of poor comprehenders, building upon effective pencil-andpaper reading strategies, and fram...

Citation Context

...sm of rules over the fuzzy sets that are used to describe the student’s knowledge level of each domain concept of the knowledge domain, which we have described in previous work (Chrysafiadi & Virvou, 2010). So, the student modeling and the adaptation decision making incorporate fuzzy logic. For the evaluation of the effectiveness and efficiency of our student model and for the estimation of the success of adaptation, which is achieved via the system’s student model, we used a combination of two well-known and used models for evaluation: the Kirkpatrick, 1979 and the layered evaluation method (Brusilovsky, Karagiannidis, & Sampson, 2004). The Kirkpatrick’s model is ideal for measuring the effectiveness of training programs and the layered evaluation method is an approach of evaluating the success of adaptation that is offered by an adaptive learning system. The remainder of this paper is organized as follows. In Section 2, we present and discuss related work in the adaptive learning systems, student modeling and evaluation methods of adaptive systems and user modeling. In Section 3, we describe the student model of our web-based educational application and in Section 4 we show how it evolves by integrating fuzzy logic into i...

Citation Context

...ults. This decomposition is very close to the way that layered evaluation frameworks are decomposing AS in separate components that can be evaluated one by one. Pu et al. suggested that layered evaluation can be used in RecSys research as a powerful technique in identifying areas of a system that require further improvements. II. ADAPTING THE LAYERED EVALUATION FRAMEWORK To illustrate how a layered de-composition can serve as a starting point for the development of a more concrete and practical evaluation framework, we elaborate on the mapping of the layers proposed by Karagiannidis & Sampson [3] to the RecSys components proposed by Pu et al. [2], in order to provide some generic principles and guidelines that a RecSys researcher could use. The aim of this exercise is to demonstrate how such a generic framework can be built using the specific evaluation layers (user model and recommendation system), rather than proposing a complete and detailed generic framework. It will help us inform the analysis of a specific case study in the section that follows. More specifically, we focus on each layer and breakdown the interaction components to distinguishable elements – for example, separatin...