The Centre for Next Generation Localisation (CNGL) is involved in building interactive adaptive systems which combine Information Retrieval (IR), Adaptive Hypermedia (AH) and adaptive web techniques and technologies. The complex functionality of these systems coupled with the variety of potential users means that the experiments necessary to evaluate such systems are difficult to plan, implement and execute. This evaluation requires both component-level scientific evaluation and userbased evaluation. Automated replication of experiments and simulation of user interaction would be hugely beneficial in the evaluation of adaptive information retrieval systems (AIRS). This paper proposes a methodology for the evaluation of AIRS which leverages simulated interaction. The hybrid approach detailed combines: (i) user-centred methods for simulating interaction and personalisation; (ii) evaluation metrics that combine Human Computer Interaction (HCI), AH and IR techniques; and (iii) the use of qualitative and quantitative evaluations. The benefits and limitations of evaluations based on user simulations are also discussed.

Abstract. As applications evolve over time, it becomes increasingly desirable to be able to adapt a system, enabling it to handle situations in different ways and to handle new situations. We refer to this as the flexibility and maintainability of a system. These features come at a cost. We argue here that they are an important aspect of evaluation, and that we need to measure these costs. To start getting a handle on how one might evaluate these aspects of a system (or of an approach), we turned to our own approach to building AH applications and designed a specific study to allow us to look at these issues. 1

Abstract. Existing adaptive learning systems use various user characteristics for preparing personalized presentation of educational material for each user (a student). In the process of learning limited capacity of the human memory plays important role. The student often forgets part of knowledge acquired during the learning. In this paper we discuss implications of remembering and forgetting for the adaptive learning systems. We present a proposal of modeling forgetting process in adaptive learning systems that enables including knowledge repetition into an educational course. We implemented proposed model within the web-based adaptive system AHA! and demonstrate its viability on domain of learning English-Slovak vocabulary. This includes also a technique for inserting concepts for knowledge repeating into a running direct guided course. Using this technique it is possible to dynamically adapt the sequence of presented concepts to the actual state of the user model. 1

Abstract: Web crawlers are a significant component in Web search engines. This paper describes a distributed, Web-based system consisting of 3 layers: presentation, business logic and data persistence. The educational benefit of this system is in the provision of an interactive reference tool to encourage the evaluation of systems that fulfill certain methodological requirements. The studies collected can be used as a basis of a searchable online database that provides an overview of the state-of-the-art to a scientific community and encourages scientists to evaluate their own systems. The system will also help students identify pitfalls in the planning process as well as in data analysis. The collaborative nature of the tool facilitates the sharing of information among research students. The architecture includes RSS Feed Management, Paper Subscription, SMART URL analysis and Document Downloading. The system has been tested and evaluated showing good performance, the results are presented below.

Abstract. Adaptive Hypermedia (AH) research is concerned with the dynamic composition and personalisation of hypermedia documents in order to provide more context sensitive retrieval and reuse of digital content. The evaluation of Adaptive Hypermedia Systems (AHS) is difficult due to the complexity of such systems [1]. Several Problems and pitfalls are encountered when evaluating these systems [2-6]. Very little research has been carried out to address this problem. This PhD work proposes a user-centred approach to the evaluation of the adaptive mechanism of AHS. The proposed approach will be validated using personalised systems developed by the Centre for Next Generation Localisation (CNGL). The framework developed by this research will help to standardise current approaches, offer hints regarding the identification of failures and misconceptions of the adaptive mechanism. It will be applicable to all adaptive systems with no limitations of domain or inference mechanism. A review of approaches, methodologies and techniques adopted by existing systems was conducted and the results analysed. An architectural design of the framework has been designed and currently implementation work is going on.