As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, evaluation methodology becomes important. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in ITS evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. By contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations, such as architecture and behaviour, others on external considerations, such as educational impact. This paper draws upon the areas of intelligent tutoring systems research, expert systems design, computer-based instruction, education, and psychology to identify techniques for the formative and summative evaluation of ITS. Evalu...

Evaluation of intelligent tutoring systems (ITS) is an important area of research in current educational practices. There are many evaluation methods available but the literature does not suggest any clear guidelines for an evaluator – normally an educator – which methods to use in particular contexts. This paper proposes a classification of evaluation methods to simplify the selection task. The classification is based on two primary questions relating to the target of evaluation and learning environment in which the evaluation would be pursued. The classification is hoped to help in improving quality of computer based education by providing a practical and to the point way of selecting the appropriate evaluation methods for intelligent tutoring systems.

Cognitive support in knowledge engineering is a growing concern, and information visualization is a useful means to address this. We identify some requirements for tools offering cognitive support, and present a tool, Jambalaya, which addresses some of these concerns. We identify some of its features and describe areas we are actively improving. 1.

The validation of a Knowledge Based System (KBS) involves comparisons between an external reference model and a system's component parts. In this paper I describe how such comparisons can be aided by the application of software visualization technology. Software visualization is the use of filmcraft, cartoon animation and graphic design techniques to display data structures, programs, and algorithms. The described approach eases the task of mapping between the comparates by the use of dynamic code, design, and domain oriented visualizations of KBS execution. 1. INTRODUCTION Software Visualization (SV) [Price et al., 1993] is the use of filmcraft, cartoon animation and graphic design techniques to display data structures, programs, and algorithms. Software visualization is basically the unification of algorithm animation and program visualization, two concurrent developments from the 1980s. Algorithm animations are high level characterisations of how data is manipulated during a progra...

VIM demonstrates a 3D interactive diagram serving as a radically co-operative user interface to an Underlying Knowledge Base System (UKBS). VIM comprises a Prolog client linked by the Internet to a diagram graphics server linked by X/Motif to conventional 2D display hardware. The diagram integrates three concurrent layers of mixed-initiative, direct-manipulation interaction: (i) User-UKBS dialogue controlling beliefs presented in the diagram; (ii) Shared user-UKBS `browsing', controlling selectively emphasised selective viewing of a potentially much larger belief structure; (iii) Novel co-operative force-directed management of diagram layout. VIM's 3D diagrams have a constraining 3D spatial arrangement scheme and a fixed principal view. Subjectively this arrangement aids expressiveness, comprehensibility and perceptual toleration of both link crossings and topological change in the 2D display, and inhibits browsing disorientation. We argue that this style of interface offers fundamental benefits to knowledge engineers and end users.

The initial six months of this research grant consisted of two major efforts: extension of existing work on OFMspert, and initiation of research on tutoring principles and models for complex, dynamic system, in particular, applications in Goddard Space Flight Center (GSFC) ground control systems. This research builds on the work done

Abstract not found

This paper presents the Transparent Rule Interpreter (TRI) a system that provides a graphical explanation of rule execution. TRI is one part of KEATS-II (Motta et. al., 1988; in press), a project with the overall aim of providing methodological and software support for all the stages of building very large, Knowledge-Based systems. Such support includes the provision of a suite of tools for maintaining knowledge bases, collectively known as Knowledge Base Maintenance Tools (KBMTs). In a host of different applications (Eisenstadt & Brayshaw, 1988; Rajan, 1986; du Boulay, O'Shea & Monk, 1981), the provision of a clear execution model has proved to be an essential tool for understanding a computational process; as such, TRI presents a clear graphical execution model to the user. TRI, implemented in Symbolics^TM Common Lisp, has been built on top of the KEATS-II forward and backward chaining production system interpreter. The forward chaining part uses a Rete network (Forgy, 1982), and the backward chaining part has a Prolog like inference mechanism. The rest of this paper is structured as follows. Section 2 discusses the motivation for TRI, that is the problems found when programming with rules, previous attempts at solving the problems and our approach. Section 3 gives a brief 'tour' of the system using an example, section 4 reviews the principles behind TRI and its extendability, and section 5 contains the conclusions.

oring systems and provided many ideas for the development of Canah-chab. Dr. Mark Guzdial provided interesting advices on the use of visualization tools to improve the interface of Canah-chab. Drs. Goel, Govindaraj, Recker and Guzdial provided challenging critiques on earlier drafts of this thesis. Their constructive criticisms helped to make explicit the contributions and limitations of this research. They also provided detailed comments that helped to significantly improve the presentation of this thesis. Drs James Foley and John Stasko provided feedback on Canah-chab's interface. The research reported in this thesis has immensely benefited from input by several people as well. Mr. Andr'es G'omez de Silva Garza played an important role in the implementation of Canah-chab. His contributions have been especially significant in the development of the illustrations of the design processes. Mr. Michael Donahoo iv also helped w

understanding the needs of cognitive support