As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, evaluation methodology becomes important. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in ITS evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. By contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations, such as architecture and behaviour, others on external considerations, such as educational impact. This paper draws upon the areas of intelligent tutoring systems research, expert systems design, computer-based instruction, education, and psychology to identify techniques for the formative and summative evaluation of ITS. Evalu...

A tutoring approach is derived from a model of problem comprehension, based on the van Dijk and Kintsch (1983; Kintsch, 1988) theory of discourse processing. A problem statement is regarded as a text from which the student must glean propositional and situational information and make critical inferences. The student must coordinate this information with known problem schemaha so that formal (i.e. algebraic) operations can be applied and exact solutions obtained. We argue that this task is a highly reading oriented one where poor text comprehension and an inability to access relevant long-term knowledge lead to serious errors. Further, formal algebraic ekpressions are so abstract their meaning is often elusive; this contributes to mistranslations and misinterpretations. We describe experimental results with ANIMATE, an unintelligent tutoring system which knows nothing of the problem at hand or of the student's actions. Subjects who build animations of situations described in typical word problems consistently outperformed tutor users with no animation and students using only equations in both training tasks and near and distant transfer tasks. Performance differences were greatest for novel problems. We conclude that by providing an environment which gives equations situation-based meaning and makes the ramifications of students' formal manipulations clear, students learn to relate formal expressions to the referent situations. This enhances problem comprehension and gives a stronger representational base to the problem solving process.