Intelligent tutoring systems are highly interactive learning environments that have been shown to improve upon typical classroom instruction. Cognitive Tutors are a type of intelligent tutor based on cognitive psychology theory of problem solving and learning. Cognitive Tutors provide a rich problem-solving environment with tutorial guidance in the form of step-by-step feedback, specific messages in response to common errors, and on-demand instructional hints. They also select problems based on individual student performance. The learning benefits of these forms of interactivity are supported, to varying extents, by a growing number of results from experimental studies. As Cognitive Tutors have matured and are being applied in new subject-matter areas, they have been used as a research platform and, particularly, to explore interactive methods to support metacognition. We review experiments with Cognitive Tutors that have compared different forms of interactivity and we reinterpret their results as partial answers to the general question: How should learning environments balance information or assistance giving and withholding to achieve optimal student learning? How best to achieve this balance remains a fundamental open problem in instructional science. We call this problem the “assistance dilemma ” and emphasize the need for further science to yield specific conditions and parameters that indicate when and to what extent to use information giving versus information withholding forms of interaction.

Abstract. Modeling students ’ knowledge is a fundamental part of intelligent tutoring systems. One of the most popular methods for estimating students ’ knowledge is Corbett and Anderson’s [6] Bayesian Knowledge Tracing model. The model uses four parameters per skill, fit using student performance data, to relate performance to learning. Beck [1] showed that existing methods for determining these parameters are prone to the Identifiability Problem: the same performance data can be fit equally well by different parameters, with different implications on system behavior. Beck offered a solution based on Dirichlet Priors [1], but, we show this solution is vulnerable to a different problem, Model Degeneracy, where parameter values violate the model’s conceptual meaning (such as a student being more likely to get a correct answer if he/she does not know a skill than if he/she does). We offer a new method for instantiating Bayesian Knowledge Tracing, using machine learning to make contextual estimations of the probability that a student has guessed or slipped. This method is no more prone to problems with Identifiability than Beck’s solution, has less Model Degeneracy than competing approaches, and fits student performance data better than prior methods. Thus, it allows for more accurate and reliable student modeling in ITSs that use knowledge tracing. 1

Phishing attacks, in which criminals lure Internet users to websites that spoof legitimate websites, are occurring with increasing frequency and are causing considerable harm to victims. While a great deal of effort has been devoted to solving the phishing problem by prevention and detection of phishing emails and phishing websites, little research has been done in the area of training users to recognize those attacks. Our research focuses on educating users about phishing and helping them make better trust decisions. We identified a number of challenges for end-user security education in general and anti-phishing education in particular: users are not motivated to learn about security; for most users, security is a secondary task; it is difficult to teach people to identify security threats without also increasing their tendency to misjudge non-threats as threats. Keeping these challenges in mind, we developed an email-based anti-phishing education system called “PhishGuru ” and an online game called “Anti-Phishing Phil ” that teaches users how to use cues in URLs to avoid falling for phishing attacks. We applied learning science instructional principles in the design of PhishGuru and Anti-Phishing Phil. In this paper we present the results of PhishGuru and Anti-Phishing Phil user studies that demonstrate the effectiveness of these tools. Our results suggest that, while automated detection systems should be used as the first line of defense against phishing attacks, user education offers a complementary approach to help people better recognize fraudulent emails and websites.

The movement towards high stakes testing promises to encourage rigor and accountability in middle school mathematics, but there is a danger that a toonarrow focus on testing will take time and attention away from mathematics instruction. The fundamental dilemma that teachers face in trying to use assessment to guide instruction (i.e., to figure out what is the next best thing for a student to try to learn) is that because assessment takes time away from instruction, how can teachers be sure the time spent assessing will improve instruction enough to justify the cost of lost instructional time. We propose to address this dilemma by building and experimentally evaluating the effectiveness of a web-based "Assistment " system for middle school mathematics. This system will 1) quickly predict a student’s score on a standard-based test, 2) provide feedback to teachers about how they can specifically adapt their instruction to address student knowledge gaps, and 3) unlike other assessments system, provide an opportunity for students to get intelligent tutoring assistance at they same time

The ASSISTment system is an online benchmark testing system that tutors as it tests. The system has been implemented for the content of the 8 th grade Mathematics portion of the Massachusetts Comprehensive Assessment System (MCAS) exams, has been developed and tested in Massachusetts middle schools, and is being adapted for use in other states such as Pennsylvania. Two main statistical goals for the ASSISTment system are to predict end-of-year MCAS scores, and to provide regular, periodic feedback to teachers on how students are doing, what to teach next, etc. In this chapter we focus on the first goal and consider 10 prediction models: how they reflect different models for student proficiency, how they account for student learning over time, and how well they predict MCAS scores. We conclude that a combination of measures, including response accuracy (right/wrong) measures that account for problem difficulty, response efficiency, and help-seeking behavior, produce the best prediction models. In addition, our investigations of prediction models reveal patterns of learning over time that should be captured in feedback reports for teachers.

Latent Response Models, intelligent agents 2 Students use intelligent tutors and other types of interactive learning environments in a considerable variety of ways. In this thesis, I detail my work to understand, automatically detect, and re-design an intelligent tutoring system to adapt to a behavior I term “gaming the system”. Students who game the system attempt to succeed in the learning environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. Within this thesis, I present a set of studies aimed towards understanding what effects gaming has on learning, and why students game, using a combination of quantitative classroom observations and machine learning. In the course of these studies, I determine that gaming the system is replicably associated with low learning. I use data from these studies to develop a profile of students who game, showing that gaming students have a consistent pattern of negative affect

Abstract. Some students, when working in interactive learning environments, attempt to “game the system”, attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector’s generalizability, and find that it transfers successfully to both new students and new tutor lessons.

Abstract. We present an approach to designing intelligent tutoring systems, termed the Difficulty Factors Approach. In this approach, the designer investigates, at each iteration of the design cycle, which skills and concepts are difficult for students, and what factors underlie those difficulties. We show that this approach complements existing design principles, producing data that helps designers apply principles in context. We also show that by continuing to investigate student difficulties throughout the design process, it is possible to discover difficulty factors initially obscured by other difficulty factors. We give an example of the application of the Difficulty Factors Approach in the context of the development of a cognitive tutor lesson on scatterplots.

Most of the prior descriptions of the important relationships in Intelligent Tutoring System (ITS) projects have focused on the relationships involved in their use in classrooms, treating their presence in the classroom as a given. There has been some discussion of how intelligent tutors are developed [13] and of how an Intelligent Tutor, once developed, can be disseminated widely [5], but there has been considerably less discussion of the deployment of prototype ITSs. In this paper, we present a model of the relationships involved in deploying a prototype intelligent tutoring system in order to conduct formative evaluation. We show that field technical personnel play a pivotal role in this process, serving as vital conduits for information and negotiation between ITS researchers and school personnel such as teachers and principals. This model was developed using Contextual Inquiries [4] and interviews of project members.

Abstract Intelligent tutoring systems are highly interactive learning environments that have been shown to improve upon typical classroom instruction. Cognitive Tutors are a type of intelligent tutor based on cognitive psychology theory of problem solving and learning. Cognitive Tutors provide a rich problem-solving environment with tutorial guidance in the form of step-by-step feedback, specific messages in response to common errors, and ondemand instructional hints. They also select problems based on individual student performance. The learning benefits of these forms of interactivity are supported, to varying extents, by a growing number of results from experimental studies. As Cognitive Tutors have matured and are being applied in new subject-matter areas, they have been used as a research platform and, particularly, to explore interactive methods to support metacognition. We review experiments with Cognitive Tutors that have compared different forms of interactivity and we reinterpret their results as partial answers to the general question: How should learning environments balance information or assistance giving and withholding to achieve optimal student learning? How best to achieve this balance remains a fundamental open problem in instructional science. We call this problem the “assistance dilemma ” and emphasize the need for further