This project augments an existing intelligent tutoring system (AutoTutor) that helps learners construct explanations by interacting with them in natural language and helping them use simulation environments. The research aims to develop an agile learning environment that is sensitive to a learner’s affective state, presuming that this will promote learning. We integrate state-of-the-art, nonintrusive, affect-sensing technology with AutoTutor in an endeavor to classify emotions on the bases of facial expressions, gross body movements, and conversational cues. This paper sketches our broad theoretical approach, our methods for data collection and evaluation, and our emotion classification techniques.

Abstract. Time on task is an important predictor for how much students learn. However, students must be focused on their learning for the time invested to be productive. Unfortunately, students do not always try their hardest to solve problems presented by computer tutors. This paper explores student disengagement and proposes an approach, engagement tracing, for detecting whether a student is engaged in answering questions. This model is based on item response theory, and uses as input the difficulty of the question, how long the student took to respond, and whether the response was correct. From these data, the model determines the probability a student was actively engaged in trying to answer the question. The model has a reliability of 0.95, and its estimate of student engagement correlates at 0.25 with student gains on external tests. We demonstrate that simultaneously modeling student proficiency in the domain enables us to better model student engagement. Our model is sensitive enough to detect variations in student engagement within a single tutoring session. The novel aspect of this work is that it requires only data normally collected by a computer tutor, and the affective model is statistically validated against student performance on an external measure. 1.

Recent years have seen significant growth in work on interactive storytelling environments. A key challenge posed by these environments is narrative planning, in which a director agent orchestrates all of the events in a storyworld to create an optimal experience for a user, who is herself an active participant in the unfolding story. To create effective stories, the director agent must cope with the task’s inherent uncertainty, including uncertainty about the user’s intentions and the absence of a complete theory of narrative. Director agents must be efficient so they can operate in real time. In this paper, we present U-DIRECTOR, a decision-theoretic narrative planning architecture that dynamically models narrative objectives (e.g., plot progress, narrative flow), storyworld state (e.g., plot focus), and user state (e.g., goals, beliefs) with a dynamic decision network that continually selects storyworld actions to maximize narrative utility on an ongoing basis. The U-DIRECTOR architecture has been implemented in a narrative planner for Crystal Island, an interactive storyworld in which users play the role of a medical detective solving a science mystery. Preliminary evaluations suggest that the U-DIRECTOR architecture satisfies the real-time constraints of interactive environments and creates engaging narrative experiences. Categories and Subject Descriptors H.5.1 [Multimedia Information Systems]: Artificial, augmented, and virtual realities.

Abstract – Given the importance of implicit communication in human interactions, it would be valuable to have this capability in robotic systems wherein a robot can detect the motivations and emotions of the person it is working with. Recognizing affective states from physiological cues is an effective way of implementing implicit human-robot interaction. Several machine learning techniques have been successfully employed in affect-recognition to predict the affective state of an individual given a set of physiological features. However, a systematic comparison of the strengths and weaknesses of these methods has not yet been done. In this paper we present a comparative study of four machine learning methods- K-Nearest Neighbor, Regression Tree, Bayesian Network and Support Vector Machine as applied to the domain of affect recognition using physiological signals. The results showed that Support Vector Machine gave the best classification accuracy even though all the methods performed competitively. Regression Tree gave the next best classification accuracy and was the most space and time efficient.

This paper investigates the reliability of detecting a learner’s affective states in an attempt to augment an Intelligent Tutoring System (AutoTutor) with the ability to incorporate such states into its pedagogical strategies to improve learning. We describe two studies that used observational and emote-aloud protocols in order to identify the affective states that learners experience while interacting with AutoTutor. In a third study, training and validation data were collected from three sensors in a learning session with AutoTutor, after which the affective states of the learner were identified by the learner, a peer, and two trained judges. The third study assessed the reliability of automatic detection of boredom, confusion, delight, flow, and frustration (versus the neutral baseline) from sensors that monitored the manner in which learners communicate affect through conversational cues, gross body language, and facial expressions. Although the primary focus of this article is on the classification of learner affect, we also explore how an affect-sensitive AutoTutor can adapt its instructional strategies to promote learning. 1.

Spurred by a range of potential applications, there has been a growing body of research in computational models of human emotion. To advance the development of these models, it is critical that we evaluate them against the phenomena they purport to model. In this paper, we present one method to evaluate an emotion model that compares the behavior of the model against human behavior using a standard clinical instrument for assessing human emotion and coping. We use this method to evaluate the EMA model of emotion [1-3]. The evaluation highlights strengths of the approach and identifies where the model needs further development. 1.

Affective reasoning holds great potential for interactive digital entertainment, education, and training. Incorporating affective reasoning into the decision-making capabilities of interactive environments could enable them to create customized experiences that are dynamically tailored to individual users ’ ever changing levels of engagement, interest, and emotional state. Because physiological responses are directly triggered by changes in affect, biofeedback data such as heart rate and galvanic skin response can be used to infer affective changes. However, biofeedback hardware is intrusive and cumbersome in deployed applications. This paper proposes an inductive framework for automatically learning models of users’ physiological response from observations of user behaviors in interactive environments. These models can be used at runtime without biofeedback hardware to continuously predict users ’ physiological state directly from situational context in the interactive environment. Empirical studies with induced decision tree, naïve Bayes, and Bayesian Network physiological response models suggest that they may be sufficiently accurate for practical use.

In this paper, we report on our efforts in developing affective character-based interfaces, i.e. interfaces that recognize and measure affective information of the user and address user affect by employing embodied characters. In particular, we describe the Empathic Companion, an animated interface agent that accompanies the user in the setting of a virtual job interview. This interface application takes physiological data (skin conductance and electromyography) of a user in real-time, interprets them as emotions, and addresses the user's affective states in the form of empathic feedback. We present preliminary results from an exploratory study that aims to evaluate the impact of the Empathic Companion by measuring users' skin conductance and heart rate.

This paper describes an intelligent system that we developed to support affective multimodal human-computer interaction (AMM-HCI) where the user’s actions and emotions are modeled and then used to adapt the HCI and support the user in his or her activity. The proposed system, which we named Gaze-X, is based on sensing and interpretation of the human part of the computer’s context, known as W5+ (who, where, what, when, why, how). It integrates a number of natural human communicative modalities including speech, eye gaze direction, face and facial expression, and a number of standard HCI modalities like keystrokes, mouse movements, and active software identification, which, in turn, are fed into processes that provide decision making and adapt the HCI to support the user in his or her activity according to his or her preferences. To attain a system that can be educated, that can improve its knowledge and decision making through experience, we use case-based reasoning as the inference engine of Gaze-X. The utilized case base is a dynamic, incrementally self-organizing event-content-addressable memory that allows fact retrieval and evaluation of encountered events based upon the user preferences and the generalizations formed from prior input. To support concepts of concurrency, modularity/scalability, persistency, and mobility, Gaze-X has been built as an agent-based system where different agents are responsible for different parts of the processing. A usability study conducted in an office scenario with a number of users indicates that Gaze-X is perceived as effective, easy to use, useful, and affectively qualitative.

Abstract. We propose and evaluate a decision-theoretic approach for selecting tutorial actions by looking ahead to anticipate their effects on the student and other aspects of the tutorial state. The approach uses a dynamic decision network to consider the tutor's uncertain beliefs and objectives in adapting to and managing the changing tutorial state. Prototype action selection engines for diverse domains- calculus and elementary reading- illustrate the approach. These applications employ a rich model of the tutorial state, including attributes such as the student's knowledge, focus of attention, affective state, and next action(s), along with task progress and the discourse state. For this study, neither of our action selection engines had been integrated into a complete ITS, so we used simulated students to evaluate their capabilities to select rational tutorial actions that emulate the behaviors of human tutors. We also evaluated their capability to select tutorial actions quickly enough for real-world tutoring applications.