This paper investigates the reliability of detecting a learner’s affective states in an attempt to augment an Intelligent Tutoring System (AutoTutor) with the ability to incorporate such states into its pedagogical strategies to improve learning. We describe two studies that used observational and emote-aloud protocols in order to identify the affective states that learners experience while interacting with AutoTutor. In a third study, training and validation data were collected from three sensors in a learning session with AutoTutor, after which the affective states of the learner were identified by the learner, a peer, and two trained judges. The third study assessed the reliability of automatic detection of boredom, confusion, delight, flow, and frustration (versus the neutral baseline) from sensors that monitored the manner in which learners communicate affect through conversational cues, gross body language, and facial expressions. Although the primary focus of this article is on the classification of learner affect, we also explore how an affect-sensitive AutoTutor can adapt its instructional strategies to promote learning. 1.

In this paper a study on multimodal automatic emotion recognition during a speech-based interaction is presented. A database was constructed consisting of people pronouncing a sentence in a scenario where they interacted with an agent using speech. Ten people pronounced a sentence corresponding to a command while making 8 different emotional expressions. Gender was equally represented, with speakers of several different native languages including French, German, Greek and Italian. Facial expression, gesture and acoustic analysis of speech were used to extract features relevant to emotion. For the automatic classification of unimodal data, bimodal data and multimodal data, a system based on a Bayesian classifier was used. After performing an automatic classification of each modality, the different modalities were combined using a multimodal approach. Fusion of the modalities at the feature level (before running the classifier) and at the results level (combining results from classifier from each modality) were compared. Fusing the multimodal data resulted in a large increase in the recognition rates in comparison to the unimodal systems: the multimodal approach increased the recognition rate by more than

Abstract. This paper presents some of the requirements for the design of long-term affect sensitive and socially interactive companions. Research conducted in the EU project LIREC (LIving with Robots and intEractive Companions) envisage some capabilities including affect sensitivity, memory and learning, cognitive and expressive behaviour, personalisation and embodiment, highlighting the key issues that research on artificial companions should address. 1 1

and speech

emotion recognition from expressive faces,

Affect sensitivity is an important requirement for artificial companions to be capable of engaging in social interaction with human users. This paper provides a general overview of some of the issues arising from the design of an affect recognition framework for artificial companions. Limitations and challenges are discussed with respect to other capabilities of companions and real world scenarios for affect sensitive human-companion interaction.

Investigating shared attention with a virtual agent using a gaze-based interface

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

student affect

Abstract. We present a new paradigm for the design of interactive robots called affective-centered design. By drawing on the disciplines of human-computer interaction (HCI), affective computing, and human-robot interaction (HRI), we suggest techniques robot designers can use to help ensure interactions with their robots are of high affective quality, and thus more likely to be enjoyed and accepted by users.