in human-robot interaction

Psycholinguistic studies of situated language processing have revealed that gaze in the visual environment is tightly coupled with both spoken language comprehension and production. It has also been established that interlocutors monitor the gaze of their partners, a phenomenon called "joint attention", as a further means for facilitating mutual understanding. We hypothesise that humanrobot interaction will benefit when the robot’s language-related gaze behaviour is similar to that of people, potentially providing the user with valuable non-verbal information concerning the robot’s intended message or the robot’s successful understanding. We report findings from two eye-tracking experiments demonstrating (1) that human gaze is modulated by both the robot speech and gaze, and (2) that human comprehension of robot speech is improved when the robot’s real-time gaze behaviour is similar to that of humans.

Recent studies show that both adults and young children possess powerful statistical learning capabilities to solve the wordto-world mapping problem. However, the underlying mechanisms that make statistical learning possible and powerful are not yet known. With the goal of providing new insights into this issue, the research reported in this paper used an eye tracker to record the moment-by-moment eye movement data of 14-month-old babies in statistical learning tasks. Various measures are applied to such fine-grained temporal data, such as looking duration and shift rate (the number of shifts in gaze from one visual object to the other) trial by trial, showing different eye movement patterns between strong and weak statistical learners. Moreover, an information-theoretic measure is developed and applied to gaze data to quantify the degree of learning uncertainty trial by trial. Next, a simple associative statistical learning model is applied to eye movement data and these simulation results are compared with empirical results from young children, showing strong correlations between these two. This suggests that an associative learning mechanism with selective attention can provide a cognitively plausible model of cross-situational statistical learning. The work represents the first steps in using eye movement data to infer underlying real-time processes in statistical word learning.

Abstract not found

Spoken dialogue is notoriously hard to process with standard NLP technologies. Natural spoken dialogue is replete with disfluent, partial, elided or ungrammatical utterances, all of which are very hard to accommodate in a dialogue system. Furthermore, speech recognition is known to be a highly error-prone task, especially for complex, open-ended discourse domains. The combination of these two problems – ill-formed and/or misrecognised speech inputs – raises a major challenge to the development of robust dialogue systems. We present an integrated approach for addressing these two issues, based on a incremental parser for Combinatory Categorial Grammar. The parser takes word lattices as input and is able to handle illformed and misrecognised utterances by selectively relaxing its set of grammatical rules. The choice of the most relevant interpretation is then realised via a discriminative model augmented with contextual information. The approach is fully implemented in a dialogue system for autonomous robots. Evaluation results on a Wizard of Oz test suite demonstrate very significant improvements in accuracy and robustness compared to the baseline. 1

The CRL Technical Report replaces the feature article previously published with every issue of the CRL Newsletter. The Newsletter is now limited to announcements and news concerning the CENTER FOR RESEARCH IN

This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier’s archiving and manuscript policies are encouraged to visit:

Abstract not found

Listeners can use speakers ’ gaze to anticipate upcoming referents. We examined whether this listener benefit is affected by different comprehension subtasks. A video-taped speaker referred to depicted characters, using either a subject-verb-object or a non-canonical object-verb-subject German sentence. She shifted gaze once from the pre-verbal to the post-verbal referent, a behavior that could allow listeners to anticipate which character would be mentioned next. We recorded participants’ eye movements to the characters during comprehension, as well as post-sentence verification times on whether a subsequent schematic depiction correctly highlighted the patient (Experiment 1) or the thematic role relations of the sentence (Experiment 2). Sentence structure affected response times only when verifying thematic roles. The eye movement data also showed reliable differences between tasks, regarding effects of sentence structure and their modulation by speaker gaze. We argue that processing accounts of situated comprehension must consider task effects on the allocation of visual attention.

Empirical evidence from studies using the visual world paradigm reveals that spoken language guides attention in a related visual scene and that scene information can influence the comprehension process. Here we model sentence comprehension using the visual context. A recurrent neural network is trained to associate the linguistic input with the visual scene and to produce the interpretation of the described event. The feedback mechanism in the form of sigma-pi connection is added to model the explicit utterance-mediated visual attention behavior revealed by the visual world paradigm. The results show that the network successfully learns sentence final interpretation and also demonstrates the hallmark anticipation behavior of predicting upcoming constituents.