Motivated by and grounded in observations of eye-gaze patterns in human-human dialogue, this study explores using eye-gaze patterns in managing human-computer dialogue. We developed an interactive system, iTourist, for city trip planning, which encapsulated knowledge of eye-gaze patterns gained from studies of human-human collaboration systems. User study results show that it was possible to sense users ’ interest based on eye-gaze patterns and manage computer information output accordingly. Study participants could successfully plan their trip with iTourist and positively rated their experience of using it. We demonstrate that eyegaze could play an important role in managing future multimodal human-computer dialogues.

Human perceptual and cognitive abilities are limited resources. Attention is the mechanism used to allocate such resources in the most effective way. Current technologies, in addition to allowing fast access to information and people, should be designed to support human attentional processes on which they impose further strain. This paper analyses the issues related to the design of systems capable of such support: Attention Aware Systems. We introduce the research aimed at understanding and modelling human attentional processes, including perceptual and cognitive processes as studied in cognitive psychology, as well as rhetorical, aesthetic, and social aspects related to attentional mechanisms. We analyse current approaches to the design of Attention Aware Systems along three major features: detection of user's current attentional state, detection and evaluation of possible alternative attentional states, strategies for focus switch or maintenance. Finally, we discuss the most promising research direction for the development of systems capable of supporting human attentional mechanisms.

We present EyePliances: appliances and devices that detect and respond to human visual attention using eye contact sensors. EyePliances receive implicit input [3] from users, in the form of eye gaze, and respond by opening communication channels. By allowing devices to recognize the attentional cues people already provide, requests for explicit input from users can be reduced. Further, eye contact sensing gives devices a mechanism to determine whether a user is available for interruption, and can provide the missing environmental context to improve speech recognition.

Abstract. The paper describes an infotainment application where lifelike characters present two MP3 players in a virtual showroom. The key feature of the system is that the presenter agents analyze the user’s gaze-behavior in real-time and may thus adapt the presentation flow accordingly. In particular, a user’s (non-)interest in interface objects and also preference in decision situations is estimated automatically by just using eye gaze as input modality. A formal study was conducted that compared two versions of the application. Results indicate that attentive presentation agents support successful grounding of deictic agent gestures and natural gaze behavior. 1

Abstract. We developed and studied an experimental system, RealTourist, which lets a user to plan a conference trip with the help of a remote tourist consultant who could view the tourist’s eye-gaze superimposed onto a shared map. Data collected from the experiment were analyzed in conjunction with literature review on speech and eye-gaze patterns. This inspective, exploratory research identified various functions of gaze-overlay on shared spatial material including: accurate and direct display of partner’s eye-gaze, implicit deictic referencing, interest detection, common focus and topic switching, increased redundancy and ambiguity reduction, and an increase of assurance, confidence, and understanding. This study serves two purposes. The first is to identify patterns that can serve as a basis for designing multimodal human-computer dialogue systems with eye-gaze locus as a contributing channel. The second is to investigate how computer-mediated communication can be supported by the display of the partner’s eye-gaze. 1

Abstract. This research proposes 3D graphical agents in the role of virtual presenters with a new type of functionality – the capability to process and respond to visual attention of users communicated by their eye movements. Eye gaze is an excellent clue to users ’ attention, visual interest, and visual preference. Using state-of-the-art non-contact eye tracking technology, eye movements can be assessed in a unobtrusive way. By analyzing and interpreting eye behavior in real-time, our proposed system can adapt to the current (visual) interest state of the user, and thus provide a more personalized, context-aware, and ‘attentive’ experience of the presentation. The system implements a virtual presentation room, where research content of our institute is presented by a team of two highly realistic 3D agents in a dynamic and interactive way. A small preliminary study was conducted to investigate users ’ gaze behavior with a non-interactive version of the system. A demo video based on our system was awarded as the best application of life-like agents at the GALA event in 2006. 1 1

In this paper, we report on an interactive system and the results ofa formal user study that was carried out with the aim of comparing two approaches to estimating users ’ interest in a multimodal presentation based on their eye gaze. The scenario consists of a virtual showroom where two 3D agents present product items in an entertaining way, and adapt their performance according to users ’ (in)attentiveness. In order to infer users ’ attention and visual interest with regard to interface objects, our system analyzes eye movements in real-time. Interest detection algorithms used in previous research determine an object of interest based on the time that eye gaze dwells on that object. However, this kind of algorithm does not seem to be well suited for dynamic presentations where the goal is to assess the user’s focus of attention with regard to a dynamically changing presentation. Here, the current context of the object of interest has to be considered, i. e., whether the visual object is part of (or contributes to) the current presentation content or not. Therefore, we propose to estimate the interest (or non-interest) of a user by means of dynamic Bayesian networks that may take into account the current context of the attention receiving object. In this way, the presentation agents can provide timely and appropriate response. The benefits of our approach will be demonstrated both theoretically and empirically.

This paper explores a novel approach to interactive user-guided image segmentation, using eyegaze information as an input. The method includes three steps: 1) eyegaze tracking for providing user input, such as setting object and background seed pixel selection; 2) an optimization method for image labeling that is constrained or affected by user input; and 3) linking the two previous steps via a graphical user interface for displaying the images and other controls to the user and for providing real-time visual feedback of eyegaze and seed locations, thus enabling the interactive segmentation procedure. We developed a new graphical user interface supported by an eyegaze tracking monitor to capture the user’s eyegaze movement and fixations (as opposed to traditional mouse moving and clicking). The user simply looks at different parts of the screen to select which image to segment, to perform foreground and background seed placement and to set optional segmentation parameters. There is an eyegaze-controlled “zoom ” feature for difficult images containing objects with narrow parts, holes or weak boundaries. The image is then segmented using the random walker image segmentation method. We performed a pilot study with 7 subjects who segmented synthetic, natural and real medical images. Our results show that getting used the new interface takes about only 5 minutes. Compared with traditional mouse-based control, the new eyegaze approach provided a 18.6 % speed improvement for more than 90 % of images with high object-background contrast. However, for low contrast and more difficult images it took longer to place seeds using the eyegaze-based “zoom ” to relax the required eyegaze accuracy of seed placement.

Eye contact in human conversations is a natural source of information about the visual attention of people talking, and also indicates who is speaking to whom. Eye gaze can be used as an interaction method, but gaze tracking can also be used to monitor a user’s eye movements and visual interest. This paper describes how gaze-based interaction can be used and implemented in an Augmented Reality (AR) system. The results of the preliminary tests of the gaze-controlled AR system show that the system does work, but that it needs considerable development and further user studies before it can be a realistic option in real end user settings.

We present EyePliances: appliances and devices that detect and respond to human visual attention using eye contact sensors. EyePliances receive implicit input [3] from users, in the form of eye gaze, and respond by opening communication channels. By allowing devices to recognize the attentional cues people already provide, requests for explicit input from users can be reduced. Further, eye contact sensing gives devices a mechanism to determine whether a user is available for interruption, and can provide the missing environmental context to improve speech recognition.