We report a new phenomenon associated with language comprehension: the action–sentence compatibility effect (ACE). Participants judged whether sentences were sensible by making a response that required moving toward or away from their bodies. When a sentence implied action in one direction (e.g., “Close the drawer ” implies action away from the body), the participants had difficulty making a sensibility judgment requiring a response in the opposite direction. The ACE was demonstrated for three sentences types: imperative sentences, sentences describing the transfer of concrete objects, and sentences describing the transfer of abstract entities, such as “Liz told you the story. ” These data are inconsistent with theories of language comprehension in which meaning is represented as a set of relations among nodes. Instead, the data support an embodied theory of meaning that relates the meaning of sentences to human action. How language conveys meaning remains an open question. The dominant approach is to treat language as a symbol manipulation system: Language conveys meaning by using abstract, amodal, and arbitrary symbols (i.e., words) combined by syntactic rules (e.g., Burgess & Lund, 1997; Chomsky, 1980; Fodor, 2000; Kintsch, 1988; Pinker, 1994). Words are abstract in that the same word, such as “chair, ” is used for big chairs and little chairs, words are amodal in that the same word is used when chairs are spoken about or written about, and words are arbitrarily related to their referents in that the phonemic and orthographic characteristics of a word bear no relationship to the physical or functional characteristics of the word’s referent. An alternative view is that linguistic meaning is

Abstract not found

In face-to-face interaction, multimodal signals are at work. We communicate not only through words, but also by intonation, body posture, hand gestures, gaze patterns, facial expressions, and so on. All these signals, verbal and nonverbal, do have a role in the communicative process. They add/modify/substitute information in discourse and are highly linked with one another. This is why facial and bodily animation is

Abstract — Why is it that people cannot keep their hands still when they talk? One reason may be that gesturing actually lightens cognitive load while a person is thinking of what to say. We asked adults and children to remember a list of letters or words while explaining how they solved a math problem. Both groups remembered significantly more items when they gestured during their math explanations than when they did not gesture. Gesturing appeared to save the speakers’ cognitive resources on the explanation task, permitting the speakers to allocate more resources to the memory task. It is widely accepted that gesturing reflects a speaker’s cognitive state, but our observations suggest that, by reducing cognitive load, gesturing may also play a role in shaping that state. Gesturing occurs across ages, tasks, and cultures (Feyereisen & de Lannoy, 1991). Although in theory gesture could be nothing more than meaningless hand waving, recent research has found that gesturing conveys meaningful information (Clark, 1996; Goldin-Meadow, Mc-Neill, & Singleton, 1996; Kendon, 1980; McNeill, 1992), information that is not always found in the speech it accompanies (Goldin-Meadow, Alibali, & Church, 1993). For example, a speaker might say, “I ran all the way upstairs ” while moving her index finger upward in a spiral. It is through the speaker’s gestures, and only her gestures, that the listener knows the staircase is a spiral. Moreover, gesture is noticed. The information that gesture conveys frequently has an impact on the message listeners take from the communication (Alibali, Flevares,

Although speech and gesture recognition has been studied extensively all the successful attempts of combining them in the unified framework were semantically motivated, e.g., keyword co-occurrence. Such formulations inherited the complexity of natural language processing. This paper presents a statistical approach that uses physiological phenomenon of gesture and speech production process for improving accuracy of automatic segmentation of continuous deictic gestures. The prosodic features from the speech signal were coanalyzed with the visual signal to create a statistical model of co-occurrence with particular kinematical phases of gestures. Results indicated that the above co-analysis improves continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected from the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.

Computational approaches towards gesture recognition and synthesis tend to regard gestures as isolated carriers of propositional content. Thus far, the imagistic role of gestures, particularly their cohesive use of space, has been barely considered in HCI, despite its significance discussed in psycholinguistic literature (McNeill, 1992). We present a representation and processing model for the understanding and synthesis of this aspect in a shape description domain. Representation of Gestural Imagery An empirical study on object descriptions unveiled that gestures often convey an underspecified abstraction of shape reducing 3D objects to projections of lower dimensions (Sowa & Wachsmuth, 2002). The most frequently observed shape properties were extents or diameters indicated

In this paper, I will give an overview of some well-studied multimodal signals that humans produce while they communicate with other humans, and discuss the implications of those studies for HCI. I will first discuss a conceptual framework that allows us to distinguish between functional and sensory modalities. This distinction is important, as there are multiple functional modalities using the same sensory modality (e.g., facial expression and eye-gaze in the visual modality). A second theoretically important issue is redundancy. Some signals appear to be redundant with a signal in another modality, whereas others give new information or even appear to give conflicting information (see e.g., the work of Susan Goldin-Meadows on speech accompanying gestures). I will argue that multimodal signals are never truly redundant. First, many gestures that appear at first sight to express the same meaning as the accompanying speech generally provide extra (analog) information about manner, path, etc. Second, the simple fact that the same information is expressed in more than one modality is itself a communicative signal. Armed with this conceptual background, I will then proceed to give an overview of some multimodal signals that have been investigated in human-human research, and the level of understanding we have of

The development of perceptual user interfaces is central to Oxygen’s mission of pervasive, human-centered computing. Based on speech, gesture, and sketching – rather than a keyboard and mouse – these new forms of interaction

Abstract. Deictic gestures are ubiquitous when people work over tables and whiteboards, but when collaboration occurs across distributed surfaces, the embodiments used to represent other members of the group often fail to convey the details of these gestures. Although both gestures and embodiments have been well studied, there is still little information available to groupware designers about what components and characteristics of deictic gesture are most important for conveying meaning through remote embodiments. To provide this information, we conducted three observational studies in which we recorded and analysed more than 450 deictic gestures. We considered four issues that are important for the design of embodiments on surfaces: what parts of the body are used to produce a deictic gesture, what atomic movements make up deixis, where gestures occur in the space above the surface, and what other characteristics deictic gestures exhibit in addition to pointing. Our observations provide a new design understanding of deictic gestures. We use our results to identify the limitations of current embodiment techniques in supporting deixis, and to propose new hybrid designs that can better represent the range of behavior seen in real-world settings.

speakers do and what listeners look at.