This paper reports on a study whose goal was to investigate how people make use of gestures and spoken utterances while playing a videogame without the support of standard input devices. We deploy a Wizard of Oz technique...

We examined the use of hand gestures while people solved spatial reasoning problem in which they had to infer motion from static diagrams (mental animation problems). In Experiment 1, participants were asked to think aloud while solving mental animation problems. They gestured on more than 90 % of problems, and most gestures expressed information about the component motions that was not stated in words. Two further experiments examined whether the gestures functioned in the mechanical inference process, or whether they merely served functions of expressing or communicating the results of this process. In these experiments, we examined the effects of instructions to think aloud, restricting participants’ hand motions, and secondary tasks on mental animation performance. Although participants who were instructed to think aloud gestured more than control groups, some gestures occurred even in control conditions. A concurrent spatial tapping task impaired performance on mechanical reasoning, whereas a simple tapping task and restricting hand motions did not. These results indicate that gestures are a natural way of expressing the results of mental animation processes and suggest that spatial working memory and premotor representations are involved in mental animation. They provide no direct evidence that gestures are functional in the thought process itself, but do not rule out a role for overt gestures in this type of spatial thinking.

Cognitive research on metaphoric concepts of time has focused on differences between moving Ego and moving time models, but even more basic is the contrast between Ego- and temporal-reference-point models. Dynamic models appear to be quasi-universal cross-culturally, as does the generalization that in Ego-reference-point models, FUTURE IS IN FRONT OF EGO and PAST IS IN BACK OF EGO. The Aymara language instead has a major static model of time wherein FUTURE IS BEHIND EGO and PAST IS IN FRONT OF EGO; linguistic and gestural data give strong confirmation of this unusual culture-specific cognitive pattern. Gestural data provide crucial information unavailable to purely linguistic analysis, suggesting that when investigating conceptual systems both forms of expression should be analyzed complementarily. Important issues in embodied cognition are raised: how fully shared are bodily grounded motivations for universal cognitive patterns, what makes a rare pattern emerge, and what are the cultural entailments of such patterns?

Perhaps we keep finding iconicity because there is no other way for a semiotic system to be created and used by human beings without a close fit between form and function. After all, is it possible to make a mold for a statue that does not conform to the shape and dimensions and substance of the statue? Dan I. Slobin (2005, p. 321) In this paper I would like to reexamine some of the traditional dichotomies between language and gesture. In order to do so, it will be necessary to consider a three-way contrast—spoken languages, signed languages, and gesture. Without this three-way comparison, we risk collapsing contrasts between visual and auditory media with contrasts between linguistic structure and co-linguistic gestural structure. Such a comparison clearly belongs in this volume because Dan Slobin’s work on Thinking for Speaking has provided a crucial impetus to the research which feeds my new evaluation—both his own work on spoken and signed language, and the new perspectives on co-speech gesture which have been inspired by that work, not to mention his general intellectual influence on my work for the last 30 years. Dan has never been never afraid to cross boundaries between modalities—or to be skeptical about accepted dichotomies. So I hope readers will see this

Abstract. Referring to objects using multimodal deictic expressions is an important form of communication. This work addresses the question on how content is distributed between the modalities speech and gesture by comparing deictic pointing gestures to objects with and without speech. As a result, two main strategies used by participants to adapt their gestures to the condition were identified. This knowledge can be used, e.g., to improve the naturalness of pointing gestures employed by embodied conversational agents.