A theoretical framework for grounding language is introduced that provides a computational path from sensing and motor action to words and speech acts. The approach combines concepts from semiotics and schema theory to develop a holistic approach to linguistic meaning. Schemas serve as structured beliefs that are grounded in an agent’s physical environment through a causal-predictive cycle of action and perception. Words and basic speech acts are interpreted in terms of grounded schemas. The framework reflects lessons learned from implementations of several language processing robots. It provides a basis for the analysis and design of situated, multimodal communication systems that straddle symbolic and non-symbolic realms.

Three theories currently compete to explain the conceptual deficits that result from brain damage: sensory-functional theory, domain-specific theory, and conceptual structure theory. We argue that all three theories capture important aspects of conceptual deficits, and offer different insights into their origins. Conceptual topography theory (CTT) integrates these insights, beginning with A. R. Damasio’s (1989) convergence zone theory and elaborating it with the similarity-in-topography (SIT) principle. According to CTT, feature maps in sensory-motor systems represent the features of a category’s exemplars. A hierarchical system of convergence zones then conjoins these features to form both property and category representations. According to the SIT principle, the proximity of two conjunctive neurons in a convergence zone increases with the similarity of the features they conjoin. As a result, conjunctive neurons become topographically organised into local regions that represent properties and categories. Depending on the level and location of a lesion in this system, a wide variety of deficits is possible. Consistent with the literature, these deficits range from the loss of a single category to the loss of multiple categories that share sensory-motor properties.

Background noise is the irregular variation across repeated measurements of human performance. Background noise remains after task and treatment effects are minimized. Background noise refers to intrinsic sources of variability, the intrinsic dynamics of mind and body, and the internal workings of a living being. Two experiments demonstrate 1/f scaling (pink noise) in simple reaction times and speeded word naming times, which round out a catalog of laboratory task demonstrations that background noise is pink noise. Ubiquitous pink noise suggests processes of mind and body that change each other’s dynamics. Such interaction-dominant dynamics are found in systems that self-organize their behavior. Self-organization provides an unconventional perspective on cognition, but this perspective closely parallels a contemporary interdisciplinary view of living systems. Psychological science usually ignores the background noise in behavioral data. Background noise is what is left over when task demands, experimental manipulations, and other external sources of variability have been eliminated or minimized. What we call background noise is treated as random variability in most research, the nuisance factor in factorial experiments. We argue, to the

Findings in the social psychology literatures on attitudes, social perception, and emotion demonstrate that social information processing involves embodiment, where embodiment refers both to actual bodily states and to simulations of experience in the brain’s modality-specific systems for perception, action, and introspection. We show that embodiment underlies social information processing when the perceiver interacts with actual social objects (online cognition) and when the perceiver represents social objects in their absence (offline cognition). Although many empirical demonstrations of social embodiment exist, no particularly compelling account of them has been offered. We propose that theories of embodied cognition, such as the Perceptual Symbol Systems (PSS) account (Barsalou, 1999), explain and integrate these findings, and that they also suggest exciting new directions for research. We compare the PSS account to a variety of related proposals and show how it addresses criticisms that have previously posed problems for the general embodiment approach. Consider the following findings. Wells and Petty (1980) reported that nodding the head (as in agreement)

We introduce a computational theory of situated language understanding in which the meaning of words and utterances depend on the physical environment and the goals and plans of communication partners. According to the theory, concepts that ground linguistic meaning are neither internal nor external to language users, but instead span the objective-subjective boundary. To model the possible interactions between subject and object, the theory relies on the notion of perceived affordances: structured units of interaction that can be used for prediction at multiple levels of abstraction. Language understanding is treated as a process of filtering perceived affordances. The theory accounts for many aspects of the situated nature of human language use and provides a unified solution to a number of demands on any theory of language understanding including conceptual combination, prototypicality effects, and the generative nature of lexical items. To support the theory, we describe an implemented system that understands verbal commands situated in a virtual gaming environment. The implementation uses probabilistic hierarchical plan recognition to generate perceived affordances. The system has been evaluated on its ability to correctly interpret free-form spontaneous verbal commands recorded from unrehearsed game play between human players. The system is able to “step into the shoes ” of human players and correctly respond to a broad range of verbal commands in which linguistic meaning depends on social and physical context. We quantitatively compare the system’s predictions in response to direct player commands with the actions taken by human players and show generalization to unseen data across a range of situations and verbal constructions. 2 1

1 Language in the World How does language relate to the non-linguistic world? If an agent is able to communicate linguistically and is also able to directly perceive and/or act on the world, how do perception, action, and language interact with and influence each other? Such questions are surely amongst the most important in Cognitive Science and Artificial Intelligence (AI). Language, after all, is a central aspect of the human mind – indeed it may be what distinguishes us from other species. There is sometimes a tendency in the academic world to study language in isolation, as a formal system with rules for well-constructed sentences; or to focus on how language relates to formal notations such as symbolic logic. But language did not evolve as an isolated system or as a way of communicating symbolic logic; it presumably evolved as a mechanism for exchanging information about the world, ultimately providing the medium for cultural transmission across generations. Motivated by these observations, the goal of this special issue is to bring together research in AI that focuses on relating language to the physical world. Language is of course also used to communicate about non-physical referents, but the ubiquity of physical metaphor in language [21] suggests that grounding in the physical world provides the foundations of semantics.

Various lines of research on language have converged on the premise that linguistic knowledge has as its basic unit pairings of form and meaning. The precise nature of the meanings involved, however, remains subject to the longstanding debate between proponents of arbitrary, abstract representations and those who argue for more detailed perceptuo-motor representations. We propose a model, Embodied Construction Grammar (ECG), which integrates these two positions by casting meanings as schematic representations embodied in human perceptual and motor systems. On this view, understanding everyday language entails running mental simulations of its perceptual and motor content. Linguistic meanings are parameterizations of aspects of such simulations; they thus serve as an interface between the relatively discrete properties of language and the detailed and encyclopedic knowledge needed for simulation. This paper assembles evidence from neural imaging and psycholinguistic experiments supporting this general approach to language understanding. It also introduces ECG as a model that fulfills the requisite constraints, and presents two kinds of support for the model. First, we describe two verbal matching studies that test predictions the model makes about the degree of motor detail available in lexical representations. Second, we demonstrate the viability and utility of ECG as a grammar formalism through its capacity to support computational models of language understanding and acquisition.

Human-centered computing (HCC) is centered on humans and what they do, i.e. human actions. Thus, developing an infrastructure for HCC requires understanding human action, at some level of detail. We need to be able to talk about actions, synthesize actions, recognize actions, manipulate actions, imitate actions, imagine and predict actions. How could we achieve this in a principled fashion? This paper proposes that the space of human actions has a linguistic structure. This is a sensory-motor space consisting of the evolution of the joint angles of the human body in movement. The space of human activity has its own

in human-robot interaction

This research uses fMRI to understand the role of eight cortical regions in a relatively complex information-processing task. Modality of input (visual versus auditory) and modality of output (manual versus vocal) are manipulated. Two perceptual regions (auditory cortex and fusiform gyrus) only reflected perceptual encoding. Two motor regions were involved in information rehearsal as well as programming of overt actions. Two cortical regions (parietal and prefrontal) performed processing (retrieval and representational change) independent of input and output modality. The final two regions (anterior cingulate and caudate) were involved in control of cognition independent of modality of input or output and content of the material. An information-processing model, based on the ACT-R theory, is described that predicts the BOLD response in these regions. Different modules in the theory vary in the degree to which they are modality-specific and the degree to which they are involved in central versus peripheral cognitive processes.