A theoretical framework for grounding language is introduced that provides a computational path from sensing and motor action to words and speech acts. The approach combines concepts from semiotics and schema theory to develop a holistic approach to linguistic meaning. Schemas serve as structured beliefs that are grounded in an agent’s physical environment through a causal-predictive cycle of action and perception. Words and basic speech acts are interpreted in terms of grounded schemas. The framework reflects lessons learned from implementations of several language processing robots. It provides a basis for the analysis and design of situated, multimodal communication systems that straddle symbolic and non-symbolic realms.

To build robots that engage in fluid face-to-face spoken conversations with people, robots must have ways to connect what they say to what they see. A critical aspect of how language connects to vision is that language encodes points of view. The meaning of my left and your left differs due to an implied shift of visual perspective. The connection of language to vision also relies on object permanence. We can talk about things that are not in view. For a robot to participate in situated spoken dialog, it must have the capacity to imagine shifts of perspective, and it must maintain object permanence. We present a set of representations and procedures that enable a robotic manipulator to maintain a “mental model” of its physical environment by coupling active vision to physical simulation. Within this model, “imagined” views can be generated from arbitrary perspectives, providing the basis for situated language comprehension and production. An initial application of mental imagery for spatial language understanding for an interactive robot is described.

1 Language in the World How does language relate to the non-linguistic world? If an agent is able to communicate linguistically and is also able to directly perceive and/or act on the world, how do perception, action, and language interact with and influence each other? Such questions are surely amongst the most important in Cognitive Science and Artificial Intelligence (AI). Language, after all, is a central aspect of the human mind – indeed it may be what distinguishes us from other species. There is sometimes a tendency in the academic world to study language in isolation, as a formal system with rules for well-constructed sentences; or to focus on how language relates to formal notations such as symbolic logic. But language did not evolve as an isolated system or as a way of communicating symbolic logic; it presumably evolved as a mechanism for exchanging information about the world, ultimately providing the medium for cultural transmission across generations. Motivated by these observations, the goal of this special issue is to bring together research in AI that focuses on relating language to the physical world. Language is of course also used to communicate about non-physical referents, but the ubiquity of physical metaphor in language [21] suggests that grounding in the physical world provides the foundations of semantics.

The natural language learning problem has attracted the attention of researchers for several decades. Computational and formal models of language acquisition have provided some preliminary, yet promising insights of how children learn the language of their community. Further, these formal models also provide an operational framework for the numerous practical applications of language learning. We will survey some of the key results in formal language learning. In particular, we will discuss the prominent computational approaches for learning different classes of formal languages and discuss how these fit in the broad context of natural language learning.

Many believe that the most important result to come out of the last ten years of neural network research is the significant change in perspective in the neuroscience community towards a theory of computational neurobiology and functional neuromodels. Arriving on a fast moving train from the other direction is semiconductor technology, one of the greatest technology success stories of all time – transistors are now approaching deep submicron (less than 100 nanometers) in size, and we will soon be building silicon chips with over 1 billion transistors. The marriage of these two technologies is creating what Andy Grove (ex-CEO of Intel) refers to as a strategic inflection point. Although previous attempts at merging these technologies were premature, silicon and computational neurobiology are now merging to create an extremely powerful, and radically new form of computation. 1.

It has recently been demonstrated that certain neural circuitry involved in the execution of specific motor actions is also used when the very same motor actions are observed or when language describing those actions is perceived. In humans, the pre-motor cortex is organized into regions that are involved in the execution and observation of actions performed by at least the following three general areas: the mouth, the hand, and the leg. The discovery of this “mirror system”, involved in production and perception of motor behavior, leads to a viable hypothesis about the processing of linguistic units that refer to these actions. It could be that understanding a verb describing an action involves the activation of the very same mirror circuitry involved in performing and recognizing that action. This hypothesis is tested in a matching task, in which subjects were presented first with an image depicting some action, followed by a verb that either described that action or did not. They were asked to decide as quickly as possible whether the verb was appropriate to the image. It was reasoned that if the verbs and images for particular actions recruited the same mirror circuitry, then there should be interference in those cases where the actions described by the verb and image were not the same but used the same effector. The results showed that it took subjects significantly longer to reject nonmatching verbs and images when the two shared an effector than when they did not. These results support the hypothesis that understanding action language requires the activation of effector-specific neural circuitry in the human mirror system.

Abstract not found

to talk about events from narrated video in a construction grammar framework

Introduction Computation E4 ycholinon tic eek to build theorie of humantheori tic proce e that take the form of implemenEnrickl@umich.eduE8:U . The e model are in8v(3E to explain(3Ew ome pycholin7b tic fun7bEwv9 accompli hed by a et of primitive computationtiveE9U e . The model perform a pycholin a tic tak anE8b889Ew behaviour that canEn8U3UEnEnnr a et of prediction to be compared to human data. A uch, computationUEn ycholinpu tic i a paradigmatic example of cognric modellindigmatic example of cognrickl@umich.eduE8:U computatiB--8 psycholiiB--8NNBT the implicationEnmple of c omethin79(UEnnEnmple of cogn non-computatiii pycholin(b tic . Thi i n4v99U env99UEw(bU e: all p ycholin7v tic theorie are, at ome level, aertion about computationw97vUEn e . ComputationUEn ycholinon tic i ditinb( hed from other form of cognothe modellinother domainother techno ), anot ditin4: hed from other form of p ycholinEw tic theoriinc theo focu onEb3883Ewv:7bEn:7bEnEw4U9UEnrickl mechan m that

Advanced article