TWIG (“Transportable Word Intension Generator”) is a system that allows a robot to learn compositional meanings for new words that are grounded in its sensory capabilities. The system is novel in its use of logical semantics to infer which entities in the environment are the referents (extensions) of unfamiliar words; its ability to learn the meanings of deictic (“I, ” “this”) pronouns in a real sensory environment; its use of decision trees to implicitly contrast new word definitions with existing ones, thereby creating more complex definitions than if each word were treated as a separate learning problem; and its ability to use words learned in an unsupervised manner in complete grammatical sentences for production, comprehension, or referent inference. In an experiment with a physically embodied robot, TWIG learns grounded meanings for the words “I ” and “you, ” learns that “this ” and “that ” refer to objects of varying proximity, that “he ” is someone talked about in the third person, and that “above ” and “below ” refer to height differences between objects. Follow-up experiments demonstrate the system’s ability to learn different conjugations of “to be”; show that removing either the extension inference or implicit contrast components of the system results in worse definitions; and demonstrate how decision trees can be used to model shifts in meaning based on context in the case of color words.

This chapter presents a physical-computational model of sensory-motor grounded language interpretation for simple speech acts. The model is based on an implemented conversational robot. It combines a cybernetic closed-loop control architecture with structured conceptual schemas. The interpretation of directive and descriptive speech acts consists of translating utterances into updates of memory systems in the controller. The same memory systems also mediate sensory-motor interactions and thus serve as a cross-modal bridge between language, perception, and action. The referential, functional, and connotative meanings of speech acts emerge from the effects of memory updates on the future dynamics of the controller as it physically interacts with its environment. 1

Naturalistic longitudinal recordings of child development promise to reveal fresh perspectives on fundamental questions of language acquisition. In a pilot effort, we have recorded 230,000 hours of audio-video recordings spanning the first three years of one child’s life at home. To study a corpus of this scale and richness, current methods of developmental cognitive science are inadequate. We are developing new methods for data analysis and interpretation that combine pattern recognition algorithms with interactive user interfaces and data visualization. Preliminary speech analysis reveals surprising levels of linguistic fine-tuning by caregivers that may provide crucial support for word learning. Ongoing analyses of the corpus aim to model detailed aspects of the child’s language development as a function of learning mechanisms combined with lifetime experience. Plans to collect similar corpora from more children based on a transportable recording system are underway. Index Terms: language acquisition, rich longitudinal data, human-machine collaborative analysis, computational models

We introduce an approach for physically-grounded natural language interpretation by robots which reacts appropriately to unanticipated physical changes in the environment and dynamically assimilates new information pertinent to ongoing tasks. At the core of the approach is a model of object schemas that enables a robot to encode beliefs about physical objects in its environment using collections of coupled processes responsible for sensorimotor interaction. These interaction processes run concurrently in order to ensure responsiveness to the environment, while coordinating sensorimotor expectations, action planning, and language use. The model has been implemented on a robot that manipulates objects on a tabletop in response to verbal input. The implementation responds to verbal requests such as “Group the green block and the red apple, ” while adapting in real-time to unexpected physical collisions and taking opportunistic advantage of any new information it may receive through perceptual and linguistic channels.

Abstract — It is thought that meaning may be grounded in early childhood language learning via the physical and social interaction of the infant with those around him or her, and that the capacity to use words, phrases and their meaning are acquired through shared referential ‘inference ’ in pragmatic interactions. In order to create appropriate conditions for language learning it would therefore be necessary to expose the robot to similar physical and social contexts. However in the early stages of language learning it is estimated that a 2-year-old child can be exposed to as many as 7,000 utterances per day in varied contextual situations. In this paper we report on our forthcoming experiments which are designed to allow a robot to carry out language learning in a manner analogous to that in early child development and which effectively ‘short cuts ’ holophrase learning. Two approaches are used: simulated babbling through mechanisms which will yield basic word or holophrase structures and an interaction environment between a human and a robot where shared ‘intentional’ referencing and the associations between physical, visual and speech modalities can be experienced by the robot. The output of these experiments, combined to yield word or holophrase structures grounded in the robot’s own actions and modalities, would provide scaffolding for further proto-grammatical usagebased learning via interaction with the physical and social environment involving human feedback to bootstrap developing linguistic competencies. These structures would then form the basis for further studies on language acquisition, including the emergence of negation and more complex grammar. I.

Initially, the analysis and development of adaptive artificial systems has been based in metaphors taken from philosophical schools as well as the disciplines of biology and cognitive science. So far, the dominant approaches exhibit many advantages in specific domains of application but there all have a certain drawback, which is their inability to produce an artificial system which will be able to internally ground its representations so as to use them to produce newer, more developed ones. The respective frameworks are studied in terms of this inability and it is concluded that the problem is traced in the purely causal treatment, function and creation of the notion of representation, wherever it is used. In the case of purely dynamic systems, where the representations seem not to be very useful, it is proposed that the incorporation of a special non-causal kind of representations would give a framework which seems promising in realizing real adaptation. The relevant architecture is analyzed and discussed mainly in terms of its functionality and its contribution to the integration of pragmatic meaning aspects in an artificial system’s interaction.

Abstract. Artificial agents trying to achieve communicative goals in situated interactions in the real-world need powerful computational systems for conceptualizing their environment. In order to provide embodied artificial systems with rich semantics reminiscent of human language complexity, agents need ways of both conceptualizing complex compositional semantic structure and actively reconstructing semantic structure, due to uncertainty and ambiguity in transmission. Furthermore, the systems must be open-ended and adaptive and allow agents to adjust their semantic inventories in order to reach their goals. This paper presents recent progress in modeling open-ended, grounded semantics through a unified software system that addresses these problems. 1

Abstract. How can goals be represented in natural and artificial systems? How can they be learned? How can they trigger actions? This paper describes, analyses and compares two of the most influential models of goal-oriented behavior: the ideomotor principle (IMP), which was introduced in the psychological literature, and the “test, operate, test, exit ” model (TOTE) proposed in the field of cybernetic. This analysis indicates that the IMP and the TOTE highlight complementary aspects of goal-orientedness. In order to illustrate this point, the paper reviews three computational architectures that implement various aspects of the IMP and the TOTE, discusses their main peculiarities and limitations, and suggests how some of their features can be translated into specific mechanisms in order to implement them in artificial intelligent systems.

We have empirically discovered that the space of human actions has a grammatical structure. This is a motoric space consisting of the evolution of the joint angles of the human body in movement. Furthermore, the process of assembling individual human movements into higher level descriptions resembles in a natural sense the process of speech recognition. Thus the space of human activity has its own phonemes, morphemes, words (verbs, nouns, adjectives, adverbs), and sentences formed by its own syntax. This has a number of implications for the grounding problem and cognition in general. With regard to WordNet, the theory points to a future Sensorimotor WordNet which contains a map between the nodes of the current WordNet and the space consisting of human action. In this paper, we suggest initial steps towards closing the semantic gap by grounding language with visuomotor information. The grounding takes place on a set of primitive words which are selected here through verb classification of the WordNet lexicon. A formal approach to the identification of primitive words would consider the basic atoms of WordNet extensions. However, one further extension is required to incorporate grounded information into WordNet in the direction of a sensorimotor WordNet, designated here as WordNet SM.

Abstract. We present a theoretical analysis of schema-based design (SBD), a methodology for designing autonomous agents architectures. We also provide an overview of the AKIRA Schema Language (AKSL), which permits to design schema-based architectures for anticipatory behavior experiments and simulations. Several simulations using AKSL are reviewed, highlighting the relations between pragmatic and epistemic aspects of behavior. Anticipation is crucial in realizing several functionalities with AKSL, such as selecting actions, orienting attention, categorizing and grounding declarative knowledge. 1