This paper presents an implemented computational model of word acquisition which learns directly from raw multimodal sensory input. Set in an information theoretic framework, the model acquires a lexicon by finding and statistically modeling consistent cross-modal structure. The model has been implemented in a system using novel speech processing, computer vision, and machine learning algorithms. In evaluations the model successfully performed speech segmentation, word discovery and visual categorization from spontaneous infant-directed speech paired with video images of single objects. These results demonstrate the possibility of using state-of-the-art techniques from sensory pattern recognition and machine learning to implement cognitive models which can process raw sensor data without the need for human transcription or labeling.

Many theorists have dismissed a priori the idea that distributional information could play a significant role in syntactic category acquisition. We demonstrate empirically that such information provides a powerful cue to syntactic category membership, which can be exploited by a variety of simple, psychologically plausible mechanisms. We present a range of results using a large corpus of child-directed speech and explore their psychological implications. While our results show that a considerable amount of information concerning the syntac-tic categories can be obtained from distributional information alone, we stress that many other sources of information may also be potential contributors to the identification of syntactic classes. I.

Human speech provides a natural and intuitive interface for both communicating with humanoid robots as well as for teaching them. In general, the acoustic pattern of speech contains three kinds of information: who the speaker is, what the speaker said, and how the speaker said it. This paper focuses on the question of recognizing affective communicative intent in robot-directed speech. We present an approach for recognizing four distinct prosodic patterns that communicate praise, prohibition, attention, and comfort to preverbal infants. These communicative intents are well matched to teaching a robot since praise, prohibition, and directing the robot’s attention to relevant aspects of a task, could be used by a human instructor to intuitively facilitate the robot’s learning process. We integrate this perceptual ability into our robot’s ”emotion ” system, thereby allowing a human to directly manipulate the robot’s affective state. This has a powerful organizing influence on the robot’s behavior, and will ultimately be used to socially communicate affective reinforcement. Communicative efficacy has been tested with people very familiar with the robot as well as with naive subjects.

We describe an empirically based approach to the computational management of dialogues. It is based on an explicit theoretically motivated position regarding the status of computational models, where it is claimed that computational models of discourse can only be about computers' processing of language. The dialogue model is based on an extensive analysis of collected dialogues from various application domains. Issues concerning computational tractability has also been decisive for its development. It is concluded that a simple dialogue grammar based model is sufficient for the management of dialogues with natural language interfaces. We also describe the grammar used by the dialogue manager for a Natural Language interface for a database system. Introduction Most, if not all, work on dialogues in presentday computational linguistics do not make explicit to which extent the models and theories developed should be seen as theories about the processing of dialogue by computers or peop...

Our long-term objective is to develop robots that engage in natural language-mediated cooperative tasks with humans. To support this goal, we are developing an amodal representation and associated processes which is called a grounded situation model (GSM). We are also developing a modular architecture in which the GSM resides in a centrally located module, around which there are language, perception, and actionrelated modules. The GSM acts as a sensor-updated "structured blackboard", that serves as a workspace with contents similar to a "theatrical stage" in the robot's "mind", which might be filled in with present, past or imagined situations. Two main desiderata drive the design of the GSM: first, "parsing" situations into ontological types and relations that reflect human language semantics, and second, allowing bidirectional translation between sensory-derived data/expectations and linguistic descriptions. We present an implemented system that allows of a range of conversational and assistive behavior by a manipulator robot. The robot updates beliefs (held in the GSM) about its physical environment, the human user, and itself, based on a mixture of linguistic, visual and proprioceptive evidence. It can answer basic questions about the present or past and also perform actions through verbal interaction. Most importantly, a novel contribution of our approach is the robot's ability for seamless integration of both language- and sensor-derived information about the situation: For example, the system can acquire parts of situations either by seeing them or by "imagining" them through descriptions given by the user: "There is a red ball at the left". These situations can later be used to create mental imagery and sensory expectations, thus enabling the aforementioned bidi...

Modeling Modeling is not merely a process of behavioral mimicry. Highly functional patterns of behavior, which constitute the proven skills and established customs of a culture, may be adopted in essentially the same form as they are exemplified. There is little leeway for 25 improvisation on how to drive automobiles or to perform arithmetic operations. However, in many activities, subskills must be improvised to suit varying circumstances. Modeling influences can convey rules for generative and innovative behavior as well. This higher-level learning is achieved through abstract modeling. Rule-governed behavior differs in specific content and other details but it contains the same underlying rule. For example, the modeled statements, "The dog is being petted," and "the window was opened" refer to different things but the linguistic rule-- the passive form--is the same. In abstract modeling, observers extract the rule embodied in the specific behavior exhibited by others. Once they lear...

We report a quantitative analysis of the cross-utterance coordination observed in child-directed language, where successive utterances often overlap in a manner that makes their constituent structure more prominent, and describe the application of a recently published unsupervised algorithm for grammar induction to the largest available corpus of such language, producing a grammar capable of accepting and generating novel wellformed sentences. We also introduce a new corpus-based method for assessing the precision and recall of an automatically acquired generative grammar without recourse to human judgment. The present work sets the stage for the eventual development of more powerful unsupervised algorithms for language acquisition, which would make use of the coordination structures present in natural child-directed speech.

Our long-term objective is to develop robots that engage in natural language-mediated cooperative tasks with humans. To support this goal, we are developing an amodal representation called a grounded situation model (GSM), as well as a modular architecture in which the GSM resides in a centrally located module.

In recent years we have seen an increase in the work on the empirical foundations of computational theories of discourse, as well as in the increased use of empirical evaluation methods for natural language systems. In this paper I argue that presenting results from well conducted empirical studies of particular discourses or dialogues is necessary and important but not enough to foster the development of computational theories of discourse. Equally important is making clear to which other cases of agents and situations the results obtained apply. As far as the issue of agents is concerned, it is argued that present day computational theories of discourse can only be seen as theories of computer's processing of language, and not for all kinds of agents, and some consequences of this position are discussed. When, on the other hand, we come to the issue of dialogue situation I do not present any specific theoretical position. Instead a number of dimensions or parameters that seem to infl...

Abstract: In a series of studies, we examined how mothers naturally stress words across multiple mentions in speech to their infants and how this marking influences infants ' recognition of words in fluent speech. We first collected samples of mothers ' infant-directed speech using a technique that induced multiple repetitions of target words. Acoustic analyses revealed that mothers systematically alternated between emphatic and nonemphatic stress when talking to their infants. Using the headturn preference procedure, we then tested 7.5-month-old infants on their ability to detect familiarized bisyllabic words in fluent speech. Stress of target words (emphatic and nonemphatic) was systematically varied across familiarization and recognition phases of four experiments. Results indicated that, although infants generally prefer listening to words produced with emphatic stress, recognition was enhanced when the degree of emphatic stress at familiarization matched the degree of emphatic stress at recognition.