To build robots that engage in fluid face-to-face spoken conversations with people, robots must have ways to connect what they say to what they see. A critical aspect of how language connects to vision is that language encodes points of view. The meaning of my left and your left differs due to an implied shift of visual perspective. The connection of language to vision also relies on object permanence. We can talk about things that are not in view. For a robot to participate in situated spoken dialog, it must have the capacity to imagine shifts of perspective, and it must maintain object permanence. We present a set of representations and procedures that enable a robotic manipulator to maintain a “mental model” of its physical environment by coupling active vision to physical simulation. Within this model, “imagined” views can be generated from arbitrary perspectives, providing the basis for situated language comprehension and production. An initial application of mental imagery for spatial language understanding for an interactive robot is described.

Abstract not found

The voluminous literature on linguistic relativity has concerned itself primarily with the search for influences of particular languages on nonlinguistic cognition in situations in which language is not being used, overtly or covertly. This represents a long tradition in which anthropologists, psychologists, and linguists have sought to relate grammatical and semantic systems of a language to the worldview or epistemology or culture of the community of speakers of the language. For example, Lucy has proposed a set of requirements for studies of linguistic relativity. He stipulates that such research “should assess the cognitive performance of individual speakers aside from explicitly verbal contexts and try to establish that any cognitive patterns that are detected also characterize everyday behavior outside of the assessment situation ” (Lucy, 1996, p. 48, emphasis added). In this view, “cognition ” is seen as a collection of concepts and procedures that come into play regardless of whether an individual is engaged in verbal behavior—speaking, listening, or verbal thinking. Such research is directed towards what Lucy calls “an independent cognitive interpretation of reality ” (Lucy, 2000, p. xii). A rather different approach to “cognition ” is provided by investigators who concern themselves with language use and cultural practice. For example, Gumperz and Levinson, introducing Rethinking linguistic relativity (1996, p. 8), underline the importance of “theories of use in context, ” including formal semantic theories (e.g., Discourse Representation Theory, Situation Semantics) and pragmatic theories (Relevance Theory, Gricean theories), along with research in sociolinguistics and linguistic anthropology. In the present paper, I begin with the fact that human beings spend a large portion of their time in linguistic behavior of one sort or another—that is, we are creatures that are almost constantly involved in preparing, producing, and interpreting verbal messages. Accordingly, research on linguistic relativity is incomplete without attention to the cognitive processes that are brought to bear, online, in the course of using language.

1 Language in the World How does language relate to the non-linguistic world? If an agent is able to communicate linguistically and is also able to directly perceive and/or act on the world, how do perception, action, and language interact with and influence each other? Such questions are surely amongst the most important in Cognitive Science and Artificial Intelligence (AI). Language, after all, is a central aspect of the human mind – indeed it may be what distinguishes us from other species. There is sometimes a tendency in the academic world to study language in isolation, as a formal system with rules for well-constructed sentences; or to focus on how language relates to formal notations such as symbolic logic. But language did not evolve as an isolated system or as a way of communicating symbolic logic; it presumably evolved as a mechanism for exchanging information about the world, ultimately providing the medium for cultural transmission across generations. Motivated by these observations, the goal of this special issue is to bring together research in AI that focuses on relating language to the physical world. Language is of course also used to communicate about non-physical referents, but the ubiquity of physical metaphor in language [21] suggests that grounding in the physical world provides the foundations of semantics.

Various lines of research on language have converged on the premise that linguistic knowledge has as its basic unit pairings of form and meaning. The precise nature of the meanings involved, however, remains subject to the longstanding debate between proponents of arbitrary, abstract representations and those who argue for more detailed perceptuo-motor representations. We propose a model, Embodied Construction Grammar (ECG), which integrates these two positions by casting meanings as schematic representations embodied in human perceptual and motor systems. On this view, understanding everyday language entails running mental simulations of its perceptual and motor content. Linguistic meanings are parameterizations of aspects of such simulations; they thus serve as an interface between the relatively discrete properties of language and the detailed and encyclopedic knowledge needed for simulation. This paper assembles evidence from neural imaging and psycholinguistic experiments supporting this general approach to language understanding. It also introduces ECG as a model that fulfills the requisite constraints, and presents two kinds of support for the model. First, we describe two verbal matching studies that test predictions the model makes about the degree of motor detail available in lexical representations. Second, we demonstrate the viability and utility of ECG as a grammar formalism through its capacity to support computational models of language understanding and acquisition.

The fact that human readers can learn about the physical world from textual descriptions leads to a number of interesting questions about the connections between our conceptual understanding of the physical world and how it is reflected in natural language. This thesis investigates some forms in which information about physical phenomena is typically expressed in natural language and how this knowledge can be used to construct models of the underlying physical processes. Based on an analysis of the representations of physical quantities in natural language and common, reoccurring syntactic patterns, we implemented a system that uses Qualitative Process (QP) Theory to guide the semantic interpretation process to capture information about physical phenomena found in natural language text. We have recast QP Theory in terms of frame semantics as FrameNet-compatible representations (QP frames) and use an extendable, controlled subset of English to capture QP specific information from natural language descriptions. In addition to general background knowledge based on a subset of the Cyc knowledge base and the lexical information supplied by a syntactic parser, the semantics of QP Theory are used in rules that guide the semantic interpretation process and the construction of QP Frames. The thesis illustrates that QP Theory, as an established theoretical framework for handling continuous parameters and causation, can provide an essential component of

Objective: Representing roles, i.e. functions of proteins, sequences and structures, is the cornerstone of knowledge representation in functional genomics. The objective of this study is to investigate representation of roles as functional categories or associative relations. We focus on GeneOntology (GO) and the UMLS and take examples from iron metabolism. Methods: The terms corresponding to the main proteins involved in iron metabolism were mapped to GO (including the annotations) and the UMLS. The representation of their biological roles was then analyzed. Results: Functional aspects are represented in both GO and the UMLS. However, the granularity may not be appropriate. Discussion: Advantages and limits of functional categories and associative relations are discussed.

Language learning is one of the most complicated feats that human beings accomplish. Any number of very real reasons exist as to why L2 learning presents tremendous challenges. However, instructed L2 learning has been further complicated by the fact that important elements of systematicity that

Abstract. In our targeted scenario, humans can flexibly establish joint object reference with a robot entirely on the basis of their own intuitions. To reach this aim, the robot needs to be equipped with the kind of knowledge that can be matched in a cognitively adequate way to users ’ intuitive conceptual and linguistic preferences. Such an endeavour requires knowledge about human spatial object reference under consideration of object affordances and functional features. In this paper we motivate our approach by reviewing relevant insights gained in the field of Spatial Cognition, and we discuss the suitability of our robotic system to incorporate these findings. In our context, affordances are visually perceivable functional object aspects shared by the designer of the recognition module and the prospective robot user or instructor. 1

Abstract not found