Vera and Simon (1993) have provided a helpful and welcome challenge in their articulate questioning of the point of view that emphasizes the situated character of action, including cognition and learning. A full discussion of their arguments and examples requires more space than we are allotted for this response, and we are preparing a longer article that will consider the issues they raised in more detail. We also welcome this opportunity to comment briefly on their provocative arguments. The Issue of Symbols Vera and Simon attribute several beliefs to researchers who are developing situativity theory, some of which we disclaim for ourselves and consider dubious regarding other situativity theorists. However, we do accept their characterization that our view "denies that symbolic processing lies at the heart of intelligence " (pp. 7-8). As we understand the current state of the debate, the issue hinges crucially on the meaning and theoretical status of the concept of symbol. In our view, the emerging scientific practices, empirical findings, and theory that we call Situativity theory; include the development of ecological

Movements of interfaces can be analysed in terms of whether they are sensible, sensable and desirable. Sensible movements are those that users naturally perform; sensable are those that can be measured by a computer; and desirable movements are those that are required by a given application. We show how a systematic comparison of sensible, sensable and desirable movements, especially with regard to how they do not precisely overlap, can reveal potential problems with an interface and also inspire new features. We describe how this approach has been applied to the design of three interfaces: the Augurscope II, a mobile augmented reality interface for outdoors; the Drift Table, an item of furniture that uses load sensing to control the display of aerial photographs; and pointing flashlights at walls and posters in order to play sounds.

Performing 3D interactions with virtual objects easily becomes a complex task, limiting the implementation of larger applications. In order to overcome some of these limitations, this paper describes a framework where the virtual object aids the user to accomplish a pre-programmed possible interaction. Such objects are called Smart Objects, in the sense that they know how the user can interact with them, giving clues to aid the interaction. We show how such objects are constructed, and exemplify the framework with an application where the user, wearing a data glove, can easily open and close drawers of some furniture. Keywords Virtual Reality, manipulation, interaction, data glove, virtual objects, virtual environments. 1. INTRODUCTION Virtual Reality (VR) technology has been employed on various different applications, as for example: human factor analysis, navigation in virtual environments, training, visualisation, and design of objects. A common point to all applications i...

Formal models of animal sensorimotor behavior can provide effective methods for generating robotic intelligence. In this article we describe how schema-theoretic models of the praying mantis derived from behavioral and neuroscientific data can be implemented on a hexapod robot equipped with a real-time color vision system. This implementation incorporates a wide range of behaviors, including obstacle avoidance, prey acquisition, predator avoidance, mating, and chantlitaxia behaviors that can provide guidance to neuroscientists, ethologists, and roboticists alike. The goals of this study are threefold: to provide an understanding and means by which fielded robotic systems are not competing with other agents that are more effective at their designated task; to permit them to be successful competitors within the ecological system and capable of displacing less efficient agents; and that they are ecologically sensitive so that agent-environment dynamics are well-modeled and as predictable ...

For the purposes of manipulation, we would like to know what parts of the environment are physically coherent ensembles -- that is, which parts will move together, and which are more or less independent. It takes a great deal of experience before this judgement can be made from purely visual information. This paper develops active strategies for acquiring that experience through experimental manipulation, using tight correlations between arm motion and optic flow to detect both the arm itself and the boundaries of objects with which it comes into contact.

An approach to information visualization based on algebraic semiotics is introduced, and illustrated with examples. Semiotics is the general theory of signs, and algebraic semiotics is a new approach combining algebraic abstract data type theory with a grounding in social reality. The most important new ideas are to use semiotic spaces for systems of related signs, semiotic morphisms for representations of signs, and preservation orderings for the quality of representations.

In 2 experiments, eye movements were monitored as participants followed instructions containing temporary syntactic ambiguities (e.g., “Pour the egg in the bowl over the flour”). The authors varied the affordances of task-relevant objects with respect to the action required by the instruction (e.g., whether 1 or both eggs in the visual workspace were in liquid form, allowing them to be poured). The number of candidate objects that could afford the action was found to determine whether listeners initially misinterpreted the ambiguous phrase (“in the bowl”) as specifying a location. The findings indicate that syntactic decisions are guided by the listener’s situation-specific evaluation of how to achieve the behavioral goal of an utterance. As a sentence unfolds in time, the grammatical relationships among its constituents are often temporarily ambiguous. For example, the phrase italicized in (1) may indicate the location where an egg is being poured, or may specify which of several eggs is intended. (1) The baker poured the egg in the bowl... (2) a....while stirring continuously.

We introduce a computational theory of situated language understanding in which the meaning of words and utterances depend on the physical environment and the goals and plans of communication partners. According to the theory, concepts that ground linguistic meaning are neither internal nor external to language users, but instead span the objective-subjective boundary. To model the possible interactions between subject and object, the theory relies on the notion of perceived affordances: structured units of interaction that can be used for prediction at multiple levels of abstraction. Language understanding is treated as a process of filtering perceived affordances. The theory accounts for many aspects of the situated nature of human language use and provides a unified solution to a number of demands on any theory of language understanding including conceptual combination, prototypicality effects, and the generative nature of lexical items. To support the theory, we describe an implemented system that understands verbal commands situated in a virtual gaming environment. The implementation uses probabilistic hierarchical plan recognition to generate perceived affordances. The system has been evaluated on its ability to correctly interpret free-form spontaneous verbal commands recorded from unrehearsed game play between human players. The system is able to “step into the shoes ” of human players and correctly respond to a broad range of verbal commands in which linguistic meaning depends on social and physical context. We quantitatively compare the system’s predictions in response to direct player commands with the actions taken by human players and show generalization to unseen data across a range of situations and verbal constructions. 2 1

The acquisition of information is generally thought to be deliberately sought using a search or query mechanism or by browsing or scanning an information space. People, however, find information without seeking it through accidental, incidental or serendipitous discoveries, often in combination with other information acquisition episodes. The value of this phenomenon to an individual or an organization can be equated with the impact of serendipitous breakthroughs in science and medicine. Although largely ignored in information systems development and research, serendipitous retrieval complements querying and browsing, and together they provide a holistic, ecological approach to information acquisition and define the key approaches to a digital library. In this paper, the concept of serendipitous information retrieval is introduced and validated with data from a study of news readers, along with some approaches for how to facilitate it.

Schema Language (ASL) MissionLab (Mlab) Perceptual-Motor Weitzenfeld Arkin Cervantes Predictions Results Common Language Simulations Robot Experiments Biological Data Neural Simulation Language (NSL) Schema Model Figure 1: Interdisciplinary Interactions 1.1 Neuroscience and Ethology The biological group has been studying visuomotor coordination phenomena in amphibia (toad) and insects (praying mantis). These animals live within a three dimensional environment, rich in multiple modes of sensory signals, but their behavior is mainly guided by visual information. From an ecological point of view, these animals react to visual environmental domains of interaction which can be classified into two groups: moving and non-moving objects. Diverse stationary objects may influence the animal's next action which, in general, is directed to improve the animal 's survival chances. For example, frogs moved towards zones in the visual field where blue is preponderant, a situation that might be asso...