1 Language in the World How does language relate to the non-linguistic world? If an agent is able to communicate linguistically and is also able to directly perceive and/or act on the world, how do perception, action, and language interact with and influence each other? Such questions are surely amongst the most important in Cognitive Science and Artificial Intelligence (AI). Language, after all, is a central aspect of the human mind – indeed it may be what distinguishes us from other species. There is sometimes a tendency in the academic world to study language in isolation, as a formal system with rules for well-constructed sentences; or to focus on how language relates to formal notations such as symbolic logic. But language did not evolve as an isolated system or as a way of communicating symbolic logic; it presumably evolved as a mechanism for exchanging information about the world, ultimately providing the medium for cultural transmission across generations. Motivated by these observations, the goal of this special issue is to bring together research in AI that focuses on relating language to the physical world. Language is of course also used to communicate about non-physical referents, but the ubiquity of physical metaphor in language [21] suggests that grounding in the physical world provides the foundations of semantics.

Cognitivism, the traditional approach to understanding cognition, has argued for the essential role of symbolic computations over internal mental representations. But this view has been criticized on a number of grounds, one in particular being the assumption of amodality: that the symbols involved in processing are arbitrarily related to their referents. An opposing view—the framework of Perceptual Symbol Systems—holds that the elements of thought should be treated not as amodal symbols, but rather as modality specific, analog representations that simulate particular aspects of perceptual experience. Though this approach has been gaining in popularity from intuitively appealing theoretical accounts, and suggestive empirical support, it has suffered from a lack of specificity for key constructs. To address this problem, this paper presents a more detailed study of the foundational concept of perceptual symbol. The proposal builds from recent work on the skill-based nature of visual perception (the Sensorimotor Contingency Theory), and research that provides tools for representing the inseparable link between