Recent studies of eye movements in reading and other information processing tasks, such as music reading, typing, visual search, and scene perception, are reviewed. The major emphasis of the review is on reading as a specific example of cognitive processing. Basic topics discussed with respect to reading are (a) the characteristics of eye movements, (b) the perceptual span, (c) integration of information across saccades, (d) eye movement control, and (e) individual differences (including dyslexia). Similar topics are discussed with respect to the other tasks examined. The basic theme of the review is that eye movement data reflect moment-to-moment cognitive processes in the various tasks examined. Theoretical and practical considerations concerning the use of eye movement data are also discussed. Many studies using eye movements to investigate cognitive processes have appeared over the past 20 years. In an earlier review, I (Rayner, 1978b) argued that since the mid-1970s we have been in a third era of eye movement research and that the success of research in the current era would depend on the ingenuity of researchers in designing interesting and informative

Abstract not found

In this article I present a series of arguments that syntactic structures are built incrementally, in a strict left-to-right order. By assuming incremental structure building it becomes possible to explain the differences between the range of constituents available to different diagnostics of constituency, including movement, ellipsis, coordination, scope and binding. In an incremental derivation structure building creates new constituents, and in doing so may destroy existing constituents. The article presents detailed evidence for the prediction of incremental grammar, that a syntactic process may refer to only those constituents that are present at the point in the derivation when the process applies. Keywords: phrase structure, constituency, incrementality, coordination, binding, scope, ellipsis, movement. 1. Introduction Tests of constituency are basic components of the syntactician's toolbox. By investigating which strings of words can and cannot be moved, deleted...

In 2 experiments, eye movements were monitored as participants followed instructions containing temporary syntactic ambiguities (e.g., “Pour the egg in the bowl over the flour”). The authors varied the affordances of task-relevant objects with respect to the action required by the instruction (e.g., whether 1 or both eggs in the visual workspace were in liquid form, allowing them to be poured). The number of candidate objects that could afford the action was found to determine whether listeners initially misinterpreted the ambiguous phrase (“in the bowl”) as specifying a location. The findings indicate that syntactic decisions are guided by the listener’s situation-specific evaluation of how to achieve the behavioral goal of an utterance. As a sentence unfolds in time, the grammatical relationships among its constituents are often temporarily ambiguous. For example, the phrase italicized in (1) may indicate the location where an egg is being poured, or may specify which of several eggs is intended. (1) The baker poured the egg in the bowl... (2) a....while stirring continuously.

1 Language in the World How does language relate to the non-linguistic world? If an agent is able to communicate linguistically and is also able to directly perceive and/or act on the world, how do perception, action, and language interact with and influence each other? Such questions are surely amongst the most important in Cognitive Science and Artificial Intelligence (AI). Language, after all, is a central aspect of the human mind – indeed it may be what distinguishes us from other species. There is sometimes a tendency in the academic world to study language in isolation, as a formal system with rules for well-constructed sentences; or to focus on how language relates to formal notations such as symbolic logic. But language did not evolve as an isolated system or as a way of communicating symbolic logic; it presumably evolved as a mechanism for exchanging information about the world, ultimately providing the medium for cultural transmission across generations. Motivated by these observations, the goal of this special issue is to bring together research in AI that focuses on relating language to the physical world. Language is of course also used to communicate about non-physical referents, but the ubiquity of physical metaphor in language [21] suggests that grounding in the physical world provides the foundations of semantics.

Multimodal conversational interfaces allow users to carry a dialog with a graphical display using speech to accomplish a particular task. Motivated by previous psycholinguistic findings, we examine how eye-gaze contributes to reference resolution in such a setting. Specifically, we present an integrated probabilistic framework that combines speech and eye-gaze for reference resolution. We further examine the relationship between eyegaze and increased domain modeling with corresponding linguistic processing. Our empirical results show that the incorporation of eye-gaze significantly improves reference resolution performance. This improvement is most dramatic when a simple domain model is used. Our results also show that minimal domain modeling combined with eye-gaze significantly outperforms complex domain modeling without eye-gaze, which indicates that eye-gaze can be used to potentially compensate a lack of domain modeling for reference resolution.

We introduce symbolic dynamics to cognitive scientists with the aim of furthering constructive debate on representation. Symbolic dynamics is a mathematical framework in which both continuous and discrete states of a system can be considered jointly. We discuss a number of theoretical implications this framework has for cognitive science, and offer some consideration of the way in which it might be employed for comparing or conciliating discrete and continuous representational theories. Symbolic dynamics may thus serve as a common, level playing field for debate in theories of cognitive representation.

Fuse is a spoken language understanding system that integrates visual context into early stages of speech recognition. Given a visual scene and a spoken description, the system finds the object in the scene that best fits the meaning of the description. To solve this task, Fuse performs speech recognition and visually-grounded language understanding. Rather than treat these two problems separately, knowledge of the visual semantics of language and the specific contents of the visual scene are fused into speech processing. As a result, the system anticipates various ways a person might describe any object in the scene, and uses these predictions to bias the speech recognizer towards likely sequences of words. A dynamic model of visual attention is used to focus processing on likely objects within the scene as spoken utterances are processed. Visual attention and language prediction reinforce one another and converge on interpretations of incoming speech signals which are most consistent with visual context. In evaluations, the introduction of visual context into the speech recognition process results in significantly improved speech recognition and understanding accuracy. The underlying principles of this model may be applied to a wide range of speech understanding problems including mobile and assistive technologies in which contextual information can be sensed by the system and semantically interpreted to bias processing.

Recent work in sentence processing has highlighted the distinction between serial and parallel application of linguistic constraints in real time. In looking at context effects in syntactic ambiguity resolution, some studies have reported an immediate influence of semantic and discourse information on syntactic parsing (e.g., McRae, Spivey-Knowlton, & Tanenhaus, 1998; Spivey & Tanenhaus, 1998). However, in looking at the effects of various constraints on grammaticality judgments, some studies have reported a temporal precedence of structural information over semantic information (e.g., McElree & Griffith, 1995, 1998). This chapter points to some computational demonstrations of how an apparent temporal dissociation between structural and non-structural information can in fact arise from the dynamics of the processing system, rather than from its architecture, coupled with the specific parameters of the individual stimuli. A prediction of parallel competitive processing systems is then empirically tested with a new methodology: speeded sentence completions. Results are consistent with a parallel account of the application of linguistic constraints and a competitive account of ambiguity resolution.

ABSTRACT—Real-time cognition is best described not as a sequence of logical operations performed on discrete symbols but as a continuously changing pattern of neuronal activity. The continuity in these dynamics indicates that, in between describable states of mind, mental activity does not lend itself to the linguistic labels relied on by much of psychology. We discuss eye-tracking and mousetracking evidence for this temporal continuity and provide geometric visualizations of mental activity, depicting it as a continuous trajectory through a state space (a multidimensional space in which locations correspond to mental states). When the state of the system travels toward a frequently visited region of that space, the destination may constitute recognition of a particular word or a particular object; but on the way there, the majority of the mental trajectory is in intermediate regions of that space, revealing graded mixtures of mental states. KEYWORDS—cognition; continuity; dynamical systems; eye movements When introspecting on one’s thought processes, one often feels as though a discrete concept is considered all by itself, followed by another discrete concept, followed by another. Such a sequence of distinct, nonoverlapping mental constituents is consistent with the string of logical symbols on the tape of a Turing machine. In the 1950s, computing theory devised the theoretical construct of a universal Turing machine, on which the computation of any algorithm could be implemented by moving the tape forward or backward so that a programmed tape head could read the symbols. Such a computing system can, by modifying the discrete symbols on the tape, exhibit a variety of intelligent behaviors that resemble our own cognitive skills, such as performing arithmetic or playing chess. The theoretical construct of a universal Turing machine was quickly imported into psy-