This paper investigates the role of resource allocation as a source of processing difficulty in human sentence comprehension. The paper proposes a simple informationtheoretic characterization of processing difficulty as the work incurred by resource reallocation during parallel, incremental, probabilistic disambiguation in sentence comprehension, and demonstrates its equivalence to the theory of Hale (2001), in which the difficulty of a word is proportional to its surprisal (its negative log-probability) in the context within which it appears. This proposal subsumes and clarifies findings that high-constraint contexts can facilitate lexical processing, and connects these findings to well-known models of parallel constraint-based comprehension. In addition, the theory leads to a number of specific predictions about the role of expectation in syntactic comprehension, including the reversal of locality-based difficulty patterns in syntactically constrained contexts, and conditions under which increased ambiguity facilitates processing. The paper examines a range of established results bearing on these predictions, and shows that they are largely consistent with the surprisal theory.

In everyday tasks, selecting actions in the proper sequence requires a continuously updated representation of temporal context. Many existing models address this problem by positing a hierarchy of processing units, mirroring the roughly hierarchical structure of naturalistic tasks themselves. Such an approach has led to a number of difficulties, including a reliance on overly rigid sequencing mechanisms, an inability to account for context sensitivity in behavior, and a failure to address learning. We consider here an alternative framework, according to which the representation of temporal context is facilitated by recurrent connections within a network mapping from environmental inputs to actions. Applying this approach to a specific, and in many ways prototypical, everyday task (coffee-making), we examine its ability to account for several central characteristics of normal and impaired human performance. The model we consider learns to deal flexibly with a complex set of sequencing constraints, encoding contextual information at multiple time-scales within a single, distributed internal representation. Mildly degrading this context representation leads

Psycholinguistic research has shown that the influence of abstract syntactic knowledge on performance is shaped by particular sentences that have been experienced. To explore this idea, the authors applied a connectionist model of sentence production to the development and use of abstract syntax. The model makes use of (a) error-based learning to acquire and adapt sequencing mechanisms and (b) meaning–form mappings to derive syntactic representations. The model is able to account for most of what is known about structural priming in adult speakers, as well as key findings in preferential looking and elicited production studies of language acquisition. The model suggests how abstract knowledge and concrete experience are balanced in the development and use of syntax.

One of the central problems in the study of human language processing is ambiguity resolution: How do people resolve the extremely pervasive ambiguity of the language they encounter? One possible answer to this question is suggested by experiencebased models, which claim that people typically resolve ambiguities in a way which has been successful in the past. In order to determine the course of action that has been "successful in the past" when faced with some ambiguity, it is necessary to generalise over past experience. In this paper, we will present a computational experience-based model, which learns to generalise over linguistic experience from exposure to syntactic structures in a corpus. The model is a hybrid system, which uses symbolic grammars to build and represent syntactic structures, and neural networks to rank these structures on the basis of its experience. We use a dynamic grammar, which provides a very tight correspondence between grammatical derivations and incremental processing, and recursive neural networks, which are able to deal with the complex hierarchical structures produced by the grammar. We demonstrate that the model reproduces a number of the structural preferences found in the experimental psycholinguistics literature, and also performs well on unrestricted text.

The lexical semantic system is an important component of human language and cognitive processing. One approach to modeling semantic knowledge makes use of hand-constructed networks or trees of interconnected word senses (Miller, Beckwith, Fellbaum, Gross, & Miller, 1990; Jarmasz & Szpakowicz, 2003). An alternative approach seeks to model word meanings as high-dimensional vectors, which are derived from the cooccurrence of words in unlabeled text corpora (Landauer & Dumais, 1997; Burgess & Lund, 1997a). This paper introduces a new vector-space method for deriving word-meanings from large corpora that was inspired by the HAL and LSA models, but which achieves better and more consistent results in predicting human similarity judgments. We explain the new model, known as COALS, and how it relates to prior methods, and then evaluate the various models on a range of tasks, including a novel set of semantic similarity ratings involving both semantically and morphologically related terms.

A method is presented for calculating the amount of information conveyed to a hearer by a speaker emitting a sentence generated by a probabilistic grammar known to both parties. The method applies the work of Grenander (1967) to the intermediate states of a top-down parser. This allows the uncertainty about structural ambiguity to be calculated at each point in a sentence. Subtracting these values at successive points gives the information conveyed by a word in a sentence. Word-byword information conveyed is calculated for several small probabilistic grammars, and it is suggested that the number of bits conveyed per word is a determinant of reading times and other measures of cognitive load. KEY WORDS: computational psycholinguistics; entropy reduction.

This book is about the interface between natural language and the sensorimotor system. It is obvious that there is an interface between language and sensorimotor cognition, because we can talk about what we see and do. The main proposal in the book is that the interface is more direct than is commonly assumed. To argue for this proposal I focus on a simple concrete episode—a man grabbing a cup—which can be reported in a simple transitive sentence (e.g. the English sentence The man grabbed a cup). In the first part of the book I present a detailed model of the sensorimotor processes involved in experiencing this episode, both as the agent bringing it about and as an observer watching it happen. The model draws on a large body of research in neuroscience and psychology. I also present a model of the syntactic structure of the associated transitive sentence, developed within the entirely separate discipline of theoretical linguistics. This latter model is a version of Chomsky’s ‘Minimalist ’ syntactic theory, which assumes that a sentence reporting the episode has the same underlying syntactic structure (called ‘logical form’) regardless of which language it is in. My main proposal is that these two independently motivated models are in fact closely

The lexical semantic system is an important component of human language and cognitive processing. One approach to modeling semantic knowledge makes use of hand-constructed networks or trees of interconnected word senses (Miller, Beckwith, Fellbaum, Gross, & Miller, 1990; Jarmasz & Szpakowicz, 2003). An alternative approach seeks to model word meanings as high-dimensional vectors, which are derived from the cooccurrence of words in unlabeled text corpora (Landauer & Dumais, 1997; Burgess & Lund, 1997a). This paper introduces a new vector-space method for deriving word-meanings from large corpora that was inspired by the HAL and LSA models, but which achieves better and more consistent results in predicting human similarity judgments. We explain the new model, known as COALS, and how it relates to prior methods, and then evaluate the various models on a range of tasks, including a novel set of semantic similarity ratings involving both semantically and morphologically related terms.

To Dandelion Kaczmarczyk, who always reminded me about the most important things in life. Acknowledgments There are so many people who supported, encouraged and mentored me while I worked on this dissertation. Most important, I would like to thank my advisor, Risto Miikkulainen. First, for supervising this interdisciplinary research. Second, for teaching me so much about conducting rigorous research and expressing my results with confidence. Third, for being a nice person. I would also like to thank the other members of my committee. Andrew Bernat, for his excellent advice on many occasions; Anthony Petrosino, for directing me to important resources on cognition and learning; Raymond Mooney, for his perspective on machine learning; Bradley Love for his perspective from cognitive psychology; Lowell Bethel, for his encouragement, especially during my early years at UT. Many other people supported me and my work at critical junctures. I want to especially thank Marilla Svinicki for her support during my comprehensive exams, and when I was developing my human subject study. Also, Jim Bednar, for his technical advice on numerous occasions, and Elaine Rich for her understanding of the importance of teaching and learning. Many of the staff in

Abstract—Connectionist natural language processing models that consider the temporal extension of sentence analysis often make use of local representation, allocating only one unit for each word at the input and output layers of the connectionist architecture. Thus, for increasing the lexicon, it is mandatory to modify the architecture and re-train the network. On the other hand, the proposed system Pred-DR attempts to predict the next word in declarative sentences presented sequentially one word at a time, giving meaning to the units of the connectionist architecture by means of distributed representations based on semantic features. The words are fractionated into their semantic microfeature arrays. Consequently, Pred-DR is able to generalize to new words without increasing the number of processors in its architecture, provided that their semantic features are supplied. This way, it is achieved a considerable performance on connectionist natural language processing using the classical semantic microfeature framework. The system learns to relate the input word array to its possible next word, “remembering ” the previous words seen before in a semantically sound sentence. For each input word, Pred-DR gives, as outcome, a list of probabilities of occurrence of next words in the sentence context.