This article addresses the question of how the processor decides on its initial strategy for syntactic ambiguity resolution. At a point of ambiguity, more than one analysis is possible. An effective strategy might be to adopt the analysis that has most frequently turned out to be correct in the past. Assuming that the world stays the same in most respects, the analysis that has most frequently been correct in the past should provide a good estimate of which analysis is most likely to be correct again. Hence, by adopting this analysis, the processor should make fewer errors than if it chose any other analysis

Abstract not found

This paper presents a theory of visual word recognition that assumes that, in the tasks of word identification, lexical decision and semantic categorization, human readers behave as optimal Bayesian decision-makers. This leads to the development of a computational model of word recognition, the Bayesian Reader. The Bayesian Reader successfully simulates some of the most significant data on human reading. The model accounts for the nature of the function relating word-frequency to reaction time and identification threshold, the effects of neighborhood density and its interaction with frequency, and the variation in the pattern of neighborhood density effects seen in different experimental tasks. Both the general behavior of the model, and the way the model predicts different patterns of results in different tasks, follow entirely from the assumption that human readers approximate optimal Bayesian decision-makers.

Data-Oriented Parsing (DOP) is a probabilistic performance approach to parsing natural language. Several DOP models have been proposed since it was introduced by Scha (1990), achieving promising results. One important feature of these models is the probability estimation procedure. Two major estimators have been put forward: Bod (1993) uses a relative frequency estimator; Bonnema (1999) adds a rescaling factor to correct for tree size effects. Both estimators, however, present biases. Moreover, Bod's estimator has been shown to be inconsistent (Johnson, 2002), meaning that the probability estimates hypothesized by the model do not approach the true probabilities that generated the data as the sample size grows. In this thesis, we implement a new estimation procedure that tackles the shortcomings of the two previous methods. The main idea is to treat derivation events not as disjoint, but as interrelated in a hierarchical cascade of parse tree derivations. We show that this new estimator -- called the Back-Off DOP (BO-DOP) estimator -- outperforms both previous models. We tested it on the OVIS treebank, a Dutch language, speech-based system, and report error reductions of up to 11.4% and 15% when compared to, respectively, Bod's and Bonnema's estimators.

We show that Viterbi (or “hard”) EM is well-suited to unsupervised grammar induction. It is more accurate than standard inside-outside re-estimation (classic EM), significantly faster, and simpler. Our experiments with Klein and Manning’s Dependency Model with Valence (DMV) attain state-of-the-art performance — 44.8% accuracy on Section 23 (all sentences) of the Wall Street Journal corpus — without clever initialization; with a good initializer, Viterbi training improves to 47.9%. This generalizes to the Brown corpus, our held-out set, where accuracy reaches 50.8 % — a 7.5 % gain over previous best results. We find that classic EM learns better from short sentences but cannot cope with longer ones, where Viterbi thrives. However, we explain that both algorithms optimize the wrong objectives and prove that there are fundamental disconnects between the likelihoods of sentences, best parses, and true parses, beyond the wellestablished discrepancies between likelihood, accuracy and extrinsic performance. 1

A fundamental goal of psycholinguistic research is to understand the architectures and mechanisms that underlie language comprehension. Such an account entails an understanding of the representation and organization of linguistic knowledge in the mind and a theory of how that knowledge is used dynamically to recover the interpretation of the utterances we encounter. While research in theoretical and computational linguistics has demonstrated the tremendous complexities of language understanding, our intuitive experience of language is rather different. For the most part people understand the utterances they encounter effortlessly and accurately. In constructing models of how people comprehend language, we are thus presented with what we dub the performance paradox: How is it that people understand language so effectively given such complexity and ambiguity? In our pursuit and evaluation of new theories, we typically consider how well a particular model is able to account for observed results from the relevant range of controlled psycholinguistic experiments (empirical adequacy), and also the ability of the model to explain why the language comprehension system has the form and function it does (explanatory adequacy). Interestingly, research over the past twenty-five years has led to tremendous variety in proposals for parsing, disambiguation, and reanalysis mechanisms, many of which have been realized as computational models. However, while it is possible to classify models – e.g., according to whether they are modular, interactive, serial, parallel, or probabilistic – consensus at any concrete level has been largely

Probabilistic methods are providing new explanatory approaches to fundamental cognitive science questions of how humans structure, process and acquire language. This review examines probabilistic models defined over traditional symbolic structures. Language comprehension and production involve probabilistic inference in such models; and acquisition involves choosing the best model, given innate constraints and linguistic and other input. Probabilistic models can account for the learning and processing of language, while maintaining the sophistication of symbolic models. A recent burgeoning of theoretical developments and online corpus creation has enabled large models to be tested, revealing probabilistic constraints in processing, undermining acquisition arguments based on a perceived poverty

This article deals with gradience in human sentence processing. We review the experimental evidence for the role of experience in guiding the decisions of the sentence processor. Based on this evidence, we argue that the gradient behavior observed in the processing of certain syntactic constructions can be traced back to the amount of past experience that the processor has had with these constructions. In modeling terms, linguistic experience can be approximated using large, balanced corpora. We give an overview of corpus-based and probabilistic models in the literature that have exploited this fact, and hence are well placed to make gradient predictions about processing behavior. Finally, we discuss a number of questions regarding the relationship between gradience in sentence processing and gradient grammaticality, and come to the conclusion that these two phenomena should be treated separately in conceptual and modeling terms.

This article examines cognitive process models of human sentence comprehension based on the idea of informed search. These models are rational in the sense that they strive to quickly find a good syntactic analysis. Informed search derives a new account of garden pathing that handles traditional counterexamples. It supports a symbolic explanation for local coherence as well as an algorithmic account of entropy reduction. The models are expressed in a broad framework for theories of human sentence comprehension. 1

Towards a rational theory of human information acquisition