This paper examines the inductive inference of a complex grammar with neural networks -- specifically, the task considered is that of training a network to classify natural language sentences as grammatical or ungrammatical, thereby exhibiting the same kind of discriminatory power provided by the Principles and Parameters linguistic framework, or Government-and-Binding theory. Neural networks are trained, without the division into learned vs. innate components assumed by Chomsky, in an attempt to produce the same judgments as native speakers on sharply grammatical/ungrammatical data. How a recurrent neural network could possess linguistic capability and the properties of various common recurrent neural network architectures are discussed. The problem exhibits training behavior which is often not present with smaller grammars and training was initially difficult. However, after implementing several techniques aimed at improving the convergence of the gradient descent backpropagation-through-time training algorithm, significant learning was possible. It was found that certain architectures are better able to learn an appropriate grammar. The operation of the networks and their training is analyzed. Finally, the extraction of rules in the form of deterministic finite state automata is investigated.

Abstract not found

The distinction between rule-based and similarity-based processes in cognition is of fundamental importance for cognitive science, and has been the focus of a large body of empirical research. However, intuitive uses of the distinction are subject to theoretical difficulties and their relation to empirical evidence is not clear. We propose a ‘core ’ distinction between ruleand similarity-based processes, in terms of the way representations of stored information are ‘matched ’ with the representation of a novel item. This explication captures the intuitively clear-cut cases of processes of each type, and resolves apparent problems with the rule/ similarity distinction. Moreover, it provides a clear target for assessing the psychological and AI literatures. We show that many lines of psychological evidence are less conclusive than sometimes assumed, but suggest that converging lines of evidence may be persuasive. We then argue that the AI literature suggests that approaches which combine rules and similarity are an important new focus for empirical work. © 1998 Elsevier Science B.V. Keywords: Similarity-based process; Rule-based process 1.

Abstract not found

This book deals with the semantics of natural language expressions that are commonly taken to refer to individuals: pronouns, definite descriptions and proper names. It claims, contrary to previous theorizing, that they all have a common syntax and semantics, roughly that which is currently associated by philosophers and linguists with definite descriptions as construed in the tradition of Frege. As well as advancing this proposal, I hope to achieve at least one other aim, that of urging semanticists dealing with pronoun interpretation, in particular donkey anaphora, to consider a wider range of theories at all times than is sometimes done at present. I am thinking particularly of the gulf that seems to have emerged between those who practice some version of dynamic semantics (including DRT) and those who eschew this approach and rely on some version of the E-type analysis for donkey anaphora (if they consider this phenomenon at all). In my opinion there is too little work directly comparing the claims of these two schools (for that is what they amount to) and testing them against the data in the way that any two rival theories might be tested. (Irene Heim’s 1990 article in Linguistics and Philosophy does this, and

We outline a formal foundation for a \structural " (or \symbolic") object/event representation, the necessity of which is acutely felt in all sciences, including mathematics and computer science. The proposed foundation incorporates two hypotheses: 1) the object's formative history must be an integral part of the object representation and 2) the process of object construction is irreversible, i.e. the \trajectory " of the object's formative evolution does not intersect itself. The last hypothesis is equivalent to the generalized axiom of (structural) induction. Some of the main diculties associated with the transition from the classical numeric to the structural representations appear to be related precisely to the development of a formal framework satisfying these two hypotheses. The concept of (inductive) class representation|which has inspired the development of this approach to structural representation|diers fundamentally from the known concepts of class. In the proposed, evolving transformations system (ETS), model, the class is dened by the transformation system|a nite set of weighted transformations acting on the class progenitor| and the generation of the class elements is associated with the corresponding generative process which also induces the class typicality measure. Moreover, in the ETS model, a fundamental role of the object's class in the object's representation is claried: the representation of an object must include the class. From the point of view of ETS model, the classical discrete representations, e.g. strings and graphs, appear now as incomplete special cases, the proper completion of which should incorporate the corresponding formative histories, i.e. those of the corresponding strings or graphs. 1 Concepts which have proved useful for ordinary things easily assume so great an authority over us, that we forget their terrestrial origin and accept them as unalterable facts. They then become labeled as \conceptual necessities", a priori situations, etc. The road of scientic progress is frequently blocked for long periods by such errors.

Abstract not found

We consider the task of training a neural network to classify natural language sentences as grammatical or ungrammatical, thereby exhibiting the same kind of discriminatory power provided by the Principles and Parameters linguistic framework, or Government and Binding theory. We investigate the following models: feed-forward neural networks, Frasconi-Gori-Soda and Back-Tsoi locally recurrent neural networks, Williams and Zipser and Elman recurrent neural networks, Euclidean and edit-distance nearest-neighbors, and decision trees. Non-neural network machine learning methods are included primarily for comparison. We find that the Elman and Williams & Zipser recurrent neural networks are able to find a representation for the grammar which we believe is more parsimonious. These models exhibit the best performance. 1 Motivation 1.1 Representational Power of Recurrent Neural Networks Natural language has traditionally been handled using symbolic computation and recursive processes. The most ...

We examine the inductive inference of a complex grammar - specifically, we consider the task of training a model to classify natural language sentences as grammatical or ungrammatical, thereby exhibiting the same kind of discriminatory power provided by the Principles and Parameters linguistic framework, or Government-and-Binding theory. We investigate the following models: feed-forward neural networks, Fransconi-Gori-Soda and Back-Tsoi locally recurrent networks, Elman, Narendra & Parthasarathy, and Williams & Zipser recurrent networks, Euclidean and edit-distance nearest-neighbors, simulated annealing, and decision trees. The feed-forward neural networks and non-neural network machine learning models are included primarily for comparison. We address the question: How can a neural network, with its distributed nature and gradient descent based iterative calculations, possess linguistic capability which is traditionally handled with symbolic computation and recursive processes? Initial...

Recurrent neural networks are complex parametric dynamic systems that can exhibit a wide range of different behavior. We consider the task of grammatical inference with recurrent neural networks. Specifically, we consider the task of classifying natural language sentences as grammatical or ungrammatical - can a recurrent neural network be made to exhibit the same kind of discriminatory power which is provided by the Principles and Parameters linguistic framework, or Government and Binding theory? We attempt to train a network, without the bifurcation into learned vs. innate components assumed by Chomsky, to produce the same judgments as native speakers on sharply grammatical/ungrammatical data. We consider how a recurrent neural network could possess linguistic capability, and investigate the properties of Elman, Narendra & Parthasarathy (N&P) and Williams & Zipser (W&Z) recurrent networks, and Frasconi-Gori-Soda (FGS) locally recurrent networks in this setting. We show that both Elman...