This paper describes NETtalk, a class of massively-parallel network systems that learn to convert English text to speech. The memory representations for pronunciations are learned by practice and are shared among many processing units. The performance of NETtalk has some similarities with observed human performance. (i) The learning follows a power law. (;i) The more words the network learns, the better it is at generalizing and correctly pronouncing new words, (iii) The performance of the network degrades very slowly as connections in the network are damaged: no single link or processing unit is essential. (iv) Relearning after damage is much faster than learning during the original training. (v) Distributed or spaced practice is more effective for long-term retention than massed practice. Network models can be constructed that have the same performance and learning characteristics on a particular task, but differ completely at the levels of synaptic strengths and single-unit responses. However, hierarchical clustering techniques applied to NETtalk reveal that these different networks have similar internal representations of letter-to-sound correspondences within groups of processing units. This suggests that invariant internal representations may be found in assemblies of neurons intermediate in size between highly localized and completely distributed representations.

A major goal of research on networks of neuron-like processing units is to discover efficient learning procedures that allow these networks to construct complex internal representations of their environment. The learning procedures must be capable of modifying the connection strengths in such a way that internal units which are not part of the input or output come to represent important features of the task domain. Several interesting gradient-descent procedures have recently been discovered. Each connection computes the derivative, with respect to the connection strength, of a global measure of the error in the performance of the network. The strength is then adjusted in the direction that decreases the error. These relatively simple, gradient-descent learning procedures work well for small tasks and the new challenge is to find ways of improving their convergence rate and their generalization abilities so that they can be applied to larger, more realistic tasks.

We discuss external and internal graphical and linguistic representational systems. We argue that a cognitive theory of peoples' reasoning performance must account for (a) the logical equivalence of inferences expressed in graphical and linguistic form; and (b) the implementational differences that affect facility of inference. Our theory proposes that graphical representations limit abstraction and thereby aid processibility. We discuss the ideas of specificity and abstraction, and their cognitive relevance. Empirical support comes from tasks involving (i) the manipulation of external graphics; and (ii) no external graphics. For (i), we take Euler's Circles, provide a novel computational reconstruction, show how it captures abstractions, and contrast it with earlier construals, and with Mental Models' representations. We demonstrate equivalence of the graphical Euler system, and the non-graphical Mental Models system. For (ii), we discuss text comprehension, and the mental ...

We describe a distributed model of information processing and memory and apply it to the representation of general and specific information. The model consists of a large number of simple processing elements which send excitatory and inhibitory signals to each other via modifiable connections. Information processing is thought of as the process whereby patterns of activation are formed over the units in the model through their excitatory and inhibitory interactions. The memory trace of a processing event is the change or increment to the strengths of the interconnections that results from the processing event. The traces of separate events are superimposed on each other in the values of the connection strengths that result from the entire set of traces stored in the memory. The model is applied to a number of findings related to the question of whether we store abstract representations or an enumeration of specific experiences in memory. The model simulates the results of a number of important experiments which have been taken as evidence for the enumeration of specific experiences. At the same time, it shows how the functional equivalent of abstract representations—prototypes, logogens

Distributed representations are attractive for a number of reasons. They offer the possibility of representing concepts in a continuous space, they degrade gracefully with noise, and they can be processed in a parallel network of simple processing elements. However, the problem of representing nested structure in distributed representations has been for some time a prominent concern of both proponents and critics of connectionism [Fodor and Pylyshyn 1988; Smolensky 1990; Hinton 1990]. The lack of connectionist representations for complex structure has held back progress in tackling higher-level cognitive tasks such as language understanding and reasoning. In this thesis I review connectionist representations and propose a method for the distributed representation of nested structure, which I call "Holographic Reduced Representations " (HRRs). HRRs provide an implementation of Hinton's [1990] "reduced descriptions". HRRs use circular convolution to associate atomic items, which are rep...

A computational model of human memory for serial order is described (OSCillator-based Associative Recall [OSCAR]). In the model, successive list items become associated to successive states of a dynamic learning-context signal. Retrieval involves reinstatement of the learning context, successive states of which cue successive recalls. The model provides an integrated account of both item memory and order memory and allows the hierarchical representation of temporal order information. The model accounts for a wide range of serial order memory data, including differential item and order memory, transposition gradients, item similarity effects, the effects of item lag and separation in judgments of relative and absolute recency, probed serial recall data, distinctiveness effects, grouping effects at various temporal resolutions, longer term memory for serial order, list length effects, and the effects of vocabulary size on serial recall. The serial ordering of behavior is central to much, perhaps most, of human cognition (e.g., Lashley, 1951). Studies of memory for serial order have provided rich data on the psychological repre-sentation of serial order information and therefore offer a signifi-cant challenge to any model of serially ordered behavior. In this

Back-propagation learning (Rumelhart, Hinton and Williams, 1986) is a useful research tool but it has a number of undesiderable features such as having the experimenter decide from outside what should be learned. We describe a number of simulations of neural networks that internally generate their own teaching input. The networks generate the teaching input by trasforming the network input through connection weights that are evolved using a form of genetic algorithm. What results is an innate (evolved) capacity not to behave efficiently in an environment but to learn to behave efficiently. The analysis of what these networks evolve to learn shows some interesting results.

Abstract not found

: Holographic Reduced Representations (HRRs) are a method for encoding nested relational structures in fixed width vector representations. HRRs encode relational structures as vector representations in such a way that the superficial similarity of the vectors reflects both superficial and structural similarity of the relational structures. HRRs also support a number of operations that could be very useful in psychological models of human analogy processing: fast estimation of superficial and structural similarity via a vector dot-product; finding corresponding objects in two structures; and chunking of vector representations. Although similarity assessment and discovery of corresponding objects both theoretically take exponential time to perform fully and accurately, with HRRs one can obtain approximate solutions in constant time. The accuracy of these operations with HRRs mirrors patterns of human performance on analog retrieval and processing tasks. Keywords: neural networks, distributed representations, binding, analogy, analog retrieval, structure, chunking, systematicity 1

C is to what? (b) is a task solved by the COPYCAT system (Hofstadter and Mitchell 1994). (c) is a task presented to human subjects in a PET scan study intended to identify regions of the brain involved in processing analogies. The task was to choose the arrangement analogous to the base arrangement. (Wharton, Grafman, Flitman, Hansen, Bruaner, Marks, and Honda 1998). (d) is a task presented to chimpanzees; the task was to place the symbol for \same" or \dierent" in the center of the gure, depending on whether the relationship between the pair of objects on the left is the same as the relationship between the pair of objects on the right (Oden, Thompson, and Premack 1998). 1. Introduction The ability to reason about knowledge itself is one of the hallmarks of higherlevel cognitive processing. The knowledge structures involved can range from simple associations or relations, e.g., as in the analogy tasks shown in Figure 1, to the rules of reasoning itself. Any theory