This paper describes NETtalk, a class of massively-parallel network systems that learn to convert English text to speech. The memory representations for pronunciations are learned by practice and are shared among many processing units. The performance of NETtalk has some similarities with observed human performance. (i) The learning follows a power law. (;i) The more words the network learns, the better it is at generalizing and correctly pronouncing new words, (iii) The performance of the network degrades very slowly as connections in the network are damaged: no single link or processing unit is essential. (iv) Relearning after damage is much faster than learning during the original training. (v) Distributed or spaced practice is more effective for long-term retention than massed practice. Network models can be constructed that have the same performance and learning characteristics on a particular task, but differ completely at the levels of synaptic strengths and single-unit responses. However, hierarchical clustering techniques applied to NETtalk reveal that these different networks have similar internal representations of letter-to-sound correspondences within groups of processing units. This suggests that invariant internal representations may be found in assemblies of neurons intermediate in size between highly localized and completely distributed representations.

Many researchers interested in connectionist models accept that such models are "neurally inspired " but do not worry too much about whether their models are biologically realistic. While such a position may be perfectly justifiable, the present paper attempts to illustrate how biological information can be used to constrain connectionist models. Two particular areas are discussed. The first section deals with visual information processing in the primate and human visual system. It is argued that speed with which visual information is processed imposes major constraints on the architecture and operation of the visual system. In particular, it seems that a great deal of processing must depend on a single bottum-up pass. The second section deals with biological aspects of learning algorithms. It is argued that although there is good evidence for certain coactivation related synaptic modification schemes, other learning mechanisms, including back-propagation, are not currently supported by experimental data.

Many different algorithms and objective functions for topographic mappings have been proposed. We show that several of these approaches can be seen as particular cases of a more general objective function. Consideration of a very simple mapping problem reveals large differences in the form of the map that each particular case favors. These differences have important consequences for the practical application of topographic mapping methods.

When recognizing patterns or objects, our visual system can easily separate what kind of pattern is seen and where (location and orientation) it is seen. Neural networks as pattern recognizers can deal well with noisy input patterns, but have difficulties when confronted with the large variety of possible geometric transformations of an object. We propose a flexible neural mechanism for invariant recognition based on correlated neuronal activity and the self-organization of dynamic links. The system can deal in parallel with different kinds of invariances such as translation, rotation, mirror-reflection and distortion. It is shown analytically that parts of the neuronal activity equations can be replaced by a faster, but functionally equivalent, algorithmic approach. We derive a measure based on the correlation of activity which allows an unsupervised decision of whether a given input pattern matches with a stored model pattern ("what"-part). At the same time, the dynamic links specify...

xi 1 Introduction 1 1.1 Problem Statement : : : : : : : : : : : : : : : : : : : : : : : : : 6 1.2 The Neuron : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7 1.2.1 Biological Model : : : : : : : : : : : : : : : : : : : : : : 7 1.2.2 Computational Model : : : : : : : : : : : : : : : : : : : 9 1.3 Implementation Technologies : : : : : : : : : : : : : : : : : : : : 11 1.4 State of the Art : : : : : : : : : : : : : : : : : : : : : : : : : : : 14 1.5 Summary of Results : : : : : : : : : : : : : : : : : : : : : : : : 16 2 Implementation of Neural Models with Static Links 19 2.1 ANN Models with Static Links : : : : : : : : : : : : : : : : : : 20 2.1.1 The Hopfield Model : : : : : : : : : : : : : : : : : : : : : 21 2.1.2 The Perceptron Model : : : : : : : : : : : : : : : : : : : 23 2.1.3 The Multi-Layer Model : : : : : : : : : : : : : : : : : : : 24 2.2 Basic Computational Requirements : : : : : : : : : : : : : : : : 25 2.2.1 Search Phase Computations : : : : : : : : : : : : : : ...

Interpretation is the process whereby a hearer reasons to an interpretation of a speaker's discourse. The hearer normally adopts a credulous attitude to the discourse, at least for the purposes of interpreting it. That is to say the hearer tries to accommodate the truth of all the speaker's utterances in deriving an intended model. We present a non-monotonic logical model of this process which defines unique minimal preferred models and so e#ciently simulates a kind of closed-world reasoning of particular interest for human cognition.

We present an algorithm to solve the graph isomorphism problem for the purpose of object recognition. Objects, such as those that exist in a robot workspace, may be represented by labeled graphs (graphs with attributes on their nodes and/or edges). Object recognition, thereafter, is achieved by matching pairs of these graphs. Assuming that all objects are sufficiently different that their corresponding representative graphs are distinct, then given a new graph, the algorithm efficiently finds the isomorphic stored graph (if it exists). The algorithm consists of three phases: preprocessing, link construction, and ambiguity resolution. Results from experiments on a wide variety and sizes of graphs are reported. Results are also reported for experiments on recognizing graphs that represent protein molecules. The algorithm works for all types of graphs except for a class of highly ambiguous graphs that includes strongly regular graphs. However, members of this class are detected in polynom...

Several factors may interact to determine the periodicity of ocular dominance stripes in cat and monkey visual cortex. Previous theoretical work has suggested roles for the width of cortical interactions and the strength of between-eye correlations. Here, a model based on an explicit optimization is presented that allows a thorough characterization of how these and other parameters of the afferent input could affect ocular dominance stripe periodicity. The principle conclusions are that increasing the width of within-eye correlations leads to wider columns, and, surprisingly, that increasing the width of cortical interactions can sometimes lead to narrower columns. 1 Introduction In cats, monkeys and humans, layer 4 of the primary visual cortex (V1) is divided up into alternating regions dominated by input from the left and right eyes (e.g. Hubel & Wiesel (1977)). These regions segregate from a spatially uniform pattern during development (Rakic, 1976; LeVay et al, 1978). A characte...

During a critical period of postnatal development of the mammalian visual cortex, synaptic connections are susceptible to use-dependent modifications. Synaptic connections strengthen if pre- and postsynaptic elements are active simultaneously and postsynaptic depolarization is sufficient to allow for the activation of A'-methyl-D-aspartate (NMDA)-receptor-gated conductances. By contrast, synaptic gain decreases if postsynaptic activation exceeds a critical threshold and presynaptic afferents are not capable of activating NMDAreceptor-dependent conductances. These processes lead to selective stabilization of connections between neuronal elements which often exhibit correlated activity and thus modify connectivity according to functional criteria. It is suggested that such experience-dependent selection of circuits serves different purposes at different levels of visual processing. At the input stage to the striate cortex it contributes to optimize the match between the representations of the two eyes. At a later stage of processing it participates in the development of selective connections between cortical columns and thereby serves to establish neuronal

This paper describes an investigation of a recurrent artificial neural network which uses association to build transform-invariant representations. The simulation implements the analytic model of Parga and Rolls [16] which defines multiple (e.g. "view") patterns to be within the basin of attraction of a shared (e.g. "object") representation. First, it was shown that the network could store and correctly retrieve an "object" representation from any one of the views which define that object, with capacity as predicted analytically. Second, new results extended the analysis by showing that correct object retrieval could occur where retrieval cues were distorted; where there was some association between the views of different objects; and where connectivity was diluted, even when this dilution was asymmetric. The simulations also extended the analysis by showing that the system could work well with sparse patterns; and showing how pattern sparseness interacts with the number of views of each object (as a result of the statistical properties of the pattern coding) to give predictable object retrieval performance. The results thus usefully extend a recurrent model of invariant pattern recognition. Key words: Object Recognition; Invariance; Recurrent Networks; Attractor Networks; Associative Learning; Sparse Coding; Invariant Visual Representations. Preprint submitted to Elsevier Preprint 16 September 1999 1