Abstract. Training recurrent neural networks is hard. Recently it has however been discovered that it is possible to just construct a random recurrent topology, and only train a single linear readout layer. State-ofthe-art performance can easily be achieved with this setup, called Reservoir Computing. The idea can even be broadened by stating that any high dimensional, driven dynamic system, operated in the correct dynamic regime can be used as a temporal ‘kernel ’ which makes it possible to solve complex tasks using just linear post-processing techniques. This tutorial will give an overview of current research on theory, application and implementations of Reservoir Computing. 1

Standard generative linguistic theory, which uses discrete symbolic models of cognition, has some strengths and weaknesses. It is strong on providing a network of outposts that make scientific travel in the jungles of natural language feasible. It is weak in that it currently depends on the elaborate and unformalized use of intuition to develop critical supporting assumptions about each data point. In this regard, it is not in a position to characterize natural language systems in the lawful terms that ecological psychologists strive for. Connectionist learning models offer some help: They define lawful relations between linguistic environments and language systems. But our understanding of them is currently weak, especially when it comes to natural language syntax. Fortunately, symbolic linguistic analysis can help connectionism if the two meet via dynamical systems theory. I discuss a case in point: Insights from linguistic explorations of natural language syntax appear to have identified information structures that are particularly relevant to understanding ecologically appealing but analytically mysterious connectionist learning models. This article is concerned with the relation between discrete, symbolic systems of the

Standard pattern recognition provides effective and noise-tolerant tools for machine learning tasks; however, most approaches only deal with real vectors of a finite and fixed dimensionality. In this tutorial paper, we give an overview about extensions of pattern recognition towards non-standard data which are not contained in a finite dimensional space, such as strings, sequences, trees, graphs, or functions. Two major directions can be distinguished in the neural networks literature: models can be based on a similarity measure adapted to non-standard data, including kernel methods for structures as a very prominent approach, but also alternative metric based algorithms and functional networks; alternatively, non-standard data can be processed recursively within supervised and unsupervised recurrent and recursive networks and fully recurrent systems.

In recent years it has been shown that first order recurrent neural networks trained by gradient-descent can learn not only regular but also simple context-free and context-sensitive languages. However, the success rate was generally low and severe instability issues were encountered. The present study examines the hypothesis that a combination of evolutionary hill climbing with incremental learning and a well-balanced training set enables first order recurrent networks to reliably learn context-free and mildly context-sensitive languages. In particular, we trained the networks to predict symbols in string sequences of the context-sensitive language Preprint submitted to Neural Networks 10 January 2003 1}. Comparative experiments with and without incremental learning indicated that incremental learning can accelerate and facilitate training. Furthermore, incrementally trained networks generally resulted in monotonic trajectories in hidden unit activation space, while the trajectories of non-incrementally trained networks were oscillating. The non-incrementally trained networks were more likely to generalise.

Although for many years a sharp distinction has been made in language research between rules and words—with primary interest on rules—this distinction is now blurred in many theories. If anything, the focus of attention has shifted in recent years in favor of words. Results from many different areas of language research suggest that the lexicon is representationally rich, that it is the source of much productive behavior, and that lexically specific information plays a critical and early role in the interpretation of grammatical structure. But how much information can or should be placed in the lexicon? This is the question I address here. I review a set of studies whose results indicate that event knowledge plays a significant role in early stages of sentence processing and structural analysis. This poses a conundrum for traditional views of the lexicon. Either the lexicon must be expanded to include factors that do not plausibly seem to belong there; or else virtually all information about word meaning is removed, leaving the lexicon impoverished. I suggest a third alternative, which provides a way to account for lexical knowledge without a mental lexicon.

There is a need to clarify the relationship between traditional symbolic computation and neural network computation. We suggest that traditional context-free grammars are best understood as a special case of neural network computation; the special case derives its power from the presence of certain kinds of symmetries in the weight values. We describe a simple class of stochastic neural networks, Stochastic Linear Dynamical Automata (SLDAs), define Lyapunov Exponents for these networks, and show that they exhibit a significant range of dynamical behaviors—contractive and chaotic, with context free grammars at the boundary between these regimes. Placing context-free languages in this more general context has allowed us, in previous work, to make headway on the challenging problem of designing neural mechanisms that can learn them.

One of the fundamental challenges to using recurrent neural net-works (RNNs) for learning natural languages is the difficulty they have with learning complex temporal dependencies in symbol sequences. Re-cently, substantial headway has been made in training RNNs to process languages which can be handled by the use of one or more symbol-counting stacks (e.g., a n b n, a n A m B m b n, a n b n c n) with compelling cases made that the networks are solving the problem in principle, not simply using finite-state methods to approximate the training data—e.g., (Gers and Schmidhuber, 2001; Bodén and Wiles, 2002). Success on cases that require stacks to function as full-fledged sequence memories (e.g., palin-drome languages), has not been as great. On the other hand, (Moore, 1998), (Siegelmann, 1996), and (Tabor, 2000) describe methods of us-ing fractal sets to keep track of arbitrary stack computations in neural units with bounded, rational-valued activation values. The present pa-per combines these hand-wiring methods with the work on learning by describing Fractal Learning Neural Networks (FLNNs), which discover such fractal encodings via hill climbing in weight space. Simulations show that FLNNs can learn several context free languages that cannot be processed by a finite set of input-symbol counters. 1

On behalf of:

Dominant theories of implicit learning assume that implicit learning merely involves the learning of chunks of adjacent elements in a sequence. In the experiments presented here, participants implicitly learned a nonlocal rule, thus suggesting that implicit learning can go beyond the learning of chunks. Participants were exposed to a set of musical tunes that were all generated using a diatonic inversion. In the subsequent test phase, participants either classified test tunes as obeying a rule (direct test) or rated their liking for the tunes (indirect test). Both the direct and indirect tests were sensitive to knowledge of chunks. However, only the indirect test was sensitive to knowledge of the inversion rule. Furthermore, the indirect test was overall significantly more sensitive than the direct test, thus suggesting that knowledge of the inversion rule was below an objective threshold of awareness.

Simple recurrent networks (SRNs) in symbolic time series prediction (e. g. language processing models) are frequently trained with gradient descent based learning algorithms, notably with variants of backpropagation (BP). A major drawback for the cognitive plausibility of BP is that it is a supervised scheme in which a teacher has to provide a fully specified target answer. Yet, agents in natural environments often receive a summary feedback about the degree of success or failure only, a view adopted in reinforcement learning schemes. In this work we show that for SRNs in prediction tasks for which there is a probability interpretation of the network’s output vector, Elman BP can be reimplemented as a reinforcement learning (RL) scheme for which the expected weight updates agree with the ones from traditional Elman BP. Network simulations on formal languages corroborate this result and show that the learning behaviours of Elman backpropagation (BP) and its reinforcement variant are very similar also in online learning tasks.