Simple Recurrent Networks (SRN) are Neural Network (connectionist) models able to process natural language. Phonotactics concerns the order of symbols in words. We continued an earlier unsuccessful trial to model the phonotactics of Dutch word corpus with SRNs. In order to overcome the previously reported obstacles, a new method for network testing was developed - optimal threshold evaluation. This method is based on minimising the erroneous character prediction of a trained SRN. The network training was improved as well. The training words were presented to the network according to their frequencies, which emphasises the more frequent sequences. The achieved results are promising and provide a base for further study. 1. Introduction to connectionist natural language processing. It is still a challenge to process natural language with connectionist paradigms. Formal language theory provides more natural methods for exploring complex language phenomena, but if we search for an approach...

Asymptotic behavior of a recurrent neural network changes qualitatively at certain points in the parameter space, which are known as "bifurcation points". At bifurcation points, the output of a network can change discontinuously with the change of parameters and therefore convergence of gradient descent algorithms is not guaranteed. Furthermore, learning equations used for error gradient estimation can be unstable. However, some kinds of bifurcations are inevitable in training a recurrent network as an automaton or an oscillator. Some of the factors underlying successful training of recurrent networks are investigated, such as choice of initial connections, choice of input patterns, teacher forcing, and truncated learning equations. 1 Introduction Recurrent neural networks are expected to have versatile capabilities for modeling and controlling dynamical systems. From the fact that multi-layer neural networks can approximate arbitrary mappings [13], it is easy to show that recurrent n...

Connectionist learning of natural language words and their phonetic regularities is presented. The Neural Network (NN) model we employ in this problem is the Simple Recurrent Network, trained with the Backpropagation Through Time (BPTT) learning algorithm. During the training, it was assigned the task of predicting the next phoneme given one phoneme at each moment and keeping information of the past phonemes from a given word in a few context neurons. The phonotactics of the Dutch language was studied among others. The shortcomings of some similar previous implementations are explained and successfully overcome. Among the techniques we employed to achieve the much-improved error rate of 1.1% with monosyllabic words and 3.5% with multisyllabic ones are new methods for network response interpretation, an evolutionary approach in training a set of networks, and the exploitation of the word frequencies in training. Finally, an analysis of the phonotactics rules extracted by a ...

The utilization of dynamic recurrent neural networks (DRNN) for the interpretation of biological signals coming from human brain and body has acquired a significant growth in the field of brain-machine interface. DRNN approaches may offer an ideal tool for the identification of input-output relationships in numerous types of neural-based signals, such