Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradientbased learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of two dimensional (2-D) shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation, recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN’s), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank check is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day.

The convergence of online learning algorithms is analyzed using the tools of the stochastic approximation theory, and proved under very weak conditions. A general framework for online learning algorithms is first presented. This framework encompasses the most common online learning algorithms in use today, as illustrated by several examples. The stochastic approximation theory then provides general results describing the convergence of all these learning algorithms at once.

As artificial neural networks (ANNs) gain popularity in a variety of application domains, it is critical that these models run fast and generate results in real time. Although a number of implementations of neural networks are available on sequential machines, most of these implementations require an inordinate amount of time to train or run ANNs, especially when the ANN models are large. One approach for speeding up the implementation of ANNs is to implement them on parallel machines. This paper surveys the area of parallel environments for the implementations of ANNs, and prescribes desired characteristics to look for in such implementations. 1 Introduction Although traditional von Neumann computing has been successful in many applications, it has not proved effective in solving a variety of important complex problems. At the same time, it has been observed that human beings solve these problems routinely in real time. Typical problems that fall into this class consist of perception...

Neural networks are non-linear static or dynamical systems that learn to solve problems from examples. Those learning algorithms that require a lot of computing power could benefit from fast dedicated hardware. This paper presents an overview of digital systems to implement neural networks. We consider three options for implementing neural networks in digital systems: serial computers, parallel systems with standard digital components, and parallel systems with special-purpose digital devices. We describe many examples under each option, with an emphasis on commercially available systems. We discuss the trend toward more general architectures, we mention a few hybrid and analog systems that can complement digital systems, and we try to answer questions that came to our minds as prospective users of these systems. We conclude that support software and in general, system integration, is beginning to reach the level of versatility that many researchers will require. The next step appears ...

: Artificial neural networks have been successfully used in many technical applications. They are also important tools for the control of autonomous agents. The major goal of research on autonomous agents is to study intelligence as the result of a system environment interaction, rather than understanding intelligence on a computational level. In contrast to other applications, autonomous agents might not distinguish between a learning and a performance phase; they have to continuously learn while they are behaving in their environment. Thus, a neural network for autonomous agents should feature incremental learning, should not exhibit overlearning and should not suffer from a high VC dimension. The review presented in this paper reveals that most existing models are not ideally suited for autonomous agents. The main goals of this paper are (1) to discuss the autonomous agents perspective, (2) to identify important properties of neural networks for autonomous agents, and (3), very impo...

Abstract:- Many soft computing applications have inherent parallelism in the form of synapses and neurons. Exploitation of parallelism will bring computers more close to their biological counterparts as well as improve the performance. Modularity and good communication capabilities are essential for practical implementations. Our multiprocessor system called PARNEU is designed according to general requirements found in artificial neural networks. PARNEU is a parallel co-processor system, which includes a bus, ring and reconfigurable partial tree communication networks. Programming is done using C-language primitives, which hide the communication topology. PARNEU is directly connected to the host computer, which includes a TCP/IP server for remote programming over Internet connection. Scalability of communication topology and artificial neural network applications, like Multilayer Perceptron (MLP), Self-Organizing Map (SOM) and Sparse Distributed Memory (SDM) are shown. Parallel efficiency with eight boards configuration is generally more than 75%. Keywords:- Neurocomputer, scalable parallel implementation, partial tree shape architecture, artificial neural networks, soft computing, hardware implementation. 1

Abstract. This contribution presents an overview of the theoretical and practical aspects of the broad family of learning algorithms based on