Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradientbased learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of two dimensional (2-D) shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation, recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN’s), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank check is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day.

is network has a 20#20 pixel input #eld and ten outputs for the classi#cation result. The #rst three layers, which contain 97 # of the connections and thus constitute the main computational load, have been implemented on a single ANNA chip. The preprocessing #size normalization and deskewing# as well as the last two layers #which require higher precision# are evaluated on a digital signal processor #DSP#. The micro code for the ANNA chip is produced automatically from a network description and tested with a register level simulator. The weightvalues are taken from the original back-propagation network and quantized to the chip's resolution. A subsequent retraining phase compensates for the low resolution of the chip resulting in a recognition performance which is similar to that of the original network. The ANNA chip is capable of processing more than 1,000 characters per second. It operates at a considerably higher rate than the other components of the recognition system

In this paper we will extend the transconductancemode (T-mode) approach [1] to implement analog continuanstime neural network hardware systems to include on-chip Hebbian learning and on-chip analog weight storage capability. The demonstration vehicle used is a 5+5 neurons bidirectional associative memory (BAM) prototype fabricated in a standard 2-tm double-metal double-polysilicon CMOS process (through and thanks to MOSIS). Mismatches and nonidealities in learning neural hardware are supposed not to be critical if on-chip learning is available, because they will be implicitly compensated. However, mismatches in the learning circuits themselves cannot always be compensated. This mismatch is specially important if the learning circuits use transistors operating in weak inversion. In this paper we will estimate the expected mismatch between learning circuits in the BAM network prototype and evaluate its effect on the learning performance, using theoretical computations and Monte Carlo Hspice simulations. Afterwards we will verify these theoretical predictions with the experimentally measured results on the test vehicle prototype.

A mixed analog#digital chip #ANNA# for fast 2-d convolution and matrix-vector multiplication has been developed#peak speed 20,000 MOPS#. Two of these chips have been integrated on a 6U VME board, to serve as a high-speed platform for a wide variety of algorithms used in neural-network applications as well as in image analysis. The system has been tested for such tasks as character recognition, noise removal, and text location as well as for emulating cellular neural networks #CNN#. A sustainedspeed of up to 2 billion connections per second #GC#s# and a recognition speed of 1000 characters per second with a sophisticated neural network has been measured. 1 Introduction Many neural-network chips for high-speed processing have been built and operated in test settings. To take full advantage of the speed of these chips, they must be integrated into a system that can support their speed, in particular can manage the overwhelming data #ow to and from the chip. The importance of such system...

Two ANNA neural-network chips are integrated on a 6U VME board, to serve as a high-speed platform for a wide variety of algorithms used in neural-network applications as well as in image analysis. The system can implement neural networks of variable sizes and architectures, but can also be used for #ltering and feature extraction tasks that are based on convolutions. The board contains a controller implemented with #eld programmable gate arrays #FPGA#, memory and bus interfaces, all designed to support the high compute power of the ANNA chips. Compared to a previous evaluation board #1#, this new system is designed for maximum speed and is roughly 10 times faster than the previous board. The system has been tested for such tasks as text location, character recognition, and noise removal as well as for emulating cellular neural networks #CNN#. A sustained speed of up to 2 billion connections per second #GC#s# and a recognition speed of 1000 characters per second has been measured. Keyw...

Abstract — Intelligent tasks, such as visual perception, auditory perception, and language understanding require the construction of good internal representations of the world (or ”features”), which must be invariant to irrelevant variations of the input while, preserving relevant information. A major question for Machine Learning is how to learn such good features automatically. Convolutional Networks (ConvNets) are a biologicallyinspired trainable architecture that can learn invariant features. Each stage in a ConvNets is composed of a filter bank, some non-linearities, and feature pooling layers. With multiple stages, a ConvNet can learn multi-level hierarchies of features. While ConvNets have been successfully deployed in many commercial applications from OCR to video surveillance, they require large amounts of labeled training samples. We describe new unsupervised learning algorithms, and new non-linear stages that allow ConvNets to be trained with very few labeled samples. Applications to visual object recognition and vision navigation for off-road mobile robots are described.

Rapid improvements in hardware and algorithms are reshaping the technological basis of postal address recognition. An increasingly important theme is automation of the engineering process itself, through trainable classifiers, realistic distortion models, and statistical contextual analysis. Relevant basic research at AT&T Bell Laboratories includes VLSI neural networks, algorithmic pattern recognition, computational linguistics, and artificial intelligence. Interdisciplinary application of these has stimulated improvements in handwritten ZIP code recognition, machine-print address recognition, and address block location. 1. Introduction The technological basis of postal address recognition is rapidly evolving due to progress in several research disciplines, including VLSI neural networks, algorithmic pattern recognition, computational linguistics, and artificial intelligence. We believe that a unifying theme is emerging: an increasing emphasis on automation of the engineering process...

Introduction AtAT&T Bell Laboratories wehave developed two neural-network chips: #i# the NET32K chip #1# with 32,768 1-bit processing elements whichiswell suited for image-analysis tasks such as feature extraction and #ii# the ANNA chip #2# with 4,096 3#6-bit processing elements designed for high-speed character recognition using our neural-net based optical character recognition #OCR# algorithms #3, 4##. Board systems for both chips have been built and are used in prototype systems. A new chip, called HIP for Homogeneous Image Processor, is now under development. This chip leverages the experience gained with the earlier chips, but is targeted at a larger range of applications. Besides neural networks, image analysis, and OCR, additional applications such as video coding#decoding, speech recognition, and graphics can take advantage of the new chip's huge computational power of 25 billion operations per second. The high #exibility and the large set of applications can potentia

Abstract. Hierarchical convolutional neural networks are a well-known robust image-recognition model. In order to apply this model to robot vision or various intelligent vision systems, its VLSI implementation with high performance and low power consumption is required. This paper proposes a convolutional network VLSI architecture using a hybrid approach composed of pulse-width modulation (PWM) and digital circuits. We call this approach merged/mixed analog-digital architecture. The VLSI includes PWM neuron circuits, PWM/digital converters, digital adder-subtracters, and digital memory. We have designed and fabricated a VLSI chip by using a 0.35 �m CMOS process. The VLSI chip can perform 6-bit precision convolution calculations for an image of 100¢100 pixels with a receptive field area of up to 20¢20 pixels within 5 ms, which means a performance of 2 GOPS. Power consumption of PWM neuron circuits is estimated to be 20 mW. We have verified successful operations using a fabricated VLSI chip. 1

In recent years, computational biologists have shown through simulation that small neural networks with fixed connectivity are capable of producing multiple output rhythms in response to transient inputs. It is believed that such networks may play a key role in certain biological behaviors such as dynamic gait control. In this paper, we present a novel method for designing continuous-time recurrent neural networks (CTRNNs) that contain multiple embedded limit cycles, and we show that it is possible to switch the networks between these embedded limit cycles with simple transient inputs. We also describe the design and testing of a fully integrated four-neuron CTRNN chip that is used to implement the neural network pattern generators. We provide two example multipattern generators and show that the measured waveforms from the chip agree well with numerical simulations.