A parallel stochastic algorithm is investigated for error-descent learning and optimization in deterministic networks of arbitrary topology. No explicit information about internal network structure is needed. The method is based on the model-free distributed learning mechanism of Dembo and Kailath. A modified parameter update rule is proposed by which each individual parameter vector perturbation contributes a decrease in error. A substantially faster learning speed is hence allowed. Furthermore, the modified algorithm supports learning time-varying features in dynamical networks. We analyze the convergence and scaling properties of the algorithm, and present simulation results for dynamic trajectory learning in recurrent networks. 1 Background and Motivation We address general optimization tasks that require finding a set of constant parameter values p i that minimize a given error functional E(p). For supervised learning, the error functional consists of some quantitativ...

In this paper we present an expandable digital architecture that provides an efficient real time implementation platform for large neural networks. The architecture makes heavy use of the techniques of bit serial stochastic computing to carry out the large number of required parallel synaptic calculations. In this design all real valued quantities are encoded on to stochastic bit streams in which the `1' density is proportional to the given quantity. The actual digital circuitry is simple and highly regular thus allowing very efficient space usage of fine grained FPGAs. Another feature of the design is that the large number of weights required by a neural network are generated by circuitry tailored to each of their specific values, thus saving valuable cells. Whenever one of these values is required to change, the appropriate circuitry must be dynamically reconfigured. This may always be achieved in a fixed and minimum number of cells for a given bit stream resolution. 1 Introduction ...

A novel technique for the generation of high speed stochastic bit streams in which the `1' density is proportional to a given value. Bit streams of this type are particularly useful in bit serial stochastic computing systems, such as digital stochastic neural networks. The proposed circuitry is highly suitable for VLSI fabrication. INTRODUCTION The use of stochastic bit streams allows a dramatic simplification of the circuitry required to implement many devices [1], since the multiplication of two values may be performed by computing the bit-wise conjunction of two corresponding streams. The highly pipelined digital design described in this paper was developed for use with high speed digital stochastic neural networks, as described in [2, 3, 4]. The proposed stochastic bit stream generator is highly pipelined and thus potentially extremely fast. The individual pipeline stages, known as modulators are simple, and the number used defines the overall resolution of the generated bit strea...

We present a fast implementation of the Boltzmann Machine based on specialized hardware. This realization faithfully implements the machine; in particular, it avoids the excess parallelism which makes other fast implementations of it only approximate. The current prototype performs 505 million additions and multiplications (or megasynapses) per second. It can emulate the fully connected Boltzmann Machine with up to 1438 variables. Boltzmann Machine's weights are expressed as 16-bit two's complement fixed-point numbers. Our implementation is built on the top of DECperle-1, a reconfigurable coprocessor board based on field programmable gate arrays (FPGAs). DECperle-1 is closely coupled with its host processor (an ordinary workstation). In our application, it only performs the simplest and most computing-intensive part of the Boltzmann Machine algorithm, namely multiplying matrices of numbers by vectors of bits. The other operations (which are complicated, but only require a modest amoun...

We present experimental results on supervised learning of dynamical features in an analog VLSI neural network chip. The recurrent network, containing six continuous-time analog neurons and 42 free parameters (connection strengths and thresholds), is trained to generate time-varying outputs approximating given periodic signals presented to the network. The chip implements a stochastic perturbative algorithm, which observes the error gradient along random directions in the parameter space for error-descent learning. In addition to the integrated learning functions and the generation of pseudo-random perturbations, the chip provides for teacher forcing and long-term storage of the volatile parameters. The network learns a 1 kHz circular trajectory in 100 sec. The chip occupies 2mm \Theta 2mm in a 2¯m CMOS process, and dissipates 1:2 mW. 1 Introduction Exact gradient-descent algorithms for supervised learning in dynamic recurrent networks [1-3] are fairly complex and do not provide for a ...

This thesis develops an engineering practice and design methodology to enable us to use CMOS analog VLSI chips to perform more accurate and precise computation. These techniques form the basis of an approach that permits us to build computer graphics and neural network applications using analog VLSI. The nature of the design methodology focuses on defining goals for circuit behavior to be met as part of the design process. To increase the accuracy of analog computation, we develop techniques for creating compensated circuit building blocks, where compensation implies the cancellation of device variations, offsets, and nonlinearities. These compensated building blocks can be used as components in larger and more complex circuits, which can then also be compensated. To this end, we develop techniques for automatically determining appropriate parameters for circuits, using constrained optimization. We also fabricate circuits that implement multi-dimensional gradient estimation for a grad...

This chapter gives an overview of existing digital VLSI implementations and discusses techniques for implementing high performance, high capacity digital neural nets. It presents a set of techniques for estimating chip area, performance, and power consumption in the early stages of design to facilitate architectural exploration. It shows how technology scaling rules can be included in the estimation process. It presents a set of basic building blocks useful in implementing digital networks. It then uses the estimation techniques to predict capacity and performance of a variety of digital architectures. Finally, it discusses implementation strategies for very large networks. 1 Introduction Neural network applications suitable for implementation in VLSI cover a wide spectrum, from dedicated feedforward nets for real time feature detection to general purpose engines for exploring learning algorithms. The DARPA Neural Network Study [76] contains a good discussion of the range of applicati...

Introduction This chapter describes a methodology for designing digital VLSI neurochips which emphasizes area, power, and performance estimation to facilitate architectural exploration in the early stages of design. It first discusses some key aspects of mapping neural net algorithms onto VLSI architectures. It then introduces a set of circuit level building blocks commonly used in constructing digital nets. It discusses how to estimate chip area, performance, and power consumption in architectures constructed from these blocks, showing how to include technology scaling rules in the estimation process. It concludes with a detailed discussion of a CMOS implementation of a digital Boltzmann machine. 2 Mapping algorithms to architectures An algorithm is a set of tasks to be applied to data in a specified order to transform inputs and internal state to desired outputs. An architecture is a set of resources and interconnections. Mapping algorithms to architectures

This work argues that it should be possible to combine pulse-based VLSI techniques with the relatively simple training rules of the Helmholtz Machine stochastic neural architecture, in order to build an analogue probabilistic hardware model of the latter. An overview of the necessary components is presented, as well as a design for a pulse-width modulation oscillator, capable of transforming a current input (which represents the squashed, post-synaptic signal processed by a particular neuron) into the probability associated with the binary state of that neuron. A CMOS hardware prototype has been designed and fabricated, and precautions were taken during the design and simulation stages in order to prevent the oscillators on the same chip from locking together. Apart from testing the hardware prototype, future plans involve the hardware implementation of other modules, such as the synapse, the squashing function and weight changing circuitry.

This paper suggests that probabilistic VLSI architectures may provide insights into naturally-stochastic processes, and describes a particular application to perform robust data fusion, using unsupervised feature extraction and compensation of sensor drift. Two very interesting, stochastic networks are (1) the Helmholtz machine, which uses a local learning rule and whose hidden units choose states according to a probability distribution, rather than deterministically; and (2) the Product-of-Experts (PoE) machine, which uses the same computational elements, but has a training algorithm that is both more reliable and more amenable to on-chip implementation. Stochasticity can be provided in hardware by sampling an oscillator's binary output that varies its mark-to-period ratio, provided that the outputs of a set of oscillators remains uncorrelated. 1Overview 1.1 A novel computational style It is well known that biological neurons exhibit and use stochastic behaviour. Spontaneous...