Parallelism and distribution have been considered the key features of neural processing. The term parallel distributed processing is even used as a synonym for artificial neural networks. Nevertheless, the actual implementations are still in search of the appropriate model to "naturally represent" neural computing. And the final judgement is always given in performance figures -- keeping the parallelization issue high on the neurosimulation agenda. Two approaches have yielded the best results: parallel simulations on general-purpose computers, and specially developed neurohardware. Programming neural networks on parallel machines requires high-level techniques reflecting both inherent features of neuromodels and characteristics of the underlying computers. On the other hand, emulation of the neuroparadigm requires that the functioning of neural operations be mimicked directly by the hardware. Both approaches are presented, and their advantages and shortcomings are outlined.

Computer vision algorithms are natural candidates for high performance computing due to their inherent parallelism and intense computational demands. For example, a simple 3 x 3 convolution on a 512 x 512 gray scale image at 30 frames per second requires 67.5 million multiplications and 60 million additions to be performed in one second. Computer vision tasks can be classified into three categories based on their computational complexity andcommunication complexity: low-level, intermediate-level and high-level. Special-purpose hardware provides better performance compared to a general-purpose hardware for all the three levels of vision tasks. With recent advances in very large scale integration (VLSI) technology, an application specific integrated circuit (ASIC) can provide the best performance in terms of total execution time. However, long design cycle time, high development cost and inflexibility of a dedicated hardware deter design of ASICs. In contrast, field programmable gate arrays (FPGAs) support lower design verification time and easier design adaptability atalower cost. Hence, FPGAs with an array of reconfigurable logic blocks canbevery useful compute elements. FPGA-based custom computing machines are

As artificial neural networks (ANNs) gain popularity in a variety of application domains, it is critical that these models run fast and generate results in real time. Although a number of implementations of neural networks are available on sequential machines, most of these implementations require an inordinate amount of time to train or run ANNs, especially when the ANN models are large. One approach for speeding up the implementation of ANNs is to implement them on parallel machines. This paper surveys the area of parallel environments for the implementations of ANNs, and prescribes desired characteristics to look for in such implementations. 1 Introduction Although traditional von Neumann computing has been successful in many applications, it has not proved effective in solving a variety of important complex problems. At the same time, it has been observed that human beings solve these problems routinely in real time. Typical problems that fall into this class consist of perception...

Efficient knowledge representation and reasoning is an important component of intelligent activity, and is a crucial aspect in the design of large-scale intelligent systems. This dissertation explores the design, analysis, and implementation of massively parallel knowledge representation and reasoning systems which can encode very large knowledge bases and respond to a class of queries in real-time, with reasoning episodes expected to span a fraction of a second. The dissertation attempts to design efficient, large-scale knowledge base systems by: (i) exploiting massive parallelism; and (ii) constraining representational and inferential capabilities to achieve tractability, while still retaining sufficient expressive power to capture a broad class of reasoning in intelligent systems. To this end, shruti, a connectionist reasoning system which models reflexive--- i.e., effortless and spontaneous---reasoning serves as the knowledge representation and reasoning framework. Shruti-based mas...

We describe how complex systems of multiple BCPNN (Bayesian Confidence Propagating Neural Networks) networks are modeled, implemented and run on parallel cluster computers. The BCPNN system is modeled in terms of populations and projections. Hypercolumns and spiking activity is used to get a scalable and efficient implementation. Three communication protocols are evaluated; MPICH, TCP/IP and UDP. Fully recurrent BCPNN:s with up to 6·10 4 units that allocate more than 34 GB of memory are run. The results are that UDP provides a scalable and efficient communication protocol. The BCPNN with hypercolumns is well suited and scales well on cluster computers.

this paper we describe the guideline of Daphne, a parallel simulator for supervised recurrent neural networks trained by Backpropagation through time. The simulator has a modular structure, based on a parallel training kernel running on the CM-2 Connection Machine. The training kernel is written in CM Fortran in order to exploit some advantages of the slicewise execution model. The other modules are written in serial C code. They are used for designing and testing the network, and for interfacing with the training data. A dedicated language is available for defining the network architecture, which allows the use of linked modules. The implementation of the learning procedures is based on training example parallelism. This dimension of parallelism has been found to be effective for learning static patterns using feedforward networks. We extend training example parallelism for learning sequences with full recurrent networks. Daphne is mainly conceived for applications in the field of Automatic Speech Recognition, though it can also serve for simulating feedforward networks.

Specification and Parallel Simulation of Neural Networks Alfred Strey Department of Neural Information Processing University of Ulm, Oberer Eselsberg, D-89069 Ulm, Germany Abstract: In this article the neural network specification language EpsiloNN is presented. From an abstract specification that is independent of the target computer architecture, a simulation source program for a workstation or a parallel computer can be generated. Neurocomputers requiring fixed-point data types and arithmetic are supported too. The language design is based on an unified neural network model and allows an object-oriented specification of synapses, neurons and networks. The optimal mapping of a neural network onto a parallel computer can be determined automatically and an efficient parallel simulation source code can be obtained from an EpsiloNN specification by program transformations. A complete specification of a radial basis function network is given as an example and the methodology for gene...

National University.

: The paper deals with the Sparse distributed memory (SDM). The SDM can be seen either as an extension of a RAM or as a neural network and it may be used either as an autoassociative or as a heteroassociative memory. The basic principle, topology, data storing and retrieving, and some practical experience with simple program models of the SDM are described here. Keywords: Neural Networks, Associative Memories, Sparse Distributed Memory. 1 Introduction Associative memory is a memory that can recall data when a reference address is sufficiently close (not only exact equal as in random-access memories) to the address at which the data were stored. It is very useful if the reference address is corrupted by random noise or outright errors or if this address is only partially specified. Suppose L pairs of vectors f(~x 1 ; ~y 1 ); (~x 2 ; ~y 2 ); :::; (~x L ; ~y L )g; where ~x i fflR n x j = f0; 1g ~y i fflR m y j = f0; 1g Three types of associative memories can be distinguished: 1. ...

With the recent development of massively parallel computing, extremely large amounts of processing power and memory capacity are available for the analysis of complex data sets. At the same time, the complexity and size of these data sets has been increasing. Both of these trends are expected to continue for the foreseeable future. This paper will provide a general overview of massively parallel architectures and algorithms for the analysis of time series data. Two distinct approaches to this problem, computational and memory-based, will be described. 1 Introduction The last decade has seen a revolution in large scale computation. The massively parallel processing (MPP) paradigm, originally seen as an outsider in the supercomputer race, is widely recognized as the technology of the future. In this paper we will discuss a number of approaches to time series data analysis using massively parallel computers. We will first review some of the current levels and trends in MPP technology and...