We had previously shown that regularization principles lead to approximation schemes which are equivalent to networks with one layer of hidden units, called Regularization Networks. In particular, standard smoothness functionals lead to a subclass of regularization networks, the well known Radial Basis Functions approximation schemes. This paper shows that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models and some of the neural networks. In particular, we introduce new classes of smoothness functionals that lead to different classes of basis functions. Additive splines as well as some tensor product splines can be obtained from appropriate classes of smoothness functionals. Furthermore, the same generalization that extends Radial Basis Functions (RBF) to Hyper Basis Functions (HBF) also leads from additive models to ridge approximation models, containing as special cases Breiman's hinge functions, som...

A nonlinear black box structure for a dynamical system is a model structure that is prepared to describe virtually any nonlinear dynamics. There has been considerable recent interest in this area with structures based on neural networks, radial basis networks, wavelet networks, hinging hyperplanes, as well as wavelet transform based methods and models based on fuzzy sets and fuzzy rules. This paper describes all these approaches in a common framework, from a user's perspective. It focuses on what are the common features in the different approaches, the choices that have to be made and what considerations are relevant for a successful system identification application of these techniques. It is pointed out that the nonlinear structures can be seen as a concatenation of a mapping from observed data to a regression vector and a nonlinear mapping from the regressor space to the output space. These mappings are discussed separately. The latter mapping is usually formed as a basis function e...

This paper examines and compares several different approaches to the design of intelligent systems for diagnosis applications. These include expert systems (or knowledgebased systems), truth (or reason) maintenance systems, case-based reasoning systems, and inductive approaches like decision trees, artificial neural networks (or connectionist systems), and statistical pattern classification systems. Each of these approaches is demonstrated through the design of a system for a simple automobile fault diagnosis task. The paper also discusses the domain characteristics and design and performance requirements that influence the choice of a specific technique (or a combination of techniques) for a given application. Keywords: Intelligent Diagnosis, Expert Systems, Model-Based Systems CaseBased Reasoning, Neural Networks, Decision Trees, Knowledge Acquisition 1 INTRODUCTION The last few decades have seen a proliferation of intelligent systems for diagnosis, advising, and related applicat...

. Neural Networks are non-linear black-box model structures, to be used with conventional parameter estimation methods. They have good general approximation capabilities for reasonable non-linear systems. When estimating the parameters in these structures, there is also good adaptability to concentrate on those parameters that have the most importance for the particular data set. Key Words. Neural Networks, Parameter estimation, Model Structures, Non-Linear Systems. 1. EXECUTIVE SUMMARY 1.1. Purpose The purpose of this tutorial is to explain how Artificial Neural Networks (NN) can be used to solve problems in System Identification, to focus on some key problems and algorithmic questions for this, as well as to point to the relationships with more traditional estimation techniques. We also try to remove some of the "mystique" that sometimes has accompanied the Neural Network approach. 1.2. What's the problem? The identification problem is to infer relationships between past inp...

A new class of neural networks are proposed for the dynamic classification of spatio-temporal signals. These networks are designed to classify signals of different durations, taking into account correlations among different signal segments. Such networks are applicable to SONAR and speech signal classification problems, among others. Network parameters are adapted based on the biologically observed habituation mechanism. This allows the storage of contextual information, without a substantial increase in network complexity. We introduce the concept of a complete memory. We then prove mathematically that a network with a complete memory temporal encoding stage followed by a sufficiently powerful feedforward network is capable of approximating arbitrarily well any continuous, causal, time-invariant discrete-time system with a uniformly bounded input domain. The memory mechanisms of the habituation based networks are complete memories, and involve nonlinear transformations of the...

If behavior networks, which use spreading activation to select actions, are analogous to connectionist methods of pattern recognition, then recurrent behavior networks, which use energy minimization, are analogous to Hopfield networks. Hopfield networks memorize patterns by making them attractors. Similarly, each behavior of a recurrent behavior network should be an attractor of the network, to inhibit fruitless, repeated switching between different behaviors in response to small changes in the environment and in motivations. I overcome two major objections to this view, and demonstrate that the performance in a test domain of the Do the Right Thing recurrent behavior network is improved by redesigning it to create desirable attractors and basins of attraction. I further show that this performance increase is correlated with an increase in persistence and a decrease in undesirable behavior-switching. On a more general level, this work encourages the study of action selection as a dynam...

This paper proposes a two layer modular neural system. The basic building blocks of the architecture are multilayer Perceptrons trained with the Backpropagation algorithm. Due to the proposed modular architecture the number of weight connections is less than in a fully connected multilayer Perceptron. The modular network is designed to combine two different approaches of generalization known from connectionist and logical neural networks; this enhances the generalization abilities of the network. The architecture introduced here is especially useful in solving problems with a large number of input attributes. 1 Introduction The multilayer Perceptron (MLP) trained by the Backpropagation (BP) algorithm has been used to solve real-world problems in prediction, recognition, and optimization. If the input dimension is small the network can be trained very quickly. However for large input spaces the performance of the BP algorithm decreases [3]. In many cases it becomes difficult to find a ...

This memory can take several forms based on the time scales at which such modifications are allowed. Some symbol structures might have the property of determining choice and the order of application of transformations to be applied on other symbol structures. These are essentially the programs. Programs when executed --- typically through the conventional process of compilation and interpretation and eventually --- when they operate on symbols that are linked through grounding to particular effectors --- produce behavior. Working memory holds symbol structures as they are being processed. Long-term memory, generally speaking, is the repository of programs and can be changed by addition, deletion, or modification of symbol structures that it holds. Such a system can compute any Turing-computable function provided it has sufficiently large memory and its primitive set of transformations are adequate for the composition of arbitrarily symbol structures (programs) and the interpreter is capable of interpreting any possible symbol structure. This also means that any particular set of symbolic processes can be carried out by an NANN --- provided it has potentially infinite memory, or finds a way to use its transducers and effectors to use the external physical environment to serve as its memory). 14 Chapter 12 Knowledge in SAI systems is typically embedded in complex symbol structures such as lists (Norvig, 1992), logical databases (Genesereth and Nilsson, 1987), semantic networks (Quillian, 1968), frames (Minsky, 1975), schemas (Arbib, 1972; 1994), and manipulated by (often serial) procedures or inferences (e.g., list processing, application of production rules (Waterman, 1985), or execution of logic programs (Kowalski, 1977) carried out by a central processor that accesse...

1 An understanding of learning -- the process by which a learner acquires and refines a broad range of knowledge and skills -- is central to the enterprise of building truly adaptive, flexible, robust, and creative intelligent systems. Significant theoretical and empirical contributions to the characterization of learning in computational terms have emerged from research in a number of disparate research paradigms. The limitations of individual paradigms and of particular classes of techniques within each paradigm are beginning to be recognized. Converging lines of evidence from multiple sources, both theoretical as well as empirical, suggest that artificial intelligence systems, in order to be able to deal with complex tasks such as recognizing and describing 3-dimensional objects, or communicating in natural language, must be able to effectively utilize a range of learning algorithms operating with an adequate repertoire of representational structures. This paper draws on a broad ran...

Abstract not found