Belief networks are directed acyclic graphs in which the nodes represent propositions (or variables), the arcs signify direct dependencies between the linked propositions, and the strengths of these dependencies are quantified by conditional probabilities. A network of this sort can be used to represent the generic knowledge of a domain expert, and it turns into a computational architecture if the links are used not merely for storing factual knowledge but also for directing and activating the data flow in the computations which manipulate this knowledge. The first part of the paper deals with the task of fusing and propagating the impacts of new information through the networks in such a way that, when equilibrium is reached, each proposition will be assigned a measure of belief consistent with the axioms of probability theory. It is shown that if the network is singly connected (e.g. tree-structured), then probabilities can be updated by local propagation in an isomorphic network of parallel and autonomous processors and that the impact of new information can be imparted to all propositions in time proportional to the longest path in the network. The second part of the paper deals with the problem of finding a tree-structured representation for a collection of probabilistically coupled propositions using auxiliary (dummy) variables, colloquially called "hidden causes. " It is shown that if such a tree-structured representation exists, then it is possible to uniquely uncover the topology of the tree by observing pairwise dependencies among the available propositions (i.e., the leaves of the tree). The entire tree structure, including the strengths of all internal relationships, can be reconstructed in time proportional to n log n, where n is the number of leaves.

We describe a distributed model of information processing and memory and apply it to the representation of general and specific information. The model consists of a large number of simple processing elements which send excitatory and inhibitory signals to each other via modifiable connections. Information processing is thought of as the process whereby patterns of activation are formed over the units in the model through their excitatory and inhibitory interactions. The memory trace of a processing event is the change or increment to the strengths of the interconnections that results from the processing event. The traces of separate events are superimposed on each other in the values of the connection strengths that result from the entire set of traces stored in the memory. The model is applied to a number of findings related to the question of whether we store abstract representations or an enumeration of specific experiences in memory. The model simulates the results of a number of important experiments which have been taken as evidence for the enumeration of specific experiences. At the same time, it shows how the functional equivalent of abstract representations—prototypes, logogens

When Newell introduced the concept of the knowledge level as a useful level of description for computer systems, he focused on the representation of knowledge. This paper applies the knowledge level notion to the problem of knowledge acquisition. Two interesting issues arise. First, some existing machine learning programs appear to be completely static when viewed at the knowledge level. These programs improve their performance without changing their "knowledge." Second, the behavior of some other machine learning programs cannot be predicted or described at the knowledge level. These programs take unjustified inductive leaps. The first programs are called symbol level learning (SLL) programs; the second, non-deductive knowledge level learning (NKLL) programs. The paper analyzes both of these classes of learning programs and speculates on the possibility of developing coherent theories of each. A theory of symbol level learning is sketched, and some reasons are presented for believing...

Fundamental developments in feedfonvard artificial neural net-works from the past thirty years are reviewed. The central theme of this paper is a description of the history, origination, operating characteristics, and basic theory of several supervised neural net-work training algorithms including the Perceptron rule, the LMS algorithm, three Madaline rules, and the backpropagation tech-nique. These methods were developed independently, but with the perspective of history they can a/ / be related to each other. The concept underlying these algorithms is the “minimal disturbance principle, ” which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. I.

A Threshold Circuit consists of an acyclic digraph of unbounded fanin, where each node computes a threshold function or its negation. This paper investigates the computational power of Threshold Circuits. A surprising relationship is uncovered between Threshold Circuits and another class of unbounded fanin circuits which are denoted Finite Field ZP (n) Circuits, where each node computes either multiple sums or products of integers modulo a prime P (n). In particular, it is proved that all functions computed by Threshold Circuits of size S(n) n and depth D(n) can also be computed by ZP (n) Circuits of size O(S(n) log S(n)+nP (n) log P (n)) and depth O(D(n)). Furthermore, it is shown that all functions computed by ZP (n) Circuits of size S(n) and depth D(n) can be computed by Threshold Circuits of size O ( 1 2 (S(n) log P (n)) 1+) and depth O ( 1 5 D(n)). These are the main results of this paper. There are many useful and quite surprising consequences of this result. For example, integer reciprocal can be computed in size n O(1) and depth O(1). More generally, anyanalytic function with a convergent rational polynomial power series (such as sine, cosine, exponentiation, square root, and logarithm) can be computed within accuracy 2,nc, for any constant c, by Threshold Circuits of

This chapter presents an overview of goals and directions in machine learning research, and is intended to serve as a conceptual road map to other chapters. It investigates intrinsic aspects of the learning process, classifies current lines of research, and presents the author's view of the relationship among learning paradigms, strategies and orientations.

Abstract not found

Abstract not found

We survey learning algorithms for recurrent neural networks with hidden units and attempt to put the various techniques into a common framework. We discuss fixpoint learning algorithms, namely recurrent backpropagation and deterministic Boltzmann Machines, and non-fixpoint algorithms, namely backpropagation through time, Elman's history cutoff nets, and Jordan's output feedback architecture. Forward propagation, an online technique that uses adjoint equations, is also discussed. In many cases, the unified presentation leads to generalizations of various sorts. Some simulations are presented, and at the end, issues of computational complexity are addressed. This research was sponsored in part by The Defense Advanced Research Projects Agency, Information Science and Technology Office, under the title "Research on Parallel Computing", ARPA Order No. 7330, issued by DARPA/CMO under Contract MDA972-90-C-0035 and in part by the National Science Foundation under grant number EET-8716324 and i...

In recent years, neural networks have been used for a wide variety of applications, from medical screening [ Rutenberg, 1992, Weber, 1990 ] to municipal power grid security [ Atlas et al., 1990c ] . Furthermore, comparisons between neural networks and more traditional techniques have shown that neural networks often produce competitive, and sometimes superior, results ( [ Weiss and Kulikowski, 1991, Shavlik et al., 1991, Thrun et al., 1991, Atlas et al., 1990b, Atlas et al., 1990c, Cole et al., 1990, Atlas et al., 1990a, Dietterich et al., 1990, Fisher and McKusick, 1989, Mooney et al., 1989, Pratt, 1990 ] ). Neural network training techniques still have room for improvement, however. Though they eventually achieve good performance levels, neural networks often require more computing time than competing methods (cf. [ Maren et al., 1990, Page 92 ] , [ Waibel et al., 1989 ] , [ Hertz et al., 1991, Page 120 ] ). One source of power which, if properly utilized, may help to alleviate thes...