Hybrid learning methods use theoretical knowledge of a domain and a set of classified examples to develop a method for accurately classifying examples not seen during training. The challenge of hybrid learning systems is to use the information provided by one source of information to offset information missing from the other source. By so doing, a hybrid learning system should learn more effectively than systems that use only one of the information sources. KBANN(Knowledge-Based Artificial Neural Networks) is a hybrid learning system built on top of connectionist learning techniques. It maps problem-specific "domain theories", represented in propositional logic, into neural networks and then refines this reformulated knowledge using backpropagation. KBANN is evaluated by extensive empirical tests on two problems from molecular biology. Among other results, these tests show that the networks created by KBANN generalize better than a wide variety of learning systems, as well as several t...

This paper will give an overview of the various approaches to neurosymbolic integration. Roughly, these can be divided into two strategies: unified strategies aim at attaining neural and symbolic capabilities using neural networks alone, while hybrid strategies combine neural networks with symbolic models such as expert systems, case-based reasoning systems, 2 Chapter 2 and decision trees. These two approaches form the main subtrees of the classification hierarchy depicted in Figure 1. Symbol Proc. Neuronal Unified approach Symbol Proc. hybrids Connectionist Localist Hybrid approach Combined L/D Neurosymbolic integration Functional Chainprocessing Translational Subprocessing hybrids Metaprocessing Distributed Coprocessing Figure 1 Classification of integrated neurosymbolic systems.

An algorithm that learns from a set of examples should ideally be able to exploit the available resources of (a) abundant computing power and (b) domain-specific knowledge to improve its ability to generalize. Connectionist theory-refinement systems, which use background knowledge to select a neural network's topology and initial weights, have proven to be effective at exploiting domain-specific knowledge; however, most do not exploit available computing power. This weakness occurs because they lack the ability to refine the topology of the neural networks they produce, thereby limiting generalization, especially when given impoverished domain theories. We present the REGENT algorithm which uses (a) domain-specific knowledge to help create an initial population of knowledge-based neural networks and (b) genetic operators of crossover and mutation (specifically designed for knowledge-based networks) to continually search for better network topologies. Experiments on three real-world domains indicate that our new algorithm is able to significantly increase generalization compared to a standard connectionist theory-refinement system, as well as our previous algorithm for growing knowledge-based networks.

This paper describes techniques for integrating neural networks and symbolic components into powerful hybrid systems. Neural networks have unique processing characteristics that enable tasks to be performed that would be di cult or intractable for a symbolic rule-based system. However, a stand-alone neural network requires an interpretation either by ahuman or a rulebased system. This motivates the integration of neural/symbolic techniques within a hybrid system. Anumber of integration possibilities exist: some systems consist of neural network components performing symbolic tasks while other systems are composed of several neural networks and symbolic components, each component acting as a self-contained module communicating with the others. Other hybrid systems are able to transform subsymbolic representations into symbolic ones and vice-versa. This paper providesanoverview and evaluation of the state of the artofseveral hybrid neural systems for rule-based processing. 1

Traditional connectionist theory-refinement systems map the dependencies of a domainspecific rule base into a neural network, and then refine this network using neural learning techniques. Most of these systems, however, lack the ability to refine their network's topology and are thus unable to add new rules to the (reformulated) rule base. Therefore, on domain theories that are lacking rules, generalization is poor, and training can corrupt the original rules, even those that were initially correct. We present TopGen, an extension to the Kbann algorithm, that heuristically searches for possible expansions to Kbann's network. TopGen does this by dynamically adding hidden nodes to the neural representation of the domain theory, in a manner analogous to adding rules and conjuncts to the symbolic rule base. Experiments indicate that our method is able to heuristically find effective places to add nodes to the knowledge bases of four realworld problems, as well as an artificial chess domai...

Many scientific and industrial problems can be better understood by learning from samples of the task at hand. For this reason, the machine learning and statistics communities devote considerable research effort on generating inductive-learning algorithms that try to learn the true "concept" of a task from a set of its examples. Often times, however, one has additional resources readily available, but largely unused, that can improve the concept that these learning algorithms generate. These resources include available computer cycles, as well as prior knowledge describing what is currently known about the domain. Effective utilization of available computer time is important since for most domains an expert is willing to wait for weeks, or even months, if a learning system can produce an improved concept. Using prior knowledge is important since it can contain information not present in the current set of training examples. In this thesis, I present three "anytime" approaches to connec...

Functionality-based object recognition systems recognize objects at the category level by reasoning about how well the objects support the expected function of the category. Such systems naturally associate a "measure of goodness" or "membership value" with a recognized object. This measure of goodness is the result of combining individual measures, or membership values, from potentially many primitive evaluations of different properties of the object's shape. A membership function is used to compute the membership value for a primitive evaluation of a particular physical property of an object. In previous versions of a recognition system known as GRUFF, the membership function for each of the primitive evaluations had to be hand-crafted by the system designer. In this paper, we provide a learning component for the GRUFF system, called Omlet, that automatically learns primitive evaluation membership functions given a set of example objects labeled with their desired category measure. T...

An ideal inductive-learning algorithm should exploit all available resources, such as computing power and domain-specific knowledge, to improve its ability to generalize. Connectionist theory-refinement systems have proven to be effective at utilizing domainspecific knowledge; however, most are unable to exploit available computing power. This weakness occurs because they lack the ability to refine the topology of the networks they produce, thereby limiting generalization, especially when given impoverished domain theories. We present the Regent algorithm, which uses genetic algorithms to broaden the type of networks seen during its search. It does this by using (a) the domain theory to help create an initial population and (b) crossover and mutation operators specifically designed for knowledgebased networks. Experiments on three realworld domains indicate that our new algorithm is able to significantly increase generalization compared to a standard connectionist theory-refinement sys...

The advantages and disadvantages of classical rule-based and neural approaches to expert system design are complementary. We propose a strictly neural expert system architecture, that enables the creation of the knowledge base automatically, by learning from example inferences. For this purpose we employ a multi-layered neural network, trained with generalized back propagation for interval training patterns, that also makes the learning of patterns with irrelevant inputs and outputs possible. We eliminate the disadvantages of the neural approach by enriching the system with the heuristics to work with incomplete information, and to explain the conclusions. The structure of the expert attributes is optional, and a user of the system can define the types of inputs and outputs (real, integer, scalar type, and set), and the manner of their coding (floating point, binary, and unary codes). We have tested our neural expert system on several non-trivial real-world problems (e.g., the diagnost...

There has been an increased interest in the use of prior knowledge for training neural networks. Prior knowledge in the form of Horn clauses has been the predominant paradigm for knowledge-based neural networks. Given a set of training examples and an initial domain theory, a neural network is constructed that fits the training examples by preprogramming some of the weights. The initialized neural network is then trained using backpropagation to refine the knowledge. This paper proposes a heuristic for determining the strength of the inductive bias by making use of gradient information in weight space in the direction of the programmed weights. The network starts its search in weight space where the gradient is maximal thus speeding-up convergence. Tests on a benchmark problem from molecular biology demonstrate that our heuristic, on average, reduces the training time by 60 % compared to a random choice of the strength of the inductive bias; this performance is within 20 % of the training time that can be achieved with the optimal inductive bias. The difference in generalization performance is not statistically significant. 1