Genetic programming is distinguished from other evolutionary algorithms in that it uses tree representations of variable size instead of linear strings of fixed length. The flexible representation scheme is very important because it allows the underlying structure of the data to be discovered automatically. One primary difficulty, however, is that the solutions may grow too bigwithout any improvement oftheir generalization ability. In this article we investigate the fundamental relationship between the performance and complexity of the evolved structures. The essence of the parsimony problem is demonstrated empirically by analyzing error landscapes of programs evolved for neural network synthesis. We consider genetic programming as a statistical inference problem and apply the Bayesian modelcomparison framework to introduce a class of fitness functions with error and complexity terms. An adaptive learning method is then presented that automatically balances the model-complexity factor to evolve parsimonious programs without losing the diversity of the population needed for achieving the desired training accuracy. The effectiveness of this approach is empirically shown on the induction of sigma-pi neural networks for solving a real-world medical diagnosis problem as well as benchmark tasks.

This paper introduces a Minimum Description Length (MDL) principle to define fitness functions in Genetic Programming (GP). In traditional (Koza-style) GP, the size of trees was usually controlled by user-defined parameters, such as the maximum number of nodes and maximum tree depth. Large tree sizes meant that the time necessary to measure their fitnesses often dominated total processing time. To overcome this difficulty, we introduce a method for controlling tree growth, which uses an MDL principle. Initially we choose a "decision tree" representation for the GP chromosomes, and then show how an MDL principle can be used to define GP fitness functions. Thereafter we apply the MDL-based fitness functions to some practical problems. Using our implemented system "STROGANOFF", we show how MDL-based fitness functions can be applied successfully to problems of pattern recognitions. The results demonstrate that our approach is superior to usual neural networks in terms of general...

Abstract not found

This paper is concerned with the automatic induction of parsimonious neural networks. In contrast to other program induction situations, network induction entails parametric learning as well as structural adaptation. We present a novel representation scheme called neural trees that allows efficient learning of both network architectures and parameters by genetic search. A hybrid evolutionary method is developed for neural tree induction that combines genetic programming and the breeder genetic algorithm under the unified framework of the minimum description length principle. The method is successfully applied to the induction of higher order neural trees while still keeping the resulting structures sparse to ensure good generalization performance. Empirical results are provided on two chaotic time series prediction problems of practical interest.

Genetic Programming (GP) uses variable size representations as programs. Size becomes an important and interesting emergent property of the structures evolved by GP. The size of programs can be both a controlling and a controlled factor in GP search. Size influences the efficiency of the search process and is related to the generality of solutions. This paper analyzes the size and generality issues in standard GP and GP using subroutines and addresses the question whether such an analysis can help control the search process. We relate the size, generalization and modularity issues for programs evolved to control an agent in a dynamic and non-deterministic environment, as exemplified by the Pac-Man game. 1 Introduction Genetic Programming (Koza, 1992) has been applied to a variety of machine learning applications formulated mostly as classification or prediction problems. Some examples include the prediction of omega loops in proteins and the transmembrane problem, symbolic regression...

Genetic algorithms (GAs) have been shown to be a promising approach for a wide range of search and optimization problems. One of the main obstacles in applying GAs to complex problems has often been the high computational cost due to their slow convergence rate. We encountered such a difficulty in an attempt to use the classical GA for estimating parameters of a metabolic model. Adopting a common strategy in the literature for addressing the problem -- integrating the GA with a complementary optimization technique, we developed a hybrid approach that combines a real-coded GA with a stochastic variant of simplex method in function optimization. Our empirical evaluations showed that the performance of our hybrid approach for the metabolic modeling problem improved those of a pure real-coded GA and an alternative simplex-GA hybrid developed by Renders and Bersini. We showed that the hybrid approach also improved GA's convergence rate for a function optimization problem. Based on an empiric...

Abstract: Developments in computational models of evolutionary processes have led to the realisation of powerful, robust and general optimization and adaptive systems collectively called evolutionary algorithms. In this paper we provide an overview of evolutionary algorithms and consider the features and characteristics that are particularly appropriate for control engineering applications. The versatile and robust qualities of these algorithms are considered and a number of application areas described.

Genetic programming has been successfully applied to evolve computer programs for solving a variety of interesting problems. In the previous work we introduced the breeder genetic programming (BGP) method that has Occam's razor in its fitness measure to evolve minimal size multilayer perceptrons. In this paper we apply the method to synthesis of sigma-pi neural networks. Unlike perceptron architectures, sigma-pi networks use product units as well as summation units to build higher-order terms. The effectiveness of the method is demonstrated on benchmark problems. Simulation results on noisy data suggest that BGP not only improves the generalization performance, it can also accelerate the convergence speed. I. Introduction Genetic programming has been successfully used to evolve computer programs for solving many interesting problems in artificial intelligence and artificial life [4, 16, 18, 19]. Similar to usual genetic algorithms (GAs), genetic programming (GP) starts with a populatio...

Computers that "program themselves"; science fact or fiction? Genetic Programming uses novel optimisation techniques to "evolve " simple programs; mimicking the way humans construct programs by progressively re-writing them. Trial programs are repeatedly modified in the search for "better/fitter " solutions. The underlying basis is Genetic Algorithms (GAs). Genetic Algorithms, pioneered by Holland [Hol92], Goldberg [Gol89] and others, are evolutionary search techniques inspired by natural selection (i.e survival of the fittest). GAs work with a "population " of trial solutions to a problem, frequently encoded as strings, and repeatedly select the "fitter " solutions, attempting to evolve better ones. The power of GAs is being demonstrated for an increasing range of applications; financial, imaging, VLSI circuit layout, gas pipeline control and production scheduling [Dav91]. But one of the most intriguing uses of GAs- driven by Koza [Koz92]- is automatic program generation. Genetic Programming applies GAs to a "population " of programs- typically encoded as tree-structures. Trial programs are evaluated against a "fitness function " and the best solutions selected for modification and re-evaluation. This modification-evaluation cycle is repeated

Complex Adaptive Structures can be viewed as a combination of Complex Adaptive Systems and fully integrated autonomous Smart Structures. Traditionally when designing a structure, one combines rules of thumb with theoretical results to develop an acceptable solution. This methodology will have to be extended for Complex Adaptive Structures, since they, by definition, will participate in their own design. In this paper we introduce a new methodology for Emergent System Identification that is concerned with combining the methodologies of self-organizing funcional networks (GMDH - Alexy G. Ivakhnenko), Particle Swarm Optimization (PSO - James Kennedy and Russel C. Eberhart) and Genetic Programming (GP - John Joza). This paper will concentrate on the utilization of Particle Swarm Optimization in this effort and discuss how Particle Swarm Optimization relates to our ultimate gooal of emergent self-organizing functional networks that can be used to identify overlapping internal structural models. The ability for Complex Adaptive Structures to identify emerging internal models will be a key component for their success.