Ensemble approaches to classification and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error “diversity”. However, the exact meaning of this concept is not clear from the literature, particularly for classification tasks. In this paper we first review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classification literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classification error diversity. 1

This paper addresses weighting and partitioning in complex reinforcement learning tasks, with the aim of facilitating learning. The paper presents some ideas regarding weighting of multiple agents and extends them into partitioning an input/state space into multiple regions with differential weighting in these regions, to exploit differential characteristics of regions and differential characteristics of agents to reduce the learning complexity of agents (and their function approximators) and thus to facilitate the learning overall. It analyzes, in reinforcement learning tasks, different ways of partitioning a task and using agents selectively based on partitioning. Based on the analysis, some heuristic methods are described and experimentally tested. We find that some off-line heuristic methods performed the best, significantly better than single-agent models. Keywords: weighting, averaging, neural networks, partitioning, gating, reinforcement learning, 1 Introduction Multiple ag...

Neural network ensembles are well accepted as a route to combining a group of weaker learning systems in order to make a composite, stronger one. It has been shown that low correlation of errors ("diverse members") will give rise to better ensemble performance. Most techniques for creating diverse ensemble members indirectly aect the learning trajectories, and are built upon heuristics and intuition. Other techniques directly influence the learning trajectory, by altering the training algorithm itself. For a particular direct technique, Negative Correlation Learning, we demonstrate the effectiveness of the algorithm in reducing correlations, as it relates to the size and complexity of the ensemble. We oer some possible research avenues on this class of ensemble methods. This work is a first step towards understanding the effectiveness of explicitly incorporating diversity measures in error functions during ensemble training.

Abstract. We study the formal basis behind Negative Correlation (NC) Learning, an ensemble technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be shown to be a derivative technique of the Ambiguity decomposition by Krogh and Vedelsby. From this formalisation, we calculate parameter bounds, and show significant improvements in empirical tests. We hypothesize that the reason for its success lies in rescaling an estimate of ensemble covariance; then show that during this rescaling, NC varies smoothly between a single neural network and an ensemble system. Finally we unify several other works in the literature, all of which have exploited the Ambiguity decomposition in some way, and term them the Ambiguity Family. 1

It is well-known that large neural networks with many unshared weights can be very difficult to train. A neural network ensemble consisting of a number of individual neural networks usually performs better than a complex monolithic neural network. One of the motivations behind neural network ensembles is the divide-and-conquer strategy, where a complex problem is decomposed into different components each of which is tackled by an individual neural network. A promising algorithm for training neural network ensembles is the negative correlation learning algorithm which penalizes positive correlations among individual networks by introducing a penalty term in the error function. A penalty coefficient is used to balance the minimization of the error and the minimization of the correlation. It is often very difficult to select an optimal penalty coefficient for a given problem because as yet there is no systematic method available for setting the parameter. This paper first applies negative correlation learning to the traffic flow prediction problem, and then proposes an evolutionary approach to deciding the penalty coefficient automatically in negative correlation learning. Experimental results on the traffic flow prediction problem will be presented.

We present an automatic method, based on a neural network ensemble, for extracting multiple, diverse and complementary sets of useful classification features from highdimensional data. We demonstrate the utility of these diverse representations for an image dataset, showing good classification accuracy and a high degree of dimensionality reduction. We then outline a number of possible extensions to the project in an evolutionary computation context.

In retrieval, indexing and classification of multimedia data an efficient information fusion of the different modalities is essential for the system’s overall performance. Since information fusion, its influence factors and performance improvement boundaries have been lively discussed in the last years in different research communities, we will review their latest findings. They most importantly point out that exploiting the feature’s and modality’s dependencies will yield to maximal performance. In data analysis and fusion tests with annotated image collections this is undermined.

Abstract — An efficient procedure for pruning regression ensembles is introduced. Starting from a bagging ensemble, pruning proceeds by ordering the regressors in the original ensemble and then selecting a subset for aggregation. Ensembles of increasing size are built by including first the regressors that perform best when aggregated. This strategy gives an approximate solution to the problem of extracting from the original ensemble the minimum error subensemble, which we prove to be NP-hard. Experiments show that pruned ensembles with only 20 % of the initial regressors achieve better generalization accuracies than the complete bagging ensembles. The performance of pruned ensembles is analyzed by means of the bias-variance decomposition of the error. I.

The current generation of non-modular neural network classifiers is unable to cope with classification problems which have a wide range of overlap among classes. This is due to the high coupling among the networks' hidden nodes. Modular neural network structures attempt to reduce this limitation via a divide-and-conquer approach. However, current modular designs are not offering a general alternative to the non-modular approach because they do not provide a reasonable balance between sub-tasks simplification, and decision-making efficiency. While the task-decomposition algorithm attempts to produce sub-tasks as simple as they can be, the modules are expected to give the multi-module decision-making strategy enough information to take an accurate global decision.

In many real-world tasks of image classification, limited amounts of labeled data are available to train automatic classifiers. Consequently, extensive human expert involvement is required for instance labeling. Detecting Egeria densa in digital imagery is one such real-world classification task. It presents an additional challenge due to subtle spectral changes in Egeria, which makes it difficult to find a single accurate classifier. A novel solution is proposed to employ an ensemble of classifiers for each class (class-specific ensembles), combined with an active learning scheme. The class-specific ensembles are implicitly diverse. Diversity is required to increase the overall accuracy when combining predictions. The combined predictions of the ensembles can be used to reduce the uncertainty in detecting Egeria. Iterative active learning is then suggested to adapt the ensembles to the new images, unseen to the active learner. A novel solution to build compact ensembles is also presented, which are needed to expedite the re-training of the active learner. The combined results are accurate and compact ensembles, which require significantly less expert involvement for image region classification. 1