Ensemble approaches to classification and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error “diversity”. However, the exact meaning of this concept is not clear from the literature, particularly for classification tasks. In this paper we first review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classification literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classification error diversity. 1

Introduction In the past decade, a number of papers 19,28 have proposed the combination of multiple classifiers for designing high performance pattern classification systems. The rationale behind the growing interest in multiple classifier systems (MCSs) is that the classical approach to designing a pattern recognition system, which focuses on the search for the best individual classifier, has some serious drawbacks. The main drawback is that the best individual classifier for the classification task at hand is very di#cult to identify, 199 200 F. Roli & G. Giacinto unless deep prior knowledge is available for such a task. 3,8 In addition, with a single classifier it is not possible to exploit the complementary discriminatory information that other classifiers may encapsulate. It is worth noting that the motivations in favour of MCS strongly resemble those of a "hybrid" intelligent system. 15,23 The obvious reason for this is that MCS can be regarded as a special-purpose hy

An appropriate use of neural computing techniques is to apply them to problems such as condition monitoring, fault diagnosis, control and sensing, where conventional solutions can be hard to obtain. However, when neural computing techniques are used, it is important that they are employed so as to maximise their performance, and improve their reliability. Their performance is typically assessed in terms of their ability to generalise to a previously unseen test set, although unless the training set is very carefully chosen, 100% accuracy is rarely achieved. Improved performance can result when sets of neural nets are combined in ensembles and ensembles can be viewed as an example of the reliability through redundancy approach that is recommended for conventional software and hardware in safety-critical or safety-related applications. Although there has been recent interest in the use of neural net ensembles, such techniques have yet to be applied to the tasks of condition ...

The performance of neural nets can be improved through the use of ensembles of redundant nets. In this paper, some of the available methods of ensemble creation are reviewed and the "test and select" methodolology for ensemble creation is considered. This approach involves testing potential ensemble combinations on a validation set, and selecting the best performing ensemble on this basis, which is then tested on a final test set. The application of this methodology, and of ensembles in general, is explored further in two case studies. The first case study is of fault diagnosis in a diesel engine, and relies on ensembles of nets trained from three different data sources. The second case study is of robot localisation, using an evidence-shifting method based on the output of trained SOMs. In both studies, improved results are obtained as a result of combining nets to form ensembles.

this paper is the exploitation of diversity to enhance computer system reliability. It is well-established that a diverse system composed of multiple alternative versions is more reliable than any single version alone, and this knowledge has occasionally been exploited in safety-critical applications. However, it is not clear what this property is, nor how the available diversity in a collection of versions is best exploited. We develop, define, illustrate and assess diversity measures, voting strategies for diversity exploitation, and interactions between the two. We take the view that a proper understanding of such issues is required if multiversion software engineering is to be elevated from the current `try it and see' procedure to a systematic technology. In addition, we introduce inductive programming techniques, particularly neural computing, as a cost-effective route to the practical use of multiversion systems outside the demanding requirements of safety-critical systems --- i.e. in general software engineering. software diversity, multiversion systems, diversity measures, voting strategies, neural computing, inductive programming

Recently, the concept of "Multiple Classifier Systems" vas proposed as a nev approach to the development of high performance image classification systems. Multiple Classifier Systems can be used to improve classification accuracy by combining the outputs of classifiers making "uncorrelated" errors. Unfortunately, in real image recognition problems, it may be very difficult to design an ensemble of classifiers that satisfies this assumption. In this paper, ve propose a different approach based on the concept of "adaptive selection" of multiple classifiers in order to select the most appropriate classifier for each input pattern.

Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally.

In the field of pattern recognition, the concept of Multiple Classifier Systems (MCSs) was proposed as a method for the development of high performance classification systems. At present, the common "operation" mechanism of MCSs is the "combination" of classifiers outputs. Recently, some researchers pointed out the potentialities of "dynamic classifier selection" as a new operation mechanism. In a previous paper, the authors discussed the advantages of "selection-based" MCSs and proposed an algorithm for dynamic classifier selection [1]. In this paper, a theoretical framework for dynamic classifier selection is described and two methods for selecting classifiers are proposed. Reported results on the classification of different data sets show that dynamic classifier selection is an effective method for the development of MCSs. 1. Introduction In the literature, many combination-based MCSs have been described [2-5]. The most of the combination methods used in such systems assume that c...

In earlier studies of multiversion programming, both empirical and analytical, emphasis switched from notions of independence to one of minimization of coincident failure. We show that neither independence of failure, nor lack of coincident failure are the single important properties. Indeed, an N-version system may deliver an optimal performance (under some voting strategy) even when the incidence of coincident failure is arbitrarily high. The key notion that this study contributes is one of distinct different failure, and hence distinct-failure diversity. The important property is not whether versions fail on the same input so much as whether they fail in the same way. If the failures of an N-version system (on some input) are dispersed over a set of distinct alternative outcomes, then this (hitherto unacknowledged) aspect of diversity may be exploited to substantially enhance system reliability. We propose measures for the traditional coincident-failure diversity (CFD)...

Neural network ensembles are well accepted as a route to combining a group of weaker learning systems in order to make a composite, stronger one. It has been shown that low correlation of errors ("diverse members") will give rise to better ensemble performance. Most techniques for creating diverse ensemble members indirectly aect the learning trajectories, and are built upon heuristics and intuition. Other techniques directly influence the learning trajectory, by altering the training algorithm itself. For a particular direct technique, Negative Correlation Learning, we demonstrate the effectiveness of the algorithm in reducing correlations, as it relates to the size and complexity of the ensemble. We oer some possible research avenues on this class of ensemble methods. This work is a first step towards understanding the effectiveness of explicitly incorporating diversity measures in error functions during ensemble training.