Using an ensemble of classifiers, instead of a single classifier, can lead to improved generalization. The gains obtained by combining however, are often affected more by the selection of what is presented to the combiner, than by the actual combining method that is chosen. In this paper we focus on data selection and classifier training methods, in order to "prepare" classifiers for combining. We review a combining framework for classification problems that quantifies the need for reducing the correlation among individual classifiers. Then, we discuss several methods that make the classifiers in an ensemble more complementary. Experimental results are provided to illustrate the benefits and pitfalls of reducing the correlation among classifiers, especially when the training data is in limited supply. 2 1 Introduction A classifier's ability to meaningfully respond to novel patterns, or generalize, is perhaps its most important property (Levin et al., 1990; Wolpert, 1990). In...

Abstract not found

Several researchers have experimentally shown that substantial improvements can be obtained in difficult pattern recognition problems by combining or integrating the outputs of multiple classifiers. This chapter provides an analytical framework to quantify the improvements in classification results due to combining. The results apply to both linear combiners and order statistics combiners. We first show that to a first order approximation, the error rate obtained over and above the Bayes error rate, is directly proportional to the variance of the actual decision boundaries around the Bayes optimum boundary. Combining classifiers in output space reduces this variance, and hence reduces the "added" error. If N unbiased classifiers are combined by simple averaging, the added error rate can be reduced by a factor of N if the individual errors in approximating the decision boundaries are uncorrelated. Expressions are then derived for linear combiners which are biased or correlated, and the effect of output correlations on ensemble performance is quantified. For order statistics based non-linear combiners, we derive expressions that indicate how much the median, the maximum and in general the ith order statistic can improve classifier performance. The analysis presented here facilitates the understanding of the relationships among error rates, classifier boundary distributions, and combining in output space. Experimental results on several public domain data sets are provided to illustrate the benefits of combining and to support the analytical results.

: Several researchers have experimentally shown that substantial improvements can be obtained in difficult pattern recognition problems by combining or integrating the outputs of multiple classifiers. This paper provides an analytical framework to quantify the improvements in classification results due to combining. The results apply to both linear combiners and the order statistics combiners introduced in this paper. We show that combining networks in output space reduces the variance of the actual decision region boundaries around the optimum boundary. For linear combiners, we show that in the absence of classifier bias, the added classification error is proportional to the boundary variance. For non-linear combiners, we show analytically that the selection of the median, the maximum and in general the ith order statistic improves classifier performance. The analysis presented here facilitates the understanding of the relationships among error rates, classifier boundary distributions...

This paper introduces a neural network capable of dynamically adapting its architecture to realize time variant non-linear input-output maps. This network has its roots in the mixture of experts framework but uses a localized model for the gating network. Modules or experts are grown or pruned depending on the complexity of the modeling problem. The structural adaptation procedure addresses the model selection problem and typically leads to much better parameter estimation. Batch mode learning equations are extended to obtain on-line update rules enabling the network to model time varying environments. Simulation results are presented throughout the paper to support the proposed techniques. This research was supported in part by ARO contracts DAAH04-94-G-0417 and 04-95-10494 and NSF grant ECS 9307632. Contents 1 Introduction 3 2 Background on Mixture of Experts 4 2.1 Generic Mixture of Experts Architecture : : : : : : : : : : : : : : : : : : : : : : : 4 2.2 Drawbacks of a Global...

Integrating the outputs of multiple classifiers via combiners or meta-learners has led to substantial improvements in several difficult pattern recognition problems. In this article we investigate a family of combiners based on order statistics, for robust handling of situations where there are large discrepancies in performance of individual classifiers. Based on a mathematical modeling of how the decision boundaries are affected by order statistic combiners, we derive expressions for the reductions in error expected when simple output combination methods based on the the median, the maximum and in general, the i th order statistic, are used. Furthermore, we analyze the trim and spread combiners, both based on linear combinations of the ordered classifier outputs, and show that in the presence of uneven classifier performance, they often provide substantial gains over both linear and simple order statistics combiners. Experimental results on both real world data and standard public domain data sets corroborate these findings.

Combining the outputs of multiple neural networks has led to substantial improvements in several difficult pattern recognition problems. In this article, we introduce and investigate robust combiners, a family of classifiers based on order statistics. We focus our study to the analysis of the decision boundaries, and how these boundaries are affected by order statistics combiners. In particular, we show that using the ith order statistic, or a linear combination of the ordered classifier outputs is quite beneficial in the presence of outliers or uneven classifier performance. Experimental results on several public domain data sets corroborate these findings. I. Introduction In recent years, a great deal of attention has been focused on pooling as a means to improve the generalization ability of neural networks [22]. Approaches to pooling classifiers can be separated into two main categories: simple combiners, e.g., voting [4], [5] or averaging [16], and computationally expensive combi...

Several researchers have experimentally shown that substantial improvements can be obtained in difficult pattern recognition problems by combining or integrating the outputs of multiple classifiers. This paper provides an analytical framework that quantifies the improvements in classification results due to linear combining. We show that combining networks in output space reduces the variance of the actual decision region boundaries around the optimum boundary. In the absence of network bias, the added classification error is directly proportional to the boundary variance. Moreover, if the network errors are independent, then the reduction in variance boundary location is by a factor of N , the number of classifiers that are combined. In the presence of network bias, the reductions are less than or equal to N , depending on the interaction between network biases. We discuss how the individual networks can be selected to achieve significant gains through combining, and support them with...