Results 1 
9 of
9
A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems
 IEEE Transactions on Pattern Analysis and Machine Intelligence
, 2005
"... In this paper, a theoretical and experimental analysis of linear combiners for multiple classifier systems is presented. Although linear combiners are the most frequently used combining rules, many important issues related to their operation for pattern classification tasks lack a theoretical basis. ..."
Abstract

Cited by 55 (2 self)
 Add to MetaCart
(Show Context)
In this paper, a theoretical and experimental analysis of linear combiners for multiple classifier systems is presented. Although linear combiners are the most frequently used combining rules, many important issues related to their operation for pattern classification tasks lack a theoretical basis. After a critical review of the framework developed in works by Tumer and Ghosh, on which our analysis is based, we focus on the simplest and most widely used implementation of linear combiners, which consists in assigning a nonnegative weight to each individual classifier. Moreover, we consider the ideal performance of this combining rule, i.e., that achievable when the optimal values of the weights are used. We do not consider the problem of weights estimation, which has been extensively addressed in the literature. Our theoretical analysis shows how the performance of linear combiners, in terms of misclassification probability, depends on the performance of individual classifiers, and on the correlation between their outputs. In particular, we evaluate the ideal performance improvement that can be achieved using the weighted average over the simple average combining rule, and investigate in what way it depends on the individual classifiers. Experimental results on real data sets show that the behaviour of linear combiners agrees with the predictions of our analytical model. Finally, we discuss the contribution to the state of the art and the practical relevance of our theoretical and experimental analysis of linear combiners for multiple classifier systems.
Classifier Ensembles: Select RealWorld Applications
, 2008
"... Broad classes of statistical classification algorithms have been developed and applied successfully to a wide range of real world domains. In general, ensuring that the particular classification algorithm matches the properties of the data is crucial in providing results that meet the needs of the p ..."
Abstract

Cited by 23 (0 self)
 Add to MetaCart
Broad classes of statistical classification algorithms have been developed and applied successfully to a wide range of real world domains. In general, ensuring that the particular classification algorithm matches the properties of the data is crucial in providing results that meet the needs of the particular application domain. One way in which the impact of this algorithm/application match can be alleviated is by using ensembles of classifiers, where a variety of classifiers (either different types of classifiers or different instantiations of the same classifier) are pooled before a final classification decision is made. Intuitively, classifier ensembles allow the different needs of a difficult problem to be handled by classifiers suited to those particular needs. Mathematically, classifier ensembles provide an extra degree of freedom in the classical bias/variance tradeoff, allowing solutions that would be difficult (if not impossible) to reach with only a single classifier. Because of these advantages, classifier ensembles have been applied to many difficult real world problems. In this paper, we survey select applications of ensemble methods to problems that have historically been most representative of the difficulties in classification. In particular, we survey applications of ensemble methods to remote sensing, person recognition, one vs. all recognition, and medicine.
Multiclassifier systems: Back to the future
 Multiple Classifier Systems, pages invited paper, 1–15. LNCS
, 2002
"... Abstract. While a variety of multiple classifier systems have been studied since at least the late 1950’s, this area came alive in the 90’s with significant theoretical advances as well as numerous successful practical applications. This article argues that our current understanding of ensembletype ..."
Abstract

Cited by 20 (3 self)
 Add to MetaCart
(Show Context)
Abstract. While a variety of multiple classifier systems have been studied since at least the late 1950’s, this area came alive in the 90’s with significant theoretical advances as well as numerous successful practical applications. This article argues that our current understanding of ensembletype multiclassifier systems is now quite mature and exhorts the reader to consider a broader set of models and situations for further progress. Some of these scenarios have already been considered in classical pattern recognition literature, but revisiting them often leads to new insights and progress. As an example, we consider how to integrate multiple clusterings, a problem central to several emerging distributed data mining applications. We also revisit output space decomposition to show how this can lead to extraction of valuable domain knowledge in addition to improved classification accuracy. 1 A Brief History of Multilearner Systems Multiple classifier systems are special cases of approaches that integrate several
Performance Analysis and Comparison of Linear Combiners for Classifier Fusion
 Proc. of IAPR Int. Workshop on Statistical Pattern Recognition (SPR 2002), in press
, 2002
"... In this paper, we report a theoretical and experimental comparison between two widely used combination rules for classifier fusion: simple average and weighted average of classifiers outputs. We analyse the conditions which affect the difference between the performance of simple and weighted ave ..."
Abstract

Cited by 15 (5 self)
 Add to MetaCart
(Show Context)
In this paper, we report a theoretical and experimental comparison between two widely used combination rules for classifier fusion: simple average and weighted average of classifiers outputs. We analyse the conditions which affect the difference between the performance of simple and weighted averaging and discuss the relation between these conditions and the concept of classifiers' "imbalance". Experiments aimed at assessing some of the theoretical results for cases where the theoretical assumptions could not be hold are reported.
Analysis of ErrorReject Tradeoff in Linearly Combined Classifiers
 Pattern Recognition
, 2002
"... In this paper, a framework for the analysis of the errorreject tradeoff in linearly combined classifiers is proposed. We start from a framework developed by Tumer and Ghosh [1,2]. We extend this framework and analyse some hypotheses under which the linear combination of classifier outputs can impr ..."
Abstract

Cited by 11 (3 self)
 Add to MetaCart
(Show Context)
In this paper, a framework for the analysis of the errorreject tradeoff in linearly combined classifiers is proposed. We start from a framework developed by Tumer and Ghosh [1,2]. We extend this framework and analyse some hypotheses under which the linear combination of classifier outputs can improve the errorreject tradeoff of the individual classifiers. Experiments that support some of the analytical results are reported.
Abstract Analysis of errorreject tradeoff in linearly combined multiple classifiers
"... In this paper, a theoretical and experimental analysis of the errorreject tradeoff achievable by linearly combining the outputs of an ensemble of classifiers is presented. To this aim, the theoretical framework previously developed by Tumer and Ghosh for the analysis of the simple average rule wit ..."
Abstract
 Add to MetaCart
(Show Context)
In this paper, a theoretical and experimental analysis of the errorreject tradeoff achievable by linearly combining the outputs of an ensemble of classifiers is presented. To this aim, the theoretical framework previously developed by Tumer and Ghosh for the analysis of the simple average rule without the reject option has been extended. Analytical results that allow to evaluate the improvement of the errorreject tradeoff achievable by simple averaging their outputs under different assumptions about the distributions of the estimation errors affecting a posteriori probabilities, are provided. The conditions under which the weighted average can provide a better errorreject tradeoff than the simple average are then determined. From the theoretical results obtained under the assumption of unbiased and uncorrelated estimation errors, simple guidelines for the design of multiple classifier systems using linear combiners are given. Finally, an experimental evaluation and comparison of the errorreject tradeoff of the simple and weighted averages is reported for five real data sets. The results show the practical relevance of the proposed guidelines in the design of linear combiners.
Dynamics of Variance Reduction in Bagging and Other Techniques Based on Randomisation
"... Abstract. In this paper the performance of bagging in classification problems is theoretically analysed, using a framework developed in works by Tumer and Ghosh and extended by the authors. A biasvariance decomposition is derived, which relates the expected misclassification probability attained by ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract. In this paper the performance of bagging in classification problems is theoretically analysed, using a framework developed in works by Tumer and Ghosh and extended by the authors. A biasvariance decomposition is derived, which relates the expected misclassification probability attained by linearly combining classifiers trained on N bootstrap replicates of a fixed training set to that attained by a single bootstrap replicate of the same training set. Theoretical results show that the expected misclassification probability of bagging has the same bias component as a single bootstrap replicate, while the variance component is reduced by a factor N. Experimental results show that the performance of bagging as a function of the number of bootstrap replicates follows quite well our theoretical prediction. It is finally shown that theoretical results derived for bagging also apply to other methods for constructing multiple classifiers based on randomisation, such as the random subspace method and tree randomisation. 1
Bagging with Adaptive Costs
"... Ensemble methods have proved to be highly effective in improving the performance of base learners under most circumstances. In this paper, we propose a new algorithm that combines the merits of some existing techniques, namely bagging, arcing and stacking. The basic structure of the algorithm resemb ..."
Abstract
 Add to MetaCart
(Show Context)
Ensemble methods have proved to be highly effective in improving the performance of base learners under most circumstances. In this paper, we propose a new algorithm that combines the merits of some existing techniques, namely bagging, arcing and stacking. The basic structure of the algorithm resembles bagging, using a linear support vector machine (SVM). However, the misclassification cost of each training point is repeatedly adjusted according to its observed outofbag vote margin. In this way, the method gains the advantage of arcing – building the classifier the ensemble needs – without fixating on potentially noisy points. Computational experiments show that this algorithm performs consistently better than bagging and arcing. 1