Results 1  10
of
13
Diversity in Neural Network Ensembles
, 2004
"... We study the issue of error diversity in ensembles of neural networks. In ensembles of regression estimators, the measurement of diversity can be formalised as the BiasVarianceCovariance decomposition. In ensembles of classifiers, there is no neat theory in the literature to date. Our objective is ..."
Abstract

Cited by 43 (4 self)
 Add to MetaCart
We study the issue of error diversity in ensembles of neural networks. In ensembles of regression estimators, the measurement of diversity can be formalised as the BiasVarianceCovariance decomposition. In ensembles of classifiers, there is no neat theory in the literature to date. Our objective is to understand how to precisely define, measure, and create diverse errors for both cases. As a focal point we study one algorithm, Negative Correlation (NC) Learning which claimed, and showed empirical evidence, to enforce useful error diversity, creating neural network ensembles with very competitive performance on both classification and regression problems. With the lack of a solid understanding of its dynamics, we engage in a theoretical and empirical investigation. In an initial empirical stage, we demonstrate the application of an evolutionary search algorithm to locate the optimal value for λ, the configurable parameter in NC. We observe the behaviour of the optimal parameter under different ensemble architectures and datasets; we note a high degree of unpredictability, and embark on a more formal investigation. During the theoretical investigations, we find that NC succeeds due to exploiting the
Stability Problems with Artificial Neural Networks and the Ensemble Solution
 Artificial Intelligence in Medicine
, 1999
"... Artificial Neural Networks (ANNs) are very popular as classification or regression mechanisms in medical decision support systems despite the fact that they are unstable predictors. This instability means that small changes in the training data used to build the model (i.e. train the ANN) may res ..."
Abstract

Cited by 29 (4 self)
 Add to MetaCart
(Show Context)
Artificial Neural Networks (ANNs) are very popular as classification or regression mechanisms in medical decision support systems despite the fact that they are unstable predictors. This instability means that small changes in the training data used to build the model (i.e. train the ANN) may result in very different models. A central implication of this is that different sets of training data may produce models with very different generalisation accuracies. In this paper we show in detail how this can happen in a prediction system for use in InVitro Fertilisation. We argue that claims for the generalisation performance of ANNs used in such a scenario should only be based on kfold cross validation tests. We also show how the accuracy of such a predictor can be improved by aggregating the output of several predictors. 1. Introduction Artificial Neural Networks (ANNs) are hugely popular in research on medical decision support systems (see Baxt's review of clinical applicatio...
The Use of the Ambiguity Decomposition in Neural Network Ensemble Learning Methods
 In Fawcett, T., & Mishra, N. (Eds.), 20th International Conference on Machine Learning (ICML’03
, 2003
"... We analyze the formal grounding behind Negative Correlation (NC) Learning, an ensemble learning technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be seen to be exploiting the wellknown Ambiguity decomposition of ..."
Abstract

Cited by 13 (3 self)
 Add to MetaCart
(Show Context)
We analyze the formal grounding behind Negative Correlation (NC) Learning, an ensemble learning technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be seen to be exploiting the wellknown Ambiguity decomposition of the ensemble error, grounding it in a statistics framework around the biasvariance decomposition. We use this grounding to find bounds for the parameters, and provide insights into the behaviour of the optimal parameter values. These observations allow us understand how NC relates to other algorithms, identifying a group of papers spread over the last decade that have all exploited the Ambiguity decomposition for machine learning problems. When taking into account our new understanding of the algorithm, significant reductions in error rates were observed in empirical tests. 1.
Negative correlation learning and the ambiguity family of ensemble methods
 In Proc. Int. Workshop on Multiple Classifier Systems (LNCS 2709
, 2003
"... Abstract. We study the formal basis behind Negative Correlation (NC) Learning, an ensemble technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be shown to be a derivative technique of the Ambiguity decomposition by ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
(Show Context)
Abstract. We study the formal basis behind Negative Correlation (NC) Learning, an ensemble technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be shown to be a derivative technique of the Ambiguity decomposition by Krogh and Vedelsby. From this formalisation, we calculate parameter bounds, and show significant improvements in empirical tests. We hypothesize that the reason for its success lies in rescaling an estimate of ensemble covariance; then show that during this rescaling, NC varies smoothly between a single neural network and an ensemble system. Finally we unify several other works in the literature, all of which have exploited the Ambiguity decomposition in some way, and term them the Ambiguity Family. 1
A Learning Algorithm for Neural Networks Ensembles
 ASAI'2000: Proceedings of the Argentine Symposium on Artificial Intelligence
, 2001
"... The performance of a single regressor/classifier can be improved by combining the outputs of several predictors. This is true provided the combined predictors are accurate and diverse enough, which posses the problem of generating suitable aggregate members in order to have optimal generalization ca ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
(Show Context)
The performance of a single regressor/classifier can be improved by combining the outputs of several predictors. This is true provided the combined predictors are accurate and diverse enough, which posses the problem of generating suitable aggregate members in order to have optimal generalization capabilities. We propose here a new method for selecting members of regression/classification ensembles. In particular, using artificial neural networks as learners in a regression context, we show that this method leads to small aggregates with few but very diverse individual networks. The algorithm is favorably tested against other methods recently proposed in the literature, producing equal performance on the standard statistical databases used as benchmarks with ensembles that have 75 % less members on average.
Diagnositic classifier ensembles: enforcing diversity for reliability in the combination
, 2000
"... This thesis is dedicated to the memory of my late parents. The construction of reliable machinery fault diagnostic systems is investigated in this thesis. The idea of using unitary sensorial information to develop automated fault classifiers is studied. The domain of fault diagnosis is used to inve ..."
Abstract

Cited by 3 (1 self)
 Add to MetaCart
(Show Context)
This thesis is dedicated to the memory of my late parents. The construction of reliable machinery fault diagnostic systems is investigated in this thesis. The idea of using unitary sensorial information to develop automated fault classifiers is studied. The domain of fault diagnosis is used to investigate the concept of enforcing methodological diversity in the solutions with a view to obtaining robust and reliable systems by combining the decisions of several independent classifiers. A twin cylinder, high speed, 4stroke diesel engine is used as an exemplar of the class of mechanical machines. Several commonly occurring faults symptomatic of early stages of fault development are induced in the engine. Data in the form of cylinder pressure and vibration are acquired. Orthogonal wavelet transforms, principal component analysis and domain expertise of the engine cycle are used to extract features from the vibration and pressure signals. Several artificial neural net classifiers are trained with these features, after establishing that simpler visualisation tech
Algorithm Selecting Diverse Members Neural Network Ensemble
 Proceedings of the VI International Congress on Information Engineering, University of Buenos Aires
, 2000
"... Ensembles of neural networks have been used in the last years as regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that, for aggregation to be effective, the individual predictors must be as accurate and divers ..."
Abstract

Cited by 2 (2 self)
 Add to MetaCart
Ensembles of neural networks have been used in the last years as regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that, for aggregation to be effective, the individual predictors must be as accurate and diverse as possible. An important problem is, then, how to choose the aggregate members in order to have an optimal compromise between these two conflicting conditions. In particular, for neural network ensembles some proposals in this direction have been already discussed in the literature. We present here a new method for selecting members of neural network ensembles that leads to small aggregates with few but very diverse individual networks. This algorithm is favorably tested against an optimal earlystopping method recently proposed, using the sunspot time series as benchmark. We also provide preliminary results on the actual prediction of the remaining of the current cycle 23 of solar activity.
Selecting Diverse Members Of Neural Network Ensembles
, 2000
"... Ensembles of artificial neural networks have been used in the last years as classification/regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that for aggregation to be effective the individual networks must be ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
Ensembles of artificial neural networks have been used in the last years as classification/regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that for aggregation to be effective the individual networks must be as accurate and diverse as possible. An important problem is, then, how to choose the aggregate members in order to have an optimal compromise between these two conflicting conditions. We propose here a new method for selecting members of regression/classification ensembles that leads to small aggregates with few but very diverse individual predictors. Using artificial neural networks as individual learners, the algorithm is favorably tested against other methods recently proposed in the literature, producing a remarkable performance improvement on the standard statistical databases used as benchmarks. In addition, and as a concrete application, we study the sunspot time series and predict the remaining of the current cycle 23 of solar activity.
Social Programming on MARS: A Benchmark Study
, 2003
"... The Social Programming methodology is base on combining the Particle Swarm methodology with The Group Method of Data Handling and Cartesian Programming. In this paper ..."
Abstract
 Add to MetaCart
The Social Programming methodology is base on combining the Particle Swarm methodology with The Group Method of Data Handling and Cartesian Programming. In this paper