Abstract

This paper reviews research on combining artificial neural nets, and provides an overview of, and an introduction to, the papers contained this Special Issue, and its companion (Connection Science, 9, 1). Two main approaches, ensemble-based, and modular, are identified and considered. An ensemble, or committee, is made up of a set of nets, each of which is a general function approximator. The members of the ensemble are combined in order to obtain better generalisation performance than would be achieved by any of the individual nets. The main issues considered here under the heading of ensemble-based approaches, are (a) how to combine the outputs of the ensemble members (b) how to create candidate ensemble members and (c) which methods lead to the most effective ensembles? Under the heading of modular approaches we begin by considering a divide-and-conquer approach by which a function is automatically decomposed into a number of subfunctions which are treated by specialis...

Citations

3909	Neural Networks for Pattern Recognition - Bishop - 1995
1998	Bagging predictors - Breiman - 1996
665	Adaptive mixtures of local experts - Jacobs, Jordan, et al. - 1991
634	Hierarchical mixtures of experts and the EM algorithm - Jordan, Jacobs - 1994
554	The strenght of weak learnability - Schapire - 1990
463	Stacked generalization - Wolpert - 1992
425	Neural network ensembles - Hansen, Salamon - 1990
354	Neural network ensembles, cross validation, and active learning - Krogh, Vedelsby - 1995
317	On learning the past tense of English verbs - Rumelhart, McClelland - 1986
267	When networks disagree: ensemble methods for hybrid neural networks - Perrone, Cooper - 1993
264	Classification and Regression - Breiman, Friedman, et al. - 1984
141	Combining forecasts: A review and annotated bibliography - Clemen - 1989
124	Combining probability distributions: A critique and an annotated bibliography - Genest, Zidek - 1986
121	Methods for combining experts’ probability assessments - Jacobs - 1995
94	Combining the Results of Several Neural-Network Classifiers. Neural Networks 7: 777–781 - Rogova - 1994
92	Conceptual modeling of coincident failures in multiversion software - B, Miller - 1989
91	variance and arcing classifiers - Breiman, “Bias - 1996
79	Back propagation is sensitive to initial conditions - Kolen, Pollack - 1991
73	A theoretical basis for the analysis of multiversion software subject to coincident errors - Eckhardt, Lee - 1985
71	Hopfield; Large automatic learning, rule extraction and generalization - Denker, Schwartz, et al. - 1987
70	Learning Machines: Foundations of Trainable Pattern-Classifying Systems - Nilsson - 1965
69	Boosting and other ensemble methods - Drucker, Cortes, et al. - 1301
52	Computational methods for a mathematical theory of evidence - Barnett - 1981
50	The meta-pi network: building distributed knowledge representations for robust pattern recognition - Hampshire, Waibel - 1989
35	Combining Forecasts – Twenty Years Later - Granger - 1989
26	1989], ‘Modularity and scaling in large phonemic neural networks - Waibel, Sawai, et al.
25	Theoretical foundations of linear and order statistics combiner for neural pattern classifiers - Tumer, Ghosh - 1995
20	Approximating a function and its derivatives using MSE-optimal linear combinations of trained feedforward neural networks - Hashem, Schmeiser
19	Modular and hierarchical learning systems - Jordan, Jacobs - 1995
17	1996], ‘Bayesian-inference in mixtures-of-experts and hierarchical mixtures- of-experts models with an application to speech recognition - Peng, Jacobs, et al.
13	The Modularity of Mind: An Essay on Faculty - Fodor - 1983
11	Learning ranks with neural networks (Invited paper - Al-Ghoneim, Kumar - 1995
9	Neural nets and diversity - Sharkey, Sharkey, et al. - 1995
6	Error Estimation by Series Association for Neural Network Systems - Kim, Bartlett - 1995
6	An experimental evaluation of independence in multiversion programming - Knight - 1986
6	Searching Weight Space for Backpropagation Solution Types. Current trends - Sharkey, Neary, et al. - 1995
5	Bias/Variance Analyses of Mixtures-of-Experts Architectures - Jacobs - 1997
1	Experiments with a new boosting algorithm, to appear 'Machine - Freund, Schapire - 1996
1	Collinearity and the use of latent root regression for combining GNP forecasts - Jr, B, et al. - 1989
1	An Analysis of Catastrophic - Sharkey - 1995
1	Multimodular architecture for remote sensing operations - Thria, Mejia, et al. - 1992