Abstract

Function approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest{descent minimization. A general gradient{descent \boosting" paradigm is developed for additive expansions based on any tting criterion. Specic algorithms are presented for least{squares, least{absolute{deviation, and Huber{M loss functions for regression, and multi{class logistic likelihood for classication. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such \TreeBoost" models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classication, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire 1996, and Frie...

Citations

6696	The Nature of Statistical Learning Theory - Vapnik - 1995
4071	C4.5: Programs for machine learning - Quinlan - 1993
3143	Classification and Regression Trees - Breiman, Friedman, et al. - 1984
1325	Experiments with a new boosting algorithm - Freund, Schapire - 1996
968	Tibshirani R: Generalized Additive Models - Hastie
896	Additive logistic regression: a statistical view of boosting - Friedman, Hastie, et al. - 2000
896	BD: Pattern Recognition and Neural Networks - Ripley - 1996
583	Learning representations by back-propagating errors - Rumelhart, Hinton, et al. - 1986
561	Y: Improved boosting algorithms using confidence-rated predictions - RE, Singer - 1999
231	Robust estimation of a location parameter - Huber - 1964
229	Radial basis functions for multivariable interpolation.A review. IMAConference on Algorithms for the Approximationof Functions and Data,1:143167(1985 - Powell
199	Soft margins for AdaBoost - Ratsch, Onoda, et al. - 1998
192	Multivariate Adaptive Regression Splines (with discussion - Friedman - 1991
113	Prediction games and arcing algorithms - Breiman - 1997
79	nonlinear wavelet methods for recovery of signals, densities, and spectra from indirect and noisy data - Donoho - 1993
55	Improving regressors using boosting techniques - Drucker - 1997
51	MJ: The Visual Design and Control of Trellis Displays - Becker, WS, et al. - 1996
33	Matching pursuit with time frequency dictionaries - Mallat, Zhang
32	Improved boosting algorithms using con predictions - Schapire, Singer - 1998
27	Pasting bites together for prediction in large data sets and on-line - Breiman - 1996
20	A geometric approach to leveraging weak learners - Duffy, Helmbold - 1999
13	Regression, prediction and shrinkage (with discussion - Copas - 1983
1	Cr--Pyrope garnets in lithospheric mantle - Griffin, Fisher, et al. - 1997
1	A mathematical model for medical diagnosis { application to congenital heart disease - Warner, Toronto, et al. - 1961