Abstract

Abstract. We describe two important sets of tools for neural network modeling: prediction risk estimation and network architecture selection. Prediction risk is defined as the expected performance of an estimator in predicting new observations. Estimated prediction risk can be used both for estimating the quality of model predictions and for model selection. Prediction risk estimation and model selection are especially important for problems with limited data. Techniques for estimating prediction risk include data resampling algorithms such as nonlinear cross–validation (NCV) and algebraic formulae such as the predicted squared error (PSE) and generalized prediction error (GPE). We show that exhaustive search over the space of network architectures is computationally infeasible even for networks of modest size. This motivates the use of heuristic strategies that dramatically reduce the search complexity. These strategies employ directed search algorithms, such as selecting the number of nodes via sequential network construction (SNC) and pruning inputs and weights via sensitivity based pruning (SBP) and optimal brain damage (OBD) respectively.

Citations

1140	A new look at the statistical model identification - AKAIKE - 1974
989	Spline models for observational data - Wahba - 1990
970	Tibshirani R: Generalized Additive Models - Hastie - 1990
891	Modeling by the Shortest Data Description - Rissanen - 1978
843	Information theory and an extension of the maximum likelihood principle - Akaike - 1973
561	Cross-validatory choice and assessment of statistical prediction - Stone - 1974
537	Neural networks and the bias/variance dilemma - Geman, Bienenstock, et al. - 1992
375	Optimum brain damage - Cun, Denker, et al. - 1990
332	Smoothing noisy data with spline functions - CRAVEN, WAHBA - 1979
235	Estimating the dimension of a model - Schwartz - 1978
201	Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter - Golub, Heath, et al. - 1979
190	Spline Smoothing and Nonparametric Regression - Eubank - 1988
159	The effective number of parameters: an analysis of generalization and regularization in nonlinear learning systems”, in eds - Moody - 1992
148	Second order derivatives for network pruning: Optimal brain surgeon - Hassibi, Stork - 1993
111	A technique for trimming the fat from a network via relevance assessment - Mozer, Smolensky, et al. - 1988
108	Gong G: A leisurely look at the bootstrap, the jackknife, and cross-validation American Statistician 1983, 37:36-48. Page 5 of 6 (page number not for citation purposes) Medical Research Methodology 2003 - Efron
76	Statistical predictor identification - Akaike - 1970
71	The predictive sample reuse method with applications - Geisser - 1974
46	A completely automatic French curve: Fitting spline functions by cross-validation,”Communications in Statistics - Wahba, Wold - 1975
45	Utans J: Principled Architecture Selection for Neural Networks: Application to Corporate Bond Rating Prediction - Moody
44	On entropy maximization principle - Akaike - 1977
37	Note on generalization, regularization, and architecture selection in nonlinear learning systems - Moody - 1991
28	Architecture Selection Strategies for Neural Networks: Application to Bond Rating Prediction - Moody, Utans - 1995
28	J: Selecting neural network architectures via the prediction risk application to corporate bond rating prediction - Utans, Moody - 1991
27	The recurrent cascade-correlation learning algorithm - Fahlman - 1991
27	Cross-validation: A review - STONE - 1977
26	Fast pruning using principal components - Levin, Leen, et al. - 1994
26	Fast learning in multi--resolution hierarchies - Moody - 1989
26	Stock performance modeling using neural networks: A comparative study with regression models - Refenes, Zapranis, et al. - 1994
21	Predicted squared error: a criterion for automatic model selection - Barron - 1984
12	Data analysis, including statistics - Mosteller, Tukey - 1968
10	The Likelihood Ratio, Wald, and Lagrange Multiplier Tests: An Expository Note - Buse - 1982
9	Networks with learned unit response functions - Moody, Yarvin - 1992
8	A Generalization Error Estimate for Nonlinear Systems - Larsen - 1992
4	Dynamic node creation in backpropagation neural networks - Ash - 1989
2	Neural network model selection using asymptotic jackknife estimator and cross-validation method, in - Liu - 1993
1	The effective number of parameters and generalized prediction error for nonlinear regression.’, Manuscript in preparation - Moody - 1995