Abstract

Model selection is important in many areas of supervised learning. Given a dataset and a set of models for predicting with that dataset, we must choose the model which is expected to best predict future data. In some situations, such as online learning for control of robots or factories, data is cheap and human expertise costly. Cross validation can then be a highly effective method for automatic model selection. Large scale cross validation search can, however, be computationally expensive. This paper introduces new algorithms to reduce the computational burden of such searches. We show how experimental design methods can achieve this, using a technique similar to a Bayesian version of Kaelbling’s Interval Estimation. Several improvements are then given, including (1) the use of blocking to quickly spot near-identical models, and (2) schemata search: a new method for quickly finding families of relevant features. Experiments are presented for robot data and noisy synthetic datasets. The new algorithms speed up computation without sacrificing reliability, and in some cases are more reliable than conventional techniques. 1

Citations

897	Instance-Based Learning Algorithms - Aha, Kibler, et al. - 1991
515	K: Irrelevant features and the subset selection problem - GH, Kohavi, et al.
280	Learning in embedded systems - Kaelbling - 1993
247	Statistics for experimenters - Box, Hunter, et al. - 1978
183	AJ: Subset selection in regression - Miller - 1990
179	Locally weighted regression – an approach to regression-analysis by local fitting - Cleveland, Devlin - 1988
172	Greedy attribute selection - Caruana, Freitag
122	Prototype and feature selection by sampling and random mutation hill climbing algorithms - Skalak - 1994
72	D.: A study of Instance-Based Algorithms for Supervised Learning Tasks. University of California at Irvine technical report 90-42 - Aha - 1990
71	Hoeffding races: Accelerating model selection search for classification and function approximation - Maron, Moore - 1993
46	A statistical approach to solving the EBL utility problem - Greiner - 1992
45	Utans J: Principled Architecture Selection for Neural Networks: Application to Corporate Bond Rating Prediction - Moody
43	Memory-based approaches to approximating continuous functions - Atkeson - 1992
42	An empirical investigation of brute force to choose features, smoothers, and function approximators - Moore, Hill, et al. - 1992
27	BL: The significance of the difference between two means when the population variances are unequal. Biometrika - Welch - 1938
19	Learning search control knowledge for deep space network scheduling - Gratch, Chien, et al. - 1993
17	robust adaptive control by learning only forward models - Fast - 1992
11	Measuring Uncertainty: An Elementary Introduction to Bayesian Statistics - Schmitt
6	Assessing the Quality of Local Linear Models - Schaal, Atkeson - 1994