Results 1  10
of
28
Combining Estimates in Regression and Classification
 Journal of the American Statistical Association
, 1993
"... We consider the problem of how to combine a collection of general regression fit vectors in order to obtain a better predictive model. The individual fits may be from subset linear regression, ridge regression, or something more complex like a neural network. We develop a general framework for this ..."
Abstract

Cited by 103 (0 self)
 Add to MetaCart
We consider the problem of how to combine a collection of general regression fit vectors in order to obtain a better predictive model. The individual fits may be from subset linear regression, ridge regression, or something more complex like a neural network. We develop a general framework for this problem and examine a recent crossvalidationbased proposal called "stacking" in this context. Combination methods based on the bootstrap and analytic methods are also derived and compared in a number of examples, including best subsets regression and regression trees. Finally, we apply these ideas to classification problems where the estimated combination weights can yield insight into the structure of the problem. 1 Introduction Consider a standard regression setup: we have predictor measurements x i = (x i1 ; . . . x ip ) T and a response measurement y i on N independent training cases. Let z represent the entire training sample. Our goal is derive a function c z (x) that accurately p...
The Covariance Inflation Criterion for Adaptive Model Selection
 J. Roy. Statist. Soc. B
, 1999
"... We propose a new criterion for model selection in prediction problems. The covariance inflation criterion adjusts the training error by the average covariance of the predictions and responses, when the prediction rule is applied to permuted versions of the dataset. This criterion can be applied to g ..."
Abstract

Cited by 40 (0 self)
 Add to MetaCart
We propose a new criterion for model selection in prediction problems. The covariance inflation criterion adjusts the training error by the average covariance of the predictions and responses, when the prediction rule is applied to permuted versions of the dataset. This criterion can be applied to general prediction problems (for example regression or classification), and to general prediction rules (for example stepwise regression, treebased models and neural nets). As a byproduct we obtain a measure of the effective number of parameters used by an adaptive procedure. We relate the covariance inflation criterion to other model selection procedures and illustrate its use in some regression and classification problems. We also revisit the conditional bootstrap approach to model selection. Keywords: model selection, adaptive, permutation, bootstrap, crossvalidation 1 Introduction This article concerns the selection of a prediction rule from a set of training data. The training set z =...
Model Selection by Normalized Maximum Likelihood
, 2005
"... The Minimum Description Length (MDL) principle is an information theoretic approach to inductive inference that originated in algorithmic coding theory. In this approach, data are viewed as codes to be compressed by the model. From this perspective, models are compared on their ability to compress a ..."
Abstract

Cited by 24 (9 self)
 Add to MetaCart
The Minimum Description Length (MDL) principle is an information theoretic approach to inductive inference that originated in algorithmic coding theory. In this approach, data are viewed as codes to be compressed by the model. From this perspective, models are compared on their ability to compress a data set by extracting useful information in the data apart from random noise. The goal of model selection is to identify the model, from a set of candidate models, that permits the shortest description length (code) of the data. Since Rissanen originally formalized the problem using the crude ‘twopart code ’ MDL method in the 1970s, many significant strides have been made, especially in the 1990s, with the culmination of the development of the refined ‘universal code’ MDL method, dubbed Normalized Maximum Likelihood (NML). It represents an elegant solution to the model selection problem. The present paper provides a tutorial review on these latest developments with a special focus on NML. An application example of NML in cognitive modeling is also provided.
HIV1 genotypic resistance patterns predict response to saquinavir ritonavir therapy in patients in whom previous protease inhibitor therapy had failed
, 1999
"... Background: Tests for resistance to HIV drugs are available for clinical use; however, their predictive value has not been fully assessed. Objectives: To determine HIV1 genotypic predictors of a virologic response to saquinavir–ritonavir therapy in patients in whom at least one previous protease ..."
Abstract

Cited by 21 (6 self)
 Add to MetaCart
(Show Context)
Background: Tests for resistance to HIV drugs are available for clinical use; however, their predictive value has not been fully assessed. Objectives: To determine HIV1 genotypic predictors of a virologic response to saquinavir–ritonavir therapy in patients in whom at least one previous protease inhibitor– containing regimen had failed and to compare the predictive value of baseline genotype with that of standard clinical evaluation. Design: Retrospective clinical cohort study. Setting: Universitybased HIV clinic. Patients: 54 HIV1–infected adults treated with saquinavir– ritonavir who had experienced virologic failure while receiving a protease inhibitor–containing regimen for at least 3 months.
Minimizing Statistical Bias with Queries
, 1995
"... I describe an exploration criterion that attempts to minimize the error of a learner by minimizing its estimated squared bias. I describe experiments with locallyweighted regression on two simple kinematics problems, and observe that this "biasonly" approach outperforms the more common ..."
Abstract

Cited by 18 (0 self)
 Add to MetaCart
(Show Context)
I describe an exploration criterion that attempts to minimize the error of a learner by minimizing its estimated squared bias. I describe experiments with locallyweighted regression on two simple kinematics problems, and observe that this "biasonly" approach outperforms the more common "varianceonly" exploration approach, even in the presence of noise.
Efron B: Prevalidation and inference in microarrays
 Statistical Applications in Genetics and Molecular Biology 2002
"... In microarray studies, an important problem is to compare a predictor of disease outcome derived from gene expression levels to standard clinical predictors. Comparing them on the same dataset that was used to derive the microarray predictor can lead to results strongly biased in favor of the microa ..."
Abstract

Cited by 17 (2 self)
 Add to MetaCart
In microarray studies, an important problem is to compare a predictor of disease outcome derived from gene expression levels to standard clinical predictors. Comparing them on the same dataset that was used to derive the microarray predictor can lead to results strongly biased in favor of the microarray predictor. We propose a new technique called “prevalidation ” for making a fairer comparison between the two sets of predictors. We study the method analytically and explore its application in a recent study on breast cancer. 1
Fast and Robust Bootstrap
 STATISTICAL METHODS AND APPLICATIONS
"... In this paper we review recent developments on a bootstrap method for robust estimators which is computationally faster and more resistant to outliers than the classical bootstrap. This fast and robust bootstrap method is, under reasonable regularity conditions, asymptotically consistent. We describ ..."
Abstract

Cited by 13 (3 self)
 Add to MetaCart
In this paper we review recent developments on a bootstrap method for robust estimators which is computationally faster and more resistant to outliers than the classical bootstrap. This fast and robust bootstrap method is, under reasonable regularity conditions, asymptotically consistent. We describe the method in general and then consider its application to perform inference based on robust estimators for the linear regression and multivariate locationscatter models. In particular, we study confidence and prediction intervals and tests of hypotheses for linear regression models, inference for locationscatter parameters and principal components, and classification error estimation for discriminant analysis.
The OutofBootstrap Method for Model Averaging and Selection
, 1996
"... We propose a bootstrapbased method for model averaging and selection that focuses on training points that are left out of individual bootstrap samples. This information can be used to estimate optimal weighting factors for combining estimates from different bootstrap samples, and also for finding t ..."
Abstract

Cited by 9 (0 self)
 Add to MetaCart
We propose a bootstrapbased method for model averaging and selection that focuses on training points that are left out of individual bootstrap samples. This information can be used to estimate optimal weighting factors for combining estimates from different bootstrap samples, and also for finding the best subsets the linear model setting. These proposals provide alternatives to Bayesian approaches to model averaging and selection, requiring less computation and fewer subjective choices. 1 Introduction In this article we use the bootstrap to attempt to "enjoy the Bayesian omelette" without making a mess in the kitchen. We try to mimic Bayesian Cleveland Clinic; srao@bio.ri.ccf.org y Department of Preventive Medicine and Biostatistics, and Department of Statistics, University of Toronto; tibs@utstat.toronto.edu methods for model averaging and selection without having to impose full Bayesian structure. We consider the prediction problem in which we have a training set X = (X 1 ...
Selection of Treebased Classifiers with the Bootstrap 632+ Rule
 Biometrical Journal
, 1997
"... This paper introduces a novel model selection procedure for treebased classifiers. The method is based on the bootstrap 632+ rule recently proposed by Efron and Tibishirani. The rule allows selecting compact, nonoverfitting classification trees by reweighting the contributions of the resubstitutio ..."
Abstract

Cited by 5 (4 self)
 Add to MetaCart
(Show Context)
This paper introduces a novel model selection procedure for treebased classifiers. The method is based on the bootstrap 632+ rule recently proposed by Efron and Tibishirani. The rule allows selecting compact, nonoverfitting classification trees by reweighting the contributions of the resubstitution and standard bootstrap estimated errors. The proposed method is applied in a medical entomology problem for modeling the risk of parasite presence. Keywords: bootstrap 632+, model selection, classification and regression trees 1 Introduction
Bootstrapping Likelihood for Model Selection with Small Samples
, 1998
"... this report we compare modelselection performance of AIC, EIC, a bootstrapsmoothed likelihood crossvalidation (BCV) and its modification (632CV) in smallsample linear regression, logistic regression and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting pro ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
this report we compare modelselection performance of AIC, EIC, a bootstrapsmoothed likelihood crossvalidation (BCV) and its modification (632CV) in smallsample linear regression, logistic regression and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting problem and that BCV may be better than EIC. Hence, the three methods based on bootstrapping the likelihood establish themselves as important alternatives to AIC in model selection with small samples. Key words: AIC; Cox regression; Crossvalidation; EIC; Linear regression; Logistic regression; Maximum likelihood. 1. INTRODUCTION