## Variable selection and Bayesian model averaging in case-control studies (1998)

### Cached

### Download Links

Citations: | 19 - 7 self |

### BibTeX

@MISC{Viallefont98variableselection,

author = {Valerie Viallefont and Adrian E. Raftery and Sylvia Richardson},

title = {Variable selection and Bayesian model averaging in case-control studies},

year = {1998}

}

### Years of Citing Articles

### OpenURL

### Abstract

Covariate and confounder selection in case-control studies is most commonly carried out using either a two-step method or a stepwise variable selection method in logistic regression. Inference is then carried out conditionally on the selected model, but this ignores the model uncertainty implicit in the variable selection process, and so underestimates uncertainty about relative risks. We report on a simulation study designed to be similar to actual case-control studies. This shows that p-values computed after variable selection can greatly overstate the strength of conclusions. For example, for our simulated case-control studies with 1,000 subjects, of variables declared to be "significant" with p-values between.01 and.05, only 49 % actually were risk factors when stepwise variable selection was used. We propose Bayesian model averaging as a formal way of taking account of model uncertainty in case-control studies. This yields an easily interpreted summary, the posterior probability that a variable is a risk factor, and our simulation study indicates this to be reasonably well calibrated in the situations simulated. The methods are applied and compared

### Citations

1235 | Bayesian Data Analysis
- Gelman, Carlin, et al.
(Show Context)
Citation Context ...solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], [16], [17]). For general introductions to Bayesian inference, see [18], [19] and =-=[20]-=-. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assuming that no interaction exists between the risk factors) defined by allowing each of X... |

1039 |
Bayesian Theory
- Bernardo, Smith
- 1994
(Show Context)
Citation Context ...Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], [16], [17]). For general introductions to Bayesian inference, see [18], =-=[19]-=- and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assuming that no interaction exists between the risk factors) defined by allowing ... |

980 | Bayes Factors
- Kass, Raftery
- 1995
(Show Context)
Citation Context ...1 Bayesian Model Averaging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], =-=[13]-=-, [14], [15], [16], [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible ... |

748 |
Applied Logistic Regression
- Hosmer, Lemeshow
- 1989
(Show Context)
Citation Context ...d by substantive considerations. The two most commonly used methods are a two-stage method [4] and backwards stepwise regression; both of these are described, for example, in the influential textbook =-=[5]-=-. Investigators typically carry out tests and compute confidence intervals [6] conditionally on the selected logistic regression model, without taking account of the fact that variable selection has b... |

525 |
Theory of Probability
- Jeffreys
- 1961
(Show Context)
Citation Context ...it has lower posterior model probability. This test has lower total error rate (i.e. sum of Type I and Type II error rates) than any other test on average over the models and the prior distributions (=-=[22]-=-, p. 396), and hence can be viewed as an automatic way of choosing a significance level so as to optimally balance power and significance. This general approach has been used in several previous analy... |

417 | Discrete Multivariate Analysis: Theory and Practice
- Bishop, Fienberg, et al.
- 1975
(Show Context)
Citation Context ...y considered. It is vital not to omit important confounders, which would point towards including all confounders considered, but doing so tends to lead to inefficient estimation, both in theory (e.g. =-=[3]-=-, chap. 9) and in practice [4]. Thus, investigators have tended to use statistical methods to choose among the many confounders indicated by substantive considerations. The two most commonly used meth... |

277 |
Lemeshow S: Applied Logistic Regression
- DW
(Show Context)
Citation Context ...ive considerations. Two commonly used methods are a two-stage method [3] and backwards stepwise regression; both of these are described, for example, in the in uential textbook by Hosmer and Lemeshow =-=[4]-=-. Investigators typically carry out tests and compute con dence intervals conditionally on the selected logistic regression model [5], without taking account of the fact that variable selection has be... |

265 | Models selection and accounting for model uncertainty in graphical models using occam’s window
- Madigan, Raftery
- 1994
(Show Context)
Citation Context ...D METHODS 2.1 Bayesian Model Averaging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], =-=[11]-=-, [12], [13], [14], [15], [16], [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2... |

256 | Bayesian Model Selection in Social Research - Raftery - 1995 |

182 | Specification searches : ad hoc inference with nonexperimental data - Leamer - 1978 |

160 |
Statistical methods in cancer research, volume II: the analysis of cohort studies
- Breslow, Day
- 1993
(Show Context)
Citation Context ...sted effects published in [9] . . . . . . . . . . . . . . . . 14 8 Cervical cancer study analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 i 1 Introduction Case-control studies (=-=[1]-=-, [2]) represent a high proportion of epidemiological practice. For example, at least 49 such studies were published in the American Journal of Epidemiology alone in 1996 (see below). The aim of case-... |

134 |
Statistical methods in cancer research. Volume II--The design and analysis of cohort studies
- NE, NE
- 1987
(Show Context)
Citation Context ... In this paper we consider variable selection in epidemiological case-control studies. Epidemiologists conduct case-control studies in order to test the existence of possible risk factors of interest =-=[1; 2]-=-, and to estimate their association with the presence or absence of a disease, after adjusting for possible confounders. A model often used is logistic regression, namely � � Pr(Y =1) log = 0 + 1X1 + ... |

111 |
Assessment and propagation of model uncertainty (with discussion
- Draper
- 1995
(Show Context)
Citation Context ...sian Model Averaging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], =-=[14]-=-, [15], [16], [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models... |

111 |
Bayesian Statistics: An Introduction
- Lee
- 1987
(Show Context)
Citation Context ...s the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], [16], [17]). For general introductions to Bayesian inference, see =-=[18]-=-, [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assuming that no interaction exists between the risk factors) defined by all... |

108 |
Regression by Leaps and Bounds
- Furnival, Wilson
- 1974
(Show Context)
Citation Context ...tific arguments supporting its use in its own right, and not just as an approximation [11]. The models in Occam's window can be found rapidly using a generalization of the leaps and bounds algorithm (=-=[27]-=-, [12], [28]). Details are given in Appendix. The BMA approach requires the specification of some prior quantities: - the prior probabilities P (M k ) of the models, which will be taken to be equal, s... |

97 | Approximate Bayes factors and accounting for model uncertainty in generalised linear models
- Raftery
- 1993
(Show Context)
Citation Context ...odel Averaging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], =-=[15]-=-, [16], [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assu... |

90 |
Model uncertainty, data mining and statistical inference (with discussion
- ChatfÏeld
- 1995
(Show Context)
Citation Context ...veraging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], =-=[16]-=-, [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assuming t... |

53 | Epidemiologic research: principles and quantitative methods - DG, LL, et al. - 1982 |

48 |
Raftery AE: Bayes factors
- RE
- 1995
(Show Context)
Citation Context ...een 75 per cent and 95 per cent there is positive evidence, between 95 per cent and 99 per cent the evidence is strong, and beyond 99 per cent the evidence is very strong (reference [28], Appendix B; =-=[12]-=-). A Bayesian point estimate of 1 is its posterior mean, given that X1 is in the model, namely E[ 1|D] = � ˆk 1 p(Mk|D) (6) where ˆk 1 is the posterior mean of 1 under model Mk, which is zero if X1 is... |

47 | Model selection and accounting for model uncertainty in linear regression models
- Raftery, Madigan, et al.
- 1997
(Show Context)
Citation Context ...ng 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], [16], =-=[17]-=-). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts by acknowledging that in the situation of equation (1) there are up to K = 2 q possible models (assuming that no... |

39 | Accounting for model uncertainty in survival analysis improves predictive performance (with discussion
- Raftery, E, et al.
- 1996
(Show Context)
Citation Context ...s less than one-twentieth of that of the best model. This device, known as Occam's window [11], reduces the number of models enormously but still seems to provide a good approximation to the full sum =-=[26]-=-. There are also scientific arguments supporting its use in its own right, and not just as an approximation [11]. The models in Occam's window can be found rapidly using a generalization of the leaps ... |

37 |
A study of the impact of confounder selection criteria on effect estimation
- Mickey, Greenland
- 1989
(Show Context)
Citation Context ...to omit important confounders, which would point towards including all confounders considered, but doing so tends to lead to inefficient estimation, both in theory (e.g. [3], chap. 9) and in practice =-=[4]-=-. Thus, investigators have tended to use statistical methods to choose among the many confounders indicated by substantive considerations. The two most commonly used methods are a two-stage method [4]... |

33 |
Epidemiologic research: principles and quantitative methods
- Kleinbaum, Kupper, et al.
- 1982
(Show Context)
Citation Context ... 1). Note that this prior choice can be easily modified by simply changing the value of OE. 2.2 Standard Methods The difficulties of variable selection techniques have been discussed by many authors (=-=[29]-=-, [30], [31], [1], [5]). In order to characterize variable selection approaches commonly used for logistic regression in epidemiology, we reviewed the case-control studies published in the American Jo... |

31 | Bayesian theory - JM, AFM - 1994 |

29 | Bayesian Model Averaging in Proportional Hazard Models: Assessing the Risk of a Stroke
- Volinsky, Madigan, et al.
- 1997
(Show Context)
Citation Context ...nts supporting its use in its own right, and not just as an approximation [11]. The models in Occam's window can be found rapidly using a generalization of the leaps and bounds algorithm ([27], [12], =-=[28]-=-). Details are given in Appendix. The BMA approach requires the specification of some prior quantities: - the prior probabilities P (M k ) of the models, which will be taken to be equal, so as not to ... |

26 |
Statistics in epidemiology: The case control study
- Breslow
- 1996
(Show Context)
Citation Context ...effects published in [9] . . . . . . . . . . . . . . . . 14 8 Cervical cancer study analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 i 1 Introduction Case-control studies ([1], =-=[2]-=-) represent a high proportion of epidemiological practice. For example, at least 49 such studies were published in the American Journal of Epidemiology alone in 1996 (see below). The aim of case-contr... |

24 |
A note on screening regression equations
- Freedman
- 1983
(Show Context)
Citation Context ..., that doing this can yield misleading results, often tending to reject null hypotheses more often than the nominal levels would suggest, and to produce confidence intervals that are too narrow (e.g. =-=[7]-=-, [8]). Here we present a method, Bayesian model averaging, that provides a formal way of taking account of this uncertainty in both tests and confidence intervals. We carry out a simulation study des... |

19 |
Testing precise hypotheses (with discussion
- Berger, Delampady
- 1987
(Show Context)
Citation Context ...a different prior in which the probability is spread over a moderately small interval about zero instead of being concentrated precisely at zero; this interval can be as wide as half a standard error =-=[33]-=-. One strong result to emerge from our simulation study is the difficulty of interpretation of the p-value in classical stepwise and two-step procedures. A smaller than expected proportion of the vari... |

11 | Model Selection for Generalized linear model via GLIB: Application to nutrition and breast cancer
- AE, Richardson
- 1996
(Show Context)
Citation Context ... of uncertainty about the existence of a carryover effect. An epidemiological study of fat and alcohol consumption as risk factors for breast cancer [24] was reanalyzed using Bayesian model averaging =-=[25]-=-. Similar analyses of coronary heart dise ase risk factors and the diagnosis of scrotal swellings have been reported [11]. In their examples, the authors found that out-of-sample predictive performanc... |

10 |
Bayesian methods in practice: experiences in the pharmaceutical industry (with discussion
- Racine, Grieve, et al.
- 1986
(Show Context)
Citation Context ... of choosing a significance level so as to optimally balance power and significance. This general approach has been used in several previous analyses of medical and epidemiological data. Racine et al =-=[23]-=- showed how this method may be used to make inference about a treatment effect in the presence of uncertainty about the existence of a carryover effect. An epidemiological study of fat and alcohol con... |

10 | Modern epidemiology. 2nd edn - KJ, Greenland - 1998 |

9 |
Bayesian model selection in social research
- AE
- 1995
(Show Context)
Citation Context ...l, using the BIC approximation to the Bayes factor and a threshold window with = 100. To do this, an adapted leaps and bounds algorithm is implemented within the bic.logit or bic.glm S-Plus functions =-=[11]-=-. (ii) Then, on the models selected, we use the GLIB approximation to the Bayes factor to re-evaluate more precisely the posterior probabilities of the models kept, and to select a thinner window with... |

7 | Enhancing the predictive performance of Bayesian graphical models
- Madigan, Gavrin, et al.
- 1995
(Show Context)
Citation Context ...d. There are various possible ways other than equal probability to assign prior model probabilities. One approach that seems promising is to elicit prior model probabilities from health professionals =-=[32]-=-. In the results reported in [32], this gave better predictive performance than BMA with equal prior model probabilities. Regression estimates also depend on the prior distribution of fi i , and more ... |

7 |
Subset Selection in Regression
- AJ
- 1990
(Show Context)
Citation Context ...can yield misleading results, often tending to reject null hypotheses more often than the nominal levels would suggest, and to produce con dence intervals that are too narrow (for example, references =-=[6; 7]-=-). Here we present a method, Bayesian model averaging, that provides a formal way of taking account of this uncertainty in both tests and con dence intervals. We illustrate the method with an applicat... |

6 |
Efficiently simulating the coverage properties of interval estimates
- Rubin, Schenker
- 1986
(Show Context)
Citation Context ...e), than any single model that could reasonably have been selected [11]. Second, inferences are well calibrated in the sense that, for example, confidence intervals have the right coverage on average =-=[21]-=-. The third property relates to Bayesian hypothesis testing using Bayes factors, when the null model is rejected against an alternative if it has lower posterior model probability. This test has lower... |

6 |
Cenée S: The role of fat, animal protein and some vitamin consumption in breast cancer: a case control study in southern France
- Richardson, Gerber
- 1991
(Show Context)
Citation Context ... inference about a treatment effect in the presence of uncertainty about the existence of a carryover effect. An epidemiological study of fat and alcohol consumption as risk factors for breast cancer =-=[24]-=- was reanalyzed using Bayesian model averaging [25]. Similar analyses of coronary heart dise ase risk factors and the diagnosis of scrotal swellings have been reported [11]. In their examples, the aut... |

6 |
Model uncertainty, data mining and statistical inference
- eld
- 1995
(Show Context)
Citation Context ...veraging 2.1.1 General Principles Bayesian model averaging (BMA) is the Bayesian solution to the problem of inference in the presence of multiple competing models ([10], [11], [12], [13], [14], [15], =-=[16]-=-, [17]). For general introductions to Bayesian inference, see [18], [19] and [20]. BMA starts byacknowledging that in the situation of equation (1) there are up to K =2q possible models (assuming that... |

5 |
Risk factors for invasive cervical cancer among Latinas and non-Latinas
- RK, Thomas, et al.
(Show Context)
Citation Context .... . . . . . . . . . . . . . . . . . . . . . . . . 12 6 Estimation of Logistic Regression Coefficients: Sums of Squared Errors . . . . . . . . 13 7 Cervical cancer study: Adjusted effects published in =-=[9]-=- . . . . . . . . . . . . . . . . 14 8 Cervical cancer study analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 i 1 Introduction Case-control studies ([1], [2]) represent a high pr... |

4 |
Approximate Bayes factors and accounting for model uncertainty in generalized linear models
- AE
- 1996
(Show Context)
Citation Context ...an be of high dimension. Fortunately, for logistic regression, and indeed for generalized linear models more broadly, an accurate and quite tractable approximation is available via the Laplace method =-=[14]-=-. This can be implemented, and BMA carried out for logistic regression, using the glib software, which runs under S-Plus and is available on the web at the BMA Home Page (www.research.att.com=˜ volins... |

4 | Specifi cation Search: Ad Hoc Inference with Nonexperimental Data - Leamer - 1978 |

3 | Bayesian Statistics: An Introduction - PM - 1989 |

3 | Raftery AE, Kronmal RA. Bayesian model averaging in proportional hazard models: predicting the risk of a stroke. Applied Statistics - CT, Madigan - 1997 |

2 | Hoeting J.A., Model selection and accounting for model uncertainty in linear regression models - Raftery, Madigan - 1997 |

2 |
Accounting for Model Uncertainty in Survival Analysis Improves Predictive Performance (with discussion
- AE, DM, et al.
- 1995
(Show Context)
Citation Context ... new observations, it gives better predictive performance, on average, than any single model that could reasonably have been selected [10]; this theoretical result has been widely veri ed in practice =-=[21]-=-. Second, inferences are well calibrated in the sense that, for example, con dence intervals have the right coverage on average [22]. This general approach has been used in several previous analyses o... |

2 |
Theory of Probability. 3rd edn
- H
- 1961
(Show Context)
Citation Context ...tor, if it is between 75 per cent and 95 per cent there is positive evidence, between 95 per cent and 99 per cent the evidence is strong, and beyond 99 per cent the evidence is very strong (reference =-=[28]-=-, Appendix B; [12]). A Bayesian point estimate of 1 is its posterior mean, given that X1 is in the model, namely E[ 1|D] = � ˆk 1 p(Mk|D) (6) where ˆk 1 is the posterior mean of 1 under model Mk, whic... |

1 |
Statistical significance testing in the american journal of epidemiology
- Savitz, Tolo, et al.
- 1994
(Show Context)
Citation Context ...stage method [4] and backwards stepwise regression; both of these are described, for example, in the influential textbook [5]. Investigators typically carry out tests and compute confidence intervals =-=[6]-=- conditionally on the selected logistic regression model, without taking account of the fact that variable selection has been done. It has been shown, particularly in the linear regression context, th... |

1 |
Subset Selection in Regrerssion
- Miller
- 1990
(Show Context)
Citation Context ...t doing this can yield misleading results, often tending to reject null hypotheses more often than the nominal levels would suggest, and to produce confidence intervals that are too narrow (e.g. [7], =-=[8]-=-). Here we present a method, Bayesian model averaging, that provides a formal way of taking account of this uncertainty in both tests and confidence intervals. We carry out a simulation study designed... |

1 |
Statistics in epidemiology: the case-control study
- NE
- 1996
(Show Context)
Citation Context ... In this paper we consider variable selection in epidemiological case-control studies. Epidemiologists conduct case-control studies in order to test the existence of possible risk factors of interest =-=[1; 2]-=-, and to estimate their association with the presence or absence of a disease, after adjusting for possible confounders. A model often used is logistic regression, namely � � Pr(Y =1) log = 0 + 1X1 + ... |

1 |
The impact of confounder selection criteria on e ect estimation
- Mickey, Greenland
- 1989
(Show Context)
Citation Context ...measurements. Thus, investigators have tended to use statistical methods to choose among the many confounders indicated by substantive considerations. Two commonly used methods are a two-stage method =-=[3]-=- and backwards stepwise regression; both of these are described, for example, in the in uential textbook by Hosmer and Lemeshow [4]. Investigators typically carry out tests and compute con dence inter... |

1 |
Statistical signi cance testing
- DA, Tolo, et al.
- 1970
(Show Context)
Citation Context ...ed, for example, in the in uential textbook by Hosmer and Lemeshow [4]. Investigators typically carry out tests and compute con dence intervals conditionally on the selected logistic regression model =-=[5]-=-, without taking account of the fact that variable selection has been done. It has been shown, particularly in the linear regression context, that doing this can yield misleading results, often tendin... |