## 2010a Statistical inference after model selection

Venue: | Journal of Quantitative Criminology |

Citations: | 7 - 6 self |

### BibTeX

@ARTICLE{Berk_2010astatistical,

author = {Richard Berk and Lawrence Brown and Linda Zhao},

title = {2010a Statistical inference after model selection},

journal = {Journal of Quantitative Criminology},

year = {}

}

### OpenURL

### Abstract

Conventional statistical inference requires that a model of how the data were generated be known before the data are analyzed. Yet in criminology, and in the social sciences more broadly, a variety of model selection procedures are routinely undertaken followed by statistical tests and confidence intervals computed for a “final ” model. In this paper, we examine such practices and show how they are typically misguided. The parameters being estimated are no longer well defined, and post-model-selection sampling distributions are mixtures

### Citations

445 | The Dantzig selector: statistical estimation when p is much larger than n,” Annals of Statistics - Candès, Tao |

233 |
Generalized Linear Models. Second edition
- McCullagh, JA
- 1989
(Show Context)
Citation Context ...ical tests and confidence intervals can be constructed. This basic framework subsumes a variety of special cases. Popular instances are included under the generalized linear model and its extensions (=-=McCullagh and Nelder, 1989-=-). Logistic regression is common example. Models with more than one regression equation (Greene, 2003: Chapters 14 and 15) are for purposes of this paper also special cases. The ubiquitous application... |

222 |
Specification Searched: Ad Hoc Inference with Non-Experimental Data
- Leamer
- 1978
(Show Context)
Citation Context ...s. The ubiquitous application of regression models in criminology, and in the social sciences more generally, has been criticized from a variety of perspectives for well over a generation (Box, 1976; =-=Leamer, 1978-=-; Rubin, 1986; Freedman, 1987; 2004; Manski, 1990; Breiman, 2001; Berk, 2003; Morgan and Winship, 2007). Despite the real merits of this literature, we assume in this paper that the idea of a “correct... |

110 |
Applied Regression Including Computing and Graphics
- Cook, Weisberg
- 1999
(Show Context)
Citation Context ...regression models through data analysis would seem to make subsequent statistical inference problematic. Yet, model selection is a routine activity and is taught in any number of respected textbooks (=-=Cook and Weisberg, 1999-=-, Greene, 2003). This practice and pedagogy, therefore, would seem to warrant some scrutiny. Is there a problem? If so, is it important? And if so, what can be done about it? What are the consequences... |

74 | Counterfactuals and causal inference: Methods and principles for social research - Morgan, Winship - 2007 |

63 |
Statistical Models: Theory and Practice
- Freedman
- 2005
(Show Context)
Citation Context ...ata on hand were generated. Regressors are canonically treated as fixed, and the model specifies how the realized distribution of the response variable came to be, given the values of the regressors (=-=Freedman, 2005-=-: 42). Causal interpretations can be introduced from information external to the model (Berk, 2003: Chapter 5). Statistical tests and confidence intervals can be constructed. This basic framework subs... |

46 | 2001. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics 29(4 - Benjamini, Yekutieli - 1964 |

46 |
2003) Regression Analysis: A Constructive Critique
- Berk
(Show Context)
Citation Context ...he realized distribution of the response variable came to be, given the values of the regressors (Freedman, 2005: 42). Causal interpretations can be introduced from information external to the model (=-=Berk, 2003-=-: Chapter 5). Statistical tests and confidence intervals can be constructed. This basic framework subsumes a variety of special cases. Popular instances are included under the generalized linear model... |

46 |
Science and Statistics
- Box
- 1976
(Show Context)
Citation Context ...pecial cases. The ubiquitous application of regression models in criminology, and in the social sciences more generally, has been criticized from a variety of perspectives for well over a generation (=-=Box, 1976-=-; Leamer, 1978; Rubin, 1986; Freedman, 1987; 2004; Manski, 1990; Breiman, 2001; Berk, 2003; Morgan and Winship, 2007). Despite the real merits of this literature, we assume in this paper that the idea... |

33 |
Statistical modeling: The two cultures [with discussion
- Breiman
- 2001
(Show Context)
Citation Context ...gy, and in the social sciences more generally, has been criticized from a variety of perspectives for well over a generation (Box, 1976; Leamer, 1978; Rubin, 1986; Freedman, 1987; 2004; Manski, 1990; =-=Breiman, 2001-=-; Berk, 2003; Morgan and Winship, 2007). Despite the real merits of this literature, we assume in this paper that the idea of a “correct model” makes sense and consider statistical inference when the ... |

23 | Pötscher (2005) “Model Selection and Inference: Facts and Fiction,” Econometric Theory 21 - Leeb, M |

21 | On specifying graphical models for causation, and the identification problem - Freedman - 2002 |

15 | Between Prison and Probation: Intermediate Punishments in a Rational Sentencing System - Morris, Tonry - 1990 |

11 |
An ancillarity paradox which appears in multiple linear regression (with discussion
- Brown
- 1990
(Show Context)
Citation Context ...t even have to exploit information from the data being analyzed. The rules may be “ancillary” in the sense that they are constructed independently of the parameters responsible for the realized data (=-=Brown, 1990-=-: 489). 3 For example, suppose the model a researcher selects depends on the day of the week. On Mondays it’s model A, on Tuesdays it’s model B, and so on up to seven different models on seven differe... |

11 |
On the Impact of Variable Selection in Fitting Regression Equations
- Freedman, Navidi, et al.
- 1988
(Show Context)
Citation Context ... model selection process in the first step is a form of data snooping. Standard errors conventionally estimated under such circumstances are well know to be incorrect; 4they are likely to too small (=-=Freedman et al., 1988-=-). False statistical power can result. In effect, there is an opportunity to look at all the face-down cards before a bet is placed. 2 For most thoughtful crime and justice researchers, this is old ne... |

10 | Comparative statistical inference, second edition - Barnett - 1982 |

6 | Nonparametric bounds on treatment effects - F - 1990 |

5 |
The multilevel context of criminal sentencing: Integrating judge and county level influences in the study of courtroom decision making. Criminology 44:259–98
- Johnson
(Show Context)
Citation Context ...encing. Determinants of post-conviction sentences have long been of interest to criminologists and to researchers from other disciplines who study sanctions (Blumstein et al., 1983; Wooldredge, 2005; =-=Johnson, 2006-=-). Probation decisions have received considerable attention (Morris and Tonry, 1980; Petersilia, 1997). When a decision is made to place an individual on probation, one might be interested in the fact... |

5 | 2005), “The Impact of Incarceration - Cho, Lalonde |

5 | Pötscher (2006) “Can one Estimate the Conditional Distribution of PostModel-Selection Estimators?” The Annals of Statistics 34(5 - Leeb, M |

3 | Pötscher (2008) “Model Selection - Leeb, M |

3 |
The conditional level of the F-test
- Olshen
- 1973
(Show Context)
Citation Context ...e more fundamental. It has long been recognized by some that when any parameter estimates are discarded, the sampling distribution of the remaining parameter estimates can be distorted. (Brown, 1967; =-=Olshen, 1973-=-). The rules by which some parameters are discarded do not even have to exploit information from the data being analyzed. The rules may be “ancillary” in the sense that they are constructed independen... |

2 |
The conditional level of Student’s t
- Brown
- 1967
(Show Context)
Citation Context ...essed here are more fundamental. It has long been recognized by some that when any parameter estimates are discarded, the sampling distribution of the remaining parameter estimates can be distorted. (=-=Brown, 1967-=-; Olshen, 1973). The rules by which some parameters are discarded do not even have to exploit information from the data being analyzed. The rules may be “ancillary” in the sense that they are construc... |

1 | Dedel (2006) “Violence Screening - Davies, K |

1 | Tibshinani (2007) “Discussion: The Dantzig Selector: Statistical Estimation with p Much Larger than n,” The Annals of Statistics 35(6 - Efron, Hastie |

1 | Correlation and Large-Scale Simultaneous Significance Testing,”Journal of the American Statistical Association 102(477) 93 - unknown authors - 2007 |

1 |
Econometric Methods, fifth edition
- Greene
- 2003
(Show Context)
Citation Context ...opular instances are included under the generalized linear model and its extensions (McCullagh and Nelder, 1989). Logistic regression is common example. Models with more than one regression equation (=-=Greene, 2003-=-: Chapters 14 and 15) are for purposes of this paper also special cases. The ubiquitous application of regression models in criminology, and in the social sciences more generally, has been criticized ... |

1 | 2008) “Déjà vu All Over Again: Investigating Temporal Continuity of Adolescent Victimization - Ousey, Wilcox, et al. |

1 |
Probation in the United
- Petersilia
- 1997
(Show Context)
Citation Context ... to researchers from other disciplines who study sanctions (Blumstein et al., 1983; Wooldredge, 2005; Johnson, 2006). Probation decisions have received considerable attention (Morris and Tonry, 1980; =-=Petersilia, 1997-=-). When a decision is made to place an individual on probation, one might be interested in the factors that could affect the length of the suspended incarceration sentence. Suspended sentence length c... |

1 |
Which Ifs Have Causal Answers.” American Statistical Association 81: 961–962
- Rubin
- 1986
(Show Context)
Citation Context ...ous application of regression models in criminology, and in the social sciences more generally, has been criticized from a variety of perspectives for well over a generation (Box, 1976; Leamer, 1978; =-=Rubin, 1986-=-; Freedman, 1987; 2004; Manski, 1990; Breiman, 2001; Berk, 2003; Morgan and Winship, 2007). Despite the real merits of this literature, we assume in this paper that the idea of a “correct model” makes... |