## The dangers of extreme counterfactuals (2006)

### Cached

### Download Links

Venue: | Political Analysis |

Citations: | 17 - 7 self |

### BibTeX

@ARTICLE{King06thedangers,

author = {Gary King and Langche Zeng},

title = {The dangers of extreme counterfactuals},

journal = {Political Analysis},

year = {2006},

volume = {14},

pages = {131--159}

}

### OpenURL

### Abstract

We address the problem that occurs when inferences about counterfactuals—predictions, ‘‘what-if’ ’ questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easyto-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed. 1

### Citations

5119 | Neural Networks for Pattern Recognition - Bishop - 1995 |

2201 | The Elements of Statistical Learning - Hastie, Tibshirani, et al. - 2001 |

537 | Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme,” Review of Economic Studies - Heckman, Ichimura, et al. - 1997 |

527 | Commputational Geometry in C - O’Rourke - 1997 |

400 | Characterizing Selection Bias using Experimental Data - Heckman, Ichimura, et al. - 1998 |

231 | Making the Most of Statistical Analyses: Improving Interpretation and Presentation
- King, Tomz, et al.
- 2000
(Show Context)
Citation Context ...nals.permissions@oxfordjournals.org 131s132 Gary King and Langche Zeng Whether such effects are calculated via analytical derivation or what is now the more common approach of statistical simulation (=-=King et al. 2000-=-), political scientists have made much progress in learning how to make sophisticated methods speak directly to their substantive research questions. Although this represents considerable progress in ... |

157 |
Causal diagrams for epidemiologic research. Epidemiology 1999;10:37–48. Baldness and Myocardial Infarction in Men 683 Am J Epidemiol 2008;167:676–683 at Pennsylvania State U niversity on February 27, 2014 http://aje.oxfordjournals.org/ D ow nloaded from
- Greenland, Pearl, et al.
(Show Context)
Citation Context ...iolated. This is the familiar omitted variable bias, which can plague any model. It can also be due to controlling for irrelevant variables in certain situations, so Z should be minimally sufficient (=-=Greenland et al. 1999-=-).sThe Dangers of Extreme Counterfactuals 147 Since endogeneity bias and selection bias can be written as omitted variable bias, Do encompasses these problems as well. To be specific, endogeneity bias... |

126 | Identification and estimation of causal effects of multiple treatments under the conditional independence assumption
- Lechner
- 2001
(Show Context)
Citation Context ...interested in the average causal effect for the entire set of observations, c 5 EðY1ÞÿEðY0Þ; ð12Þ 13 The analysis of treatments with more than two levels follows analogously (e.g., Imai and Dyk 2004; =-=Lechner 1999-=-). We focus on the binary case for expository purposes.sThe Dangers of Extreme Counterfactuals 145 where both terms have a counterfactual element, since each expectation is taken over all dyads, but Y... |

68 | The Consequences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment - Rosenbaum - 1984 |

66 | Convex sets - Valentine - 1964 |

63 |
The Estimation of Causal Effects from Observational Data.” Annual Review of Sociology
- Winship, Morgan
- 1999
(Show Context)
Citation Context ...ithout a specified functional form via matching, inverse propensity score weighting, or nonparametric smoothing (e.g., Rosenbaum and Rubin 1984; Heckman, Ichimura, and Todd 1998; Robins 1999a, 1999b; =-=Winship and Morgan 1999-=-). Ultimately, whatever method of adjustment is used, the two multivariate densities of Z for the control and treatment groups need to be the same for interpolation bias to be eliminated. We provide f... |

55 |
Bayesian Model Averaging: A Tutorial.” Statistical Science 14(4): 382–401
- Hoeting, others
- 1999
(Show Context)
Citation Context ...tivity analysis, Bayesian model averaging, committee methods, or transdimensional Markov Chains are also useful, but only when it is feasible to identify the relevant class of models for exploration (=-=Hoeting et al. 1999-=-; Imai and King 2004; Sisson 2005) In most situations, the approach we recommend, which does not require choosing classes of models or specifying or estimating any models at all, should be of wider ap... |

47 |
Bounds on multivariate polynomials and exponential error estimates for multiquadric interpolation
- Madych, Nelson
- 1992
(Show Context)
Citation Context ...rther than the Lipschitz condition, then model dependence can be shown to depend even more strongly or in specific ways on the distance between the counterfactual and observed data (see, for example, =-=Madych and Nelson 1992-=-; Wu and Schaback 1993; and Shaback 1996). The same qualitative conclusion would, of course, still hold. 4 Ordinal explanatory variables are typically assumed interval or coded as a set of dichotomous... |

43 | Causal inference with general treatment regimes: Generalizing the propensity score - Imai, Dyk - 2004 |

42 |
On the complexity of d-dimensional voronoi diagrams
- Klee
- 1980
(Show Context)
Citation Context ...ing even on very fast computers. The core problem is that the number of facets (or ‘‘hyperfacets’’) a highdimensional convex hull can have is on the order of n k/2 for k variables and n observations (=-=Klee 1980-=-), so even listing all the facets would be impractical. Problems with more than eight or nine variables appear to have been attempted only rarely, if ever. Of course, restricting ourselves in this way... |

37 | MatchIt: Nonparametric Preprocessing for Parametric Causal Inference
- Ho, Imai, et al.
- 2006
(Show Context)
Citation Context ... an estimated propensity score is close to being true is to check whether it balances the Z distributions, the theory of the propensity score from the point of view of the researcher is tautological (=-=Ho et al. 2005-=-). For the task of identifying the extrapolation region or nonoverlap in the support of Z j D 5 1andZ j D 5 0, using the propensity score involves not only what (Ho et al. 2005) call the Propensity Sc... |

34 |
Marginal Structural Models Versus Structural Nested Models as Tools for Causal Inference," Statistical Models in Epidemiology: The Environment and Clinical
- Robins
- 1999
(Show Context)
Citation Context ... is observed only for democratic dyads and Y 0 only for nondemocratic dyads. These definitions of causal effects are used in a wide variety of literatures (Rubin 1974; Holland 1986; King et al. 1994; =-=Robins 1999-=-a, 1999b; Pearl 2000). A counterfactual x in this context therefore takes the form of some observed data with only one element changed—for example, the Mexico-Spain dyad with all its attributes fixed ... |

33 |
Estimating Causal Effects of Treatments
- RUBIN
- 1974
(Show Context)
Citation Context ...expectation is taken over all dyads, but Y 1 is observed only for democratic dyads and Y 0 only for nondemocratic dyads. These definitions of causal effects are used in a wide variety of literatures (=-=Rubin 1974-=-; Holland 1986; King et al. 1994; Robins 1999a, 1999b; Pearl 2000). A counterfactual x in this context therefore takes the form of some observed data with only one element changed—for example, the Mex... |

24 |
State Failure Task Force Report
- Esty
- 1998
(Show Context)
Citation Context ...he effect of a democratic form of government (as compared to less democratic forms)? We study counterfactuals relating to the degree of democracy using data collected by the State Failure Task Force (=-=Esty et al. 1998-=-). See King and Zeng (2002) for an independent evaluation. This dataset is among the best ever collected in this area. The task force’s dependent variable is the onset of state failure, but our analys... |

20 | The proximity of an individual to a population with applications to discriminant analysis
- Cuadras, Fortiana, et al.
- 1997
(Show Context)
Citation Context ...ables, the concept is well defined for any number of dimensions. For three explanatory variables, and 6 The geometric variability is also known as the generalized variance (Cuadras and Fortiana 1995; =-=Cuadras et al. 1997-=-) and is what we would refer to as the squared generalized standard deviation. It is a generalized version of the usual variance definition in that for Euclidean distances (which are inappropriate wit... |

20 | Transdimensional Markov Chains: A Decade of Progress and Future Perspectives - Sisson - 2005 |

19 | A continuous metric scaling solution for a random variable
- Cuadras, Fortiana
- 1995
(Show Context)
Citation Context ...ll for two explanatory variables, the concept is well defined for any number of dimensions. For three explanatory variables, and 6 The geometric variability is also known as the generalized variance (=-=Cuadras and Fortiana 1995-=-; Cuadras et al. 1997) and is what we would refer to as the squared generalized standard deviation. It is a generalized version of the usual variance definition in that for Euclidean distances (which ... |

16 | Improving Forecasts of State Failure', World Politics 53(4): 623−658 - King, Zeng - 2001 |

9 |
2006) “WhatIf: R Software for Evaluating Counterfactuals
- Stoll, King, et al.
(Show Context)
Citation Context ...ed up the result. We also offer easy-to-use software, ‘‘WhatIf: Software for Evaluating Counterfactuals,’’ that automates our algorithm and implements the other methods discussed in this article (see =-=Stoll et al. 2005-=-). 11 2.4 Democracy Counterfactuals We now apply these methods of evaluating counterfactuals to address one of the most asked questions in political science: what is the effect of a democratic form of... |

8 |
Truth’ is Stranger than Prediction, More Questionable Than Causal Inference
- King
- 1991
(Show Context)
Citation Context ... 5 Z. If this assumption holds, then Dp 5 0 in Eq. (24) vanishes. Post-treatment variable bias is a large and often overlooked component of bias in estimating causal effects in political science (see =-=King 1991-=-; King et al. 1994, pp. 173ff). It is known in the statistical literature but is assumed away in most models and decompositions (Frangakis and Rubin 2002). This decision may be reasonable in other fie... |

3 |
Party Competition and Media Messages in U.S. Presidential Election Campaigns
- Gelman, King
- 1994
(Show Context)
Citation Context ...ted.s134 Gary King and Langche Zeng 2 Forecasts and ‘‘What-If’’ Questions Although statistical technology sometimes differs for making forecasts and estimating the answers to what-if questions (e.g., =-=Gelman and King 1994-=-), the logic is sufficiently similar that we consider them together here. In regression-type models, including least squares, logit, probit, event counts, duration models, and most others used in the ... |

3 | A General coefficient of similarity - Gower - 1971 |

3 | Approximation by Radia Basis Functions with Finitely Many Centers.’’ Constructive Approximation 12:331–340 - Schaback - 1996 |

1 |
The Dangers of Extreme Counterfactuals 159
- Holland
- 1986
(Show Context)
Citation Context ...is taken over all dyads, but Y 1 is observed only for democratic dyads and Y 0 only for nondemocratic dyads. These definitions of causal effects are used in a wide variety of literatures (Rubin 1974; =-=Holland 1986-=-; King et al. 1994; Robins 1999a, 1999b; Pearl 2000). A counterfactual x in this context therefore takes the form of some observed data with only one element changed—for example, the Mexico-Spain dyad... |