We address the problem that occurs when inferences about counterfactuals—predictions, ‘‘what-if’ ’ questions, and causal effects—are attempted far from the available data. The danger of these extreme counterfactuals is that substantive conclusions drawn from statistical models that fit the data well turn out to be based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Yet existing statistical strategies provide few reliable means of identifying extreme counterfactuals. We offer a proof that inferences farther from the data allow more model dependence and then develop easyto-apply methods to evaluate how model dependent our answers would be to specified counterfactuals. These methods require neither sensitivity testing over specified classes of models nor evaluating any specific modeling assumptions. If an analysis fails the simple tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. Free software that accompanies this article implements all the methods developed. 1

Inferences about counterfactuals are essential for prediction, answering ‘‘what if ’ ’ questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based on speculation and convenient but indefensible model assumptions rather than empirical evidence. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, so this problem can be hard to detect. We develop easy-to-apply methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests we offer, then we know that substantive results are sensitive to at least some modeling choices that are not based on empirical evidence. We use these methods to evaluate the extensive scholarly literatures on the effects of changes in the degree of democracy in a country (on any dependent variable) and separate analyses of the effects of UN peacebuilding efforts. We find evidence that many scholars are inadvertently drawing conclusions based more on modeling hypotheses than on evidence in the data. For some research questions, history contains insufficient information to be our guide. Free software that accompanies this paper implements all our suggestions. Social science is about making inferencesFusing facts we know to learn about facts we do not know. Some inferential targets (the facts we do not know) are factual, which means that they exist even if we do not know them. In early 2003, Saddam Hussein was obviously either alive or dead, but the world did not know which it was

Does the U.S. Supreme Court curtail rights and liberties when the nation’s security is under threat? In hundreds of articles and books, and with renewed fervor since September 11, 2001, members of the legal community have warred over this question. Yet, not a single large-scale, quantitative study exists on the subject. Using the best data available on the causes and outcomes of every civil rights and liberties case decided by the Supreme Court over the past six decades and employing methods chosen and tuned especially for this problem, our analyses demonstrate that when crises threaten the nation’s security, the justices are substantially more likely to curtail rights and liberties than when peace prevails. Yet paradoxically, and in contradiction to virtually every theory of crisis jurisprudence, war appears to affect only cases that are unrelated to the war. For these cases, the effect of war and other international crises is so substantial, persistent, and consistent that it may surprise even those commentators who long have argued that the Court rallies around the flag in times of crisis. On the other hand, we find no evidence that cases most directly related to the war are affected. We attempt

Propensity score approaches in outcomes and epidemiological research are most often used for sample selection by matching, analysis of causal effect by stratification, or risk adjustment by combining propensity score and regression models. Several computing tools are available including SAS, S-PLUS/R, and SPSS to develop and implement propensity score approaches in a variety of applications. This paper reviews the history, statistical definitions, and application of the propensity score in non-randomized observational studies, pre-estimation for randomized designs, and poorly randomized experiments. Several distinct propensity score matching methods, both simple and sophisticated are described in detail to enable users to choose the most appropriate solutions to fit their study objectives. Special cases of propensity score applications discussed include multi-treatment studies, multi-control designs, and missing data processing, where the definitions, estimations and utilization of propensity scores are far different from the general, treatment-control approach. Finally, a discussion of the limitations of propensity score applications in health

“Matching ” portfolios is a technique for generating a reasonable benchmark for determining the relative performance of a specific equity portfolio and is based on the work in Ho et al. (2005a). Consider the simplest case of a long-only mutual fund that has returned 10 % in the last year. Has the portfolio done

Abstract not found

conducted a two-year (2007-2009) study to examine the effectiveness of the 2009 Pearson enVisionMATH program in helping elementary students improve their mathematics skills and understanding. While overall results from this national randomized control trial (RCT) showed that students who use the enVisionMATH program perform significantly better than students using other math programs, its impact on minority, English Language Learners (ELLs), and economically disadvantaged students were inconclusive given the small sample of these subgroups that were included. Therefore, in order to more closely

Severe housing shortages and high unemployment prevail in Addis Ababa. The Addis Ababa Integrated Housing Programme (AAI-HDP) is an active labour market programme that attempts to tackle these problems simultaneously by creating and supporting Small and Medium Enterprises to construct housing using low-cost technologies novel for Ethiopia. This paper analyses the employment creation impact of the program and shows it to be negligible. Program participants do bene…t from higher earnings and the program premium is highest for those at the bottom of the income distribution. Paradoxically, the program premium is due to the fact that the AAIHDP has created …rms which are larger than pre-existing construction …rms, though selection on unobservables cannot be ruled out. More generally, this paper, which is (one of the) …rst to analyse the impact of SME support programs on employment and earnings in a developing country context, challenges the case for promoting SMEs to generate employment. This is a very …rst draft: Please do not quote without permission! 1 1

WhatIf is an R package that implements the methods for evaluating counterfactuals introduced in King and Zeng (2006a) and King and Zeng (2006b). It offers easy-to-use techniques for assessing a counterfactual’s model dependence without having to conduct sensitivity testing over specified classes of models. These same methods can be used to approximate the common support of the treatment and control groups in causal inference.

Abstract: Does a state’s use of indiscriminate violence incite insurgent attacks? Nearly all existing theories and empirical studies conclude that such actions only fuel insurgencies by provoking insurgent mobilization. This proposition is tested using a natural experiment that draws on random artillery strikes by Russian forces in Chechnya (2000-05) to estimate the impact of indiscriminate violence on subsequent insurgent violence. A difference-in-difference (DD) estimation method is adopted in which shelled villages are matched with similar non-repressed settlements over identical time periods to estimate treatment effects. The findings are counterintuitive. Shelled villages and their home districts (raiony) exhibit less post-treatment violence than control groups. In addition, commonly-cited “triggers ” for insurgent retaliation, including the lethality and duration of indiscriminate violence, are either insignificant or negatively correlated with insurgent attack propensity. Acknowledgements: Paper prepared for presentation at the Olin Institute for Strategic Studies,