## The Journal of Socio-Economics 33 (2004) 527–546 Size matters: the standard error of regressions in the American Economic Review

### Cached

### Download Links

### BibTeX

@MISC{A_thejournal,

author = {Stephen T. Ziliak A and Deirdre N. Mccloskey B},

title = {The Journal of Socio-Economics 33 (2004) 527–546 Size matters: the standard error of regressions in the American Economic Review},

year = {}

}

### OpenURL

### Abstract

Significance testing as used has no theoretical justification. Our article in the Journal of Economic Literature (1996) showed that of the 182 full-length papers published in the 1980s in the American Economic and Review 70 % did not distinguish economic from statistical significance. Since 1996 many colleagues have told us that practice has improved. We interpret their response as an empirical claim, a judgment about a fact. Our colleagues, unhappily, are mistaken: significance testing is getting worse. We find here that in the next decade, the 1990s, of the 137 papers using a test of statistical significance in the AER fully 82 % mistook a merely statistically significant finding for an economically significant finding. A super majority (81%) believed that looking at the sign of a coefficient sufficed for science, ignoring size. The mistake is causing economic damage: losses of jobs and justice, and indeed of human lives (especially in, to mention another field enchanted with statistical significance as against substantive significance, medical science). The confusion between fit and importance is causing false hypotheses to be accepted and true hypotheses to be rejected. We propose a publication standard for the future: “Tell me the oomph of your coefficient; and do not confuse it with merely statistical significance.”

### Citations

596 |
Statistical Methods for Research Workers
- Fisher
- 1932
(Show Context)
Citation Context ...nal of Socio-Economics 33 (2004) 527–546 531 still escape notice if the data are insufficiently numerous to bring them out, but no lowering of the standard of significance would meet this difficulty. =-=Fisher 1925-=- [1941], p. 42; emphasis added). Notice how a standard of “convenience” rapidly became in Fisher’s prose an item to be “formally regarded.” With Fisher there’s no loss function. There’s no thinking be... |

465 | The Federal Funds Rate and the Channels of Monetary Transmission.” American Economic Review 82: 901–21 - Bernanke, Blinder - 1992 |

304 | Myth and Measurement: The New Economics of the Minimum Wage - Card, Krueger - 1995 |

274 | Applied Statistical Decision Theory - Raiffa, Schlaifer - 1961 |

261 | Minimum Wages and Employment: A Case Study of the Fast-Food Industry in - Card, Krueger - 1994 |

174 |
The Rhetoric of Economics
- McCloskey, N
- 1985
(Show Context)
Citation Context ... record (Table 5). 4. We are not original We are not the first social scientists to make the distinction between economic and statistical significance. One of us has been making the point since 1985 (=-=McCloskey, 1985-=-a,542 S.T. Ziliak, D.N. McCloskey / The Journal of Socio-Economics 33 (2004) 527–546 Table 5 If only statistical significance is said to be of importance at its first use (Question 7), then statistic... |

119 | An Empirical Analysis of Cigarette Addiction - Becker, Grossman, et al. - 1994 |

103 | Examining risk preferences under high monetary incentives: experimental evidence from the Peoples - Kachelmeier, Shehata - 1992 |

91 |
The standard error of regressions
- McCloskey, Ziliak
- 1996
(Show Context)
Citation Context ...in “The Standard Error of Regressions,” we showed how significance testing was used during the 1980s in the leading general interest journal of the economics profession, the American Economic Review (=-=McCloskey and Ziliak, 1996-=-). The paper reported results from a 19-item “questionnaire” applied to all of the full-length papers using regression analysis. Of the 182 papers 70% did not distinguish statistical significance from... |

65 | Intergenerational Income Mobility in the United States - Solon - 1992 |

40 | Analyzing Categorical Data - Simonoff - 2003 |

32 | The Significance Test Controversy - Morrison, Henkel - 1970 |

31 |
Introductory Econometrics
- Wooldridge
- 2000
(Show Context)
Citation Context ...995; McCloskey and Ziliak, 1996, p. 99 and numerous other references on pp. 112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; Hamermesh, 1999; Colander, 2000; =-=Wooldridge, 2000-=-, pp. 131–134; Keuzenkamp, 2000, p. 266; Hirshleifer, 2004; Ziliak, 2004; and so forth). Recently, to pick one among the small, bright stream of revisions of standard practice that appear in our mailb... |

27 | The loss function has been mislaid: The rhetoric of significance tests - McCloskey - 1985 |

21 | Probability and statistics for business decisions - Schlaifer - 1959 |

20 | A Tax-Based Test of the Dividend Signalling Hypothesis - Bernheim, Wantz - 1995 |

18 |
The Craft of Labormetrics
- Hamermesh
- 2000
(Show Context)
Citation Context ...231–232; Keuzenkamp and Magnus, 1995; McCloskey and Ziliak, 1996, p. 99 and numerous other references on pp. 112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; =-=Hamermesh, 1999-=-; Colander, 2000; Wooldridge, 2000, pp. 131–134; Keuzenkamp, 2000, p. 266; Hirshleifer, 2004; Ziliak, 2004; and so forth). Recently, to pick one among the small, bright stream of revisions of standard... |

16 |
Basic Issues in Econometrics
- Zellner, Siow
- 1984
(Show Context)
Citation Context ...o reject a hypothesis because the data show “large” departures from the prediction “requires a quantitative criterion of what is to be considered a large departure” (Jeffreys, 1967, p. 384, quoted in =-=Zellner, 1984-=-, p. 277n). Just so. Scientific judgment requires quantitative judgment, not endlessly more machinery. As lovely and useful as the machinery is, at the end, having skillfully used it, the economic sci... |

15 | The Economic Returns to Schooling in the West - Angrist - 1995 |

13 | The insignificance of statistical significance - McCloskey - 1995 |

10 | Problems in Making Policy Inferences from the Coleman Report - Cain, Watts - 1972 |

10 | The Timing and Incidence of Exploratory Drilling on Offshore Wildcat Tracts - Hendricks, Porter - 1996 |

9 | Some Logic and History of Hypothesis Testing - Good - 1981 |

9 |
Probability, Econometrics and Truth
- Keuzenkamp
- 2000
(Show Context)
Citation Context ..., p. 99 and numerous other references on pp. 112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; Hamermesh, 1999; Colander, 2000; Wooldridge, 2000, pp. 131–134; =-=Keuzenkamp, 2000-=-, p. 266; Hirshleifer, 2004; Ziliak, 2004; and so forth). Recently, to pick one among the small, bright stream of revisions of standard practice that appear in our mailboxes, Clinton Greene (2003) has... |

7 |
Reconstruction of educational research
- Shulman
- 1970
(Show Context)
Citation Context ...n the Supreme Court of the United States. Yet some simple souls got it right. Educators have written about the difference between substantive and statistical significance early and late (Tyler, 1931; =-=Shulman, 1970-=-; Carter, 1978). Psychologists have known about the difference for nearly a century, though most of them continue like economists to ignore it (a committee of the American Psychological Association wa... |

6 |
On Tests and Significance in Econometrics
- Keuzenkamp, Magnus
- 1995
(Show Context)
Citation Context ...re not dunces and they have not minced words (Lange, 1959 [1978], pp. 13–15, 133–157 [on page 151 Lange speaks of “practical significance,” his main concern]; Cain and Watts (1970), pp. 229, 231–232; =-=Keuzenkamp and Magnus, 1995-=-; McCloskey and Ziliak, 1996, p. 99 and numerous other references on pp. 112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; Hamermesh, 1999; Colander, 2000; Woo... |

6 |
What is statistical significance
- Tyler
- 1931
(Show Context)
Citation Context ...isleading even the Supreme Court of the United States. Yet some simple souls got it right. Educators have written about the difference between substantive and statistical significance early and late (=-=Tyler, 1931-=-; Shulman, 1970; Carter, 1978). Psychologists have known about the difference for nearly a century, though most of them continue like economists to ignore it (a committee of the American Psychological... |

5 |
Statistics: The Field
- Kruskal
- 1968
(Show Context)
Citation Context ...o badly mixed up, even in the hands of professional statisticians?” “Well,” said Kruskal, who long ago had published in the Encyclopedia a devastating survey on “significance” in theory and practice (=-=Kruskal, 1968-=-a), “I guess it’s a cheap way to get marketable results.” Bingo. Finding statistical significance is simple, and publishing statistically significant coefficients survives at least that market test. B... |

4 | Methods of statistics. Jubilee Volume of the Statistical - Edgeworth - 1885 |

3 |
New Millenium Economics: How Did It Get This Way, and What Way Is It
- Colander
- 2000
(Show Context)
Citation Context ...mp and Magnus, 1995; McCloskey and Ziliak, 1996, p. 99 and numerous other references on pp. 112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; Hamermesh, 1999; =-=Colander, 2000-=-; Wooldridge, 2000, pp. 131–134; Keuzenkamp, 2000, p. 266; Hirshleifer, 2004; Ziliak, 2004; and so forth). Recently, to pick one among the small, bright stream of revisions of standard practice that a... |

3 |
Tests of Statistical Significance
- Kruskal
(Show Context)
Citation Context ...o badly mixed up, even in the hands of professional statisticians?” “Well,” said Kruskal, who long ago had published in the Encyclopedia a devastating survey on “significance” in theory and practice (=-=Kruskal, 1968-=-a), “I guess it’s a cheap way to get marketable results.” Bingo. Finding statistical significance is simple, and publishing statistically significant coefficients survives at least that market test. B... |

2 |
Significance tests, history and logic
- Baird
- 1988
(Show Context)
Citation Context ..., we believe, is the first source of the word “significance” in a context of hypothesis testing. Our earlier paper claimed erroneously that John Venn was first (McCloskey and Ziliak, 1996, p. 97; see =-=Baird, 1988-=-, p. 468). Anyway, the 1880s: for some purposes not a meaningful difference.S.T. Ziliak, D.N. McCloskey / The Journal of Socio-Economics 33 (2004) 527–546 531 still escape notice if the data are insu... |

2 |
Mathematical versus Scientific Significance
- Boring
- 1919
(Show Context)
Citation Context ..., an eminent experimental psychologist, the alarmingly named Edwin Boring, published an article unmasking the confusion between arbitrarily-judged-statistical significance and practical significance (=-=Boring, 1919-=-). And empirical sociology would be less easy for economists to sneer at if more realized that a good many sociologists grasped the elementary statistical point decades before even a handful of the ec... |

2 |
The Case Against Statistical Significance Testing
- Carter
- 1978
(Show Context)
Citation Context ...ourt of the United States. Yet some simple souls got it right. Educators have written about the difference between substantive and statistical significance early and late (Tyler, 1931; Shulman, 1970; =-=Carter, 1978-=-). Psychologists have known about the difference for nearly a century, though most of them continue like economists to ignore it (a committee of the American Psychological Association was recently cha... |

2 | Imprecise Tests and Imprecise Hypotheses - Darnell - 1997 |

2 | Towards Economic Measures of Cointegration and Non-Cointegration. Unpublished paper - Greene - 1999 |

1 | The Likelihood Principle, second ed - Berger, Wolpert - 1988 |

1 | Eds.), Lives of the Laureates - Breit, Spencer - 1990 |

1 |
Alternative approaches to analyzing data
- Friedman, Schwartz
- 1991
(Show Context)
Citation Context ...llurgist what breaking time the equations predicted. That caution proved wise, because the first one of those alloys broke in about 2 h and the second one in about three. Friedman, 1985, reprinted in =-=Friedman and Schwartz, 1991-=-, pp. 48–49. Friedman learned that statistical significance is not the same as metallurgical significance.534 S.T. Ziliak, D.N. McCloskey / The Journal of Socio-Economics 33 (2004) 527–546 The core c... |

1 |
Theory of Probability, Third revised
- Jeffreys
- 1967
(Show Context)
Citation Context ...rold Jeffreys remarked long ago, to reject a hypothesis because the data show “large” departures from the prediction “requires a quantitative criterion of what is to be considered a large departure” (=-=Jeffreys, 1967-=-, p. 384, quoted in Zellner, 1984, p. 277n). Just so. Scientific judgment requires quantitative judgment, not endlessly more machinery. As lovely and useful as the machinery is, at the end, having ski... |

1 |
Introduction to Econometrics. PWN—Polish Scientific Publishers/Pergamon
- Lange
- 1959
(Show Context)
Citation Context ... Hirshleifer, Glen Cain, Gordon Tullock, Gary Solon, Daniel Hamermesh, Thomas Mayer, David Colander, Jeffrey Wooldridge, Jan Magnus, and Hugo Keuzenkamp are not dunces and they have not minced words (=-=Lange, 1959-=- [1978], pp. 13–15, 133–157 [on page 151 Lange speaks of “practical significance,” his main concern]; Cain and Watts (1970), pp. 229, 231–232; Keuzenkamp and Magnus, 1995; McCloskey and Ziliak, 1996, ... |

1 |
Why I Left Alan Greenspan to Seek Economic Significance: The Confessions of an �-Male, Rethinking Marxism, forthcoming
- Ziliak
- 2004
(Show Context)
Citation Context ...112–114; McCloskey’s citations in her works cited; Darnell’s comprehensive review of 1997; Hamermesh, 1999; Colander, 2000; Wooldridge, 2000, pp. 131–134; Keuzenkamp, 2000, p. 266; Hirshleifer, 2004; =-=Ziliak, 2004-=-; and so forth). Recently, to pick one among the small, bright stream of revisions of standard practice that appear in our mailboxes, Clinton Greene (2003) has applied the argument to time-series econ... |