Abstract. An important component of many data mining projects is finding a good classification algorithm, a process that requires very careful thought about experimental design. If not done very carefully, comparative studies of classification and other types of algorithms can easily result in statistically invalid conclusions. This is especially true when one is using data mining techniques to analyze very large databases, which inevitably contain some statistically unlikely data. This paper describes several phenomena that can, if ignored, invalidate an experimental comparison. These phenomena and the conclusions that follow apply not only to classification, but to computational experiments in almost any aspect of data mining. The paper also discusses why comparative analysis is more important in evaluating some types of algorithms than for others, and provides some suggestions about how to avoid the pitfalls suffered by many experimental studies.

The fragility of the CAPM has led to a resurgence of research that frequently uses trading strategies based on sorting procedures to uncover relations between firm characteristics (such as “value ” or “glamour”) and equity returns. We examine the propensity of these strategies to generate statistically and economically significant profits due to our familiarity with the data. Under plausible assumptions, data-snooping can account for up to 50 percent of the insample relations between firm characteristics and returns uncovered using single (one-way) sorts. The biases can be much larger if we simultaneously condition returns on two (or more) characteristics.

Todd E. Clark is an assistant vice president and economist at the Federal Reserve Bank of Kansas City. He gratefully acknowledges the helpful comments of Mike McCracken and seminar participants at the Federal Reserve Bank of Kansas City. The views expressed are those of the author and not necessarily those of the Federal Reserve Bank of Kansas City or the Federal Reserve System. Clark

There now appears to be overwhelming evidence of stock market predictability. A large body of research shows that excess returns on the aggregate market are forecastable from the default spread, dividend yield, dividend payout, the term spread, consumption data, inflation, industrial production, wealth, and labor income, to name but a few variables. 1 Yet, despite this seemingly overwhelming evidence, there appear to be few real-world investors capable of taking advantage of this time-series predictability, especially at the levels of profits suggested by the academic pre-

Abstract. An important component of many data mining projects is finding a good classification algorithm, a process that requires very careful thought about experimental design. If not done very carefully, comparative studies of classification and other types of algorithms can easily result in statistically invalid conclusions. This is especially true when one is using data mining techniques to analyze very large databases, which inevitably contain some statistically unlikely data. This paper describes several phenomena that can, if ignored, invalidate an experimental comparison. These phenomena and the conclusions that follow apply not only to classification, but to computational experiments in almost any aspect of data mining. The paper also discusses why comparative analysis is more important in evaluating some types of algorithms than for others, and provides some suggestions about how to avoid the pitfalls suffered by many experimental studies.

The purpose of this report is to review the evidence on the profitability of technical analysis. To achieve this purpose, the report comprehensively reviews survey, theoretical and empirical studies regarding technical trading strategies. We begin by overviewing survey studies that have directly investigated market participants ’ experience and views on technical analysis. The survey literature indicates that technical analysis has been widely used by market participants in futures markets and foreign exchange markets, and that about 30 % to 40 % of practitioners appear to believe that technical analysis is an important factor in determining price movement at shorter time horizons up to 6 months. Then we provide an overview of theoretical models that include implications about the profitability of technical analysis. Conventional efficient market theories, such as the martingale model and random walk models, rule out the possibility of technical trading profits in speculative markets, while relatively recent models such as noisy rational expectation models or behavioral models suggest that technical trading strategies may be profitable due to noise in the market or investors ’ irrational behavior. Finally, empirical studies are surveyed. In this report, the empirical literature is categorized into two

Discussion Papers contain preliminary material and research results, and are circulated prior to a full peer review in order to stimulate discussion and critical comment. It is expected that most Discussion Papers will eventually be published in some other form, and that their content may also be revised. Little is known about the impact of social science research in general, and food policy research, in particular. In order to expand the scope of available academic research and to develop quantitative methods for estimating the impact of IFPRI's work, several papers were commissioned from social scientists. Furthermore, IFPRI held an essay contest to solicit research from a broader range of scientists. The resulting papers were discussed at a two-day symposium organized by IFPRI in 1997. This Discussion Paper is a revised version of a paper prepared for and discussed at the symposium. Other papers will be published in this Discussion Paper series over the next months. CONTENTS iii Page

The slope coefficient estimator in predictive regressions for stock returns is biased by a lagged stochastic regressor. There is also a spurious regression if the underlying expected return is highly persistent. This paper studies how the interactions between the two biases affect inferences about the predictability in international equity market returns. The analysis considers how the biases work in the presence of data mining for the predictive variables. I find that the two biases can reinforce or offset each other, depending on the parameters of the model. I present a new correction for the bias in the presence of both effects and evaluate its economic significance. Adjusting for data mining associated with both effects, I find that many of the global predictors have a weak explanatory power when they are individually regressed for the world market return and that 8 of the 18 national equity market returns may have at least one significant predictive variable after the apparent number of searches are accounted for.

An Empirical Re-examination of the Relation

Meetings for their helpful comments and suggestions. Also, we thank James Garret for his excellent