Abstract

While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

Citations

2529	C.J.E.: Uci repository of machine learning databases, http://www.ics.uci.edu/ ∼ mlearn/MLRepository.html - Blake, Keogh
420	Robust locally weighted regression and smoothing scatterplots - Cleveland - 1979
269	The case against accuracy estimation for comparing induction algorithms - Provost, Fawcett, et al. - 1998
251	Bagging, boosting, and C4.5 - Quinlan - 1996
239	A simple sequentially rejective multiple test procedure - Holm
217	Individual comparisons by ranking methods - Wilcoxon - 1945
147	A Comparison of Classifiers and Document Representations for the Routing Problem - Schutze, Hull, et al. - 1995
141	Handbook of parametric and nonparametric statistical procedures - Sheskin - 1997
120	On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach - Salzberg - 1997
115	Inference for the Generalization Error - Bengio, Nadeau - 1999
95	A sharper Bonferroni procedure for multiple tests of significance. Biometrika - Hochberg - 1988
73	The use of ranks to avoid the assumption of normality implicit in the analysis of variance - FRIEDMAN - 1937
72	Multiboosting: A technique for combining boosting and wagging - Webb - 2000
70	Curk T: Orange: From experimental machine learning to interactive data mining - Demšar, Zupan, et al.
63	The earth is round (P <:05 - Cohen - 1994
49	Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological methods - Schmidt - 1996
38	Multi-interval discretization of continuous valued attributes for classification learning - Fayyad, Irani - 1993
36	Multiple hypothesis testing - Shaffer - 1995
27	Choosing between two learning algorithms based on calibrated tests - Bouckaert - 2003
25	A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75 - Hommel - 1988
21	Multiple Comparisons Among Means - Dunn - 1961
17	Crafting papers on machine learning - Langley - 2000
16	No unbiased estimator of the variance of K-fold cross-validation - Bengio, Grandvalet - 2004
14	Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms - Bouckaert, Frank - 2004
12	Distribution-free multiple comparisons,” unpublished doctoral dissertation - Nemenyi - 1963
9	The use of ROC curves in test performance evaluation - Schultz - 1986
8	Information Retrieval Using Statistical Classification - Hull - 1994
7	Statistical Methods and Scientific Inference, 2nd. edition (Oliver and - Fisher - 1959
6	2004. Estimating replicability of classifier learning experiments - Bouckaert
5	Comparing Individual Means in the Analysis of Variance - Tukey - 1949
3	Modern data analysis: A first course in applied statistics - Hamilton - 1990
2	Multiple comparison procedures applied to model selection - Pizarro, Guerrero, et al.
1	On the application of three modified Bonferroni procedures to pairwise multiple comparisons in balanced repeated measures designs - Holland - 1991