Abstract

Abstract--In this paper we investigate the use of the area under the receiver operating characteristic (ROC) curve (AUC) as a performance measure for machine learning algorithms. As a case study we evaluate six machine learning algorithms (C4.5, Multiscale Classifier, Perceptron, Multi-layer Perceptron, k-Nearest Neighbours, and a Quadratic Discriminant Function) on six "real world " medical diagnostics data sets. We compare and discuss the use of AUC to the more conventional overall accuracy and find that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities. The paper concludes with the recommendation that AUC be used in preference to overall accuracy for "single number " evaluation of machine

Citations

4072	JR: C4.5: Programs for Machine Learning - Quinlan
3144	Classification and Regression Trees - Breiman, Friedman, et al. - 1984
2307	Learning internal representations by error propagation - Rumelhart, Hinton, et al. - 1986
2109	Introduction to Statistical Pattern Recognition - Fukunaga - 1990
531	Pattern Recognition: A statistical approach - PA, Kittler - 1982
528	Bootstrap methods: another look at the jackknife - Efron - 1979
511	Signal detection theory and psychophysics - Green, JA - 1966
349	The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve - HANLEY, MCNEIL - 1982
192	Pattern Recognition Principles - Tou, Gonzalez - 1974
113	Multisurface method of pattern separation for medical diagnosis applied to breast cytology - Wolberg, Mangasarian - 1990
80	Probability and Statistics for Engineers and Scientists - Walpole, Myers - 1978
76	Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights - Nguyen, Widrow - 1990
70	R.S.,‖ Using the ADAP learning algorithm to forecast the onset of diabetes mellitus - Everhart, Dickson, et al.
61	An overview of predictive learning and function approximation - Friedman - 1994
60	Decision estimation and classification: An introduction to pattern recognition and related topics - Therrien - 1989
57	A method of comparing the areas under receiver operating characteristic curves derived from the same cases - Hanley, McNeil - 1983
43	International application of a new probability algorithm for the diagnosis of coronary artery disease - Detrano, Janosi, et al. - 1989
25	Learning Internal Representations by Error - Rumelhart, Hinton, et al. - 1986
19	ROC analysis applied to the evaluation of medical imaging techniques - Swets - 1979
18	Computer Systems that Learn: Classi cation and Prediction Methods from - Weiss, Kulikowski - 1991
13	Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals. Rating method data - DD, Alf - 1969
13	Classification of cervical cell nuclei using morphological segmentation and textural feature extraction - Walker, Jackway, et al. - 1994
9	tting Avoidance as Bias - er, \Over - 1993
8	Introduction to Computational Learning and Statistical Prediction - Friedman
6	The Multiscale Classifier - Lovell, Bradley - 1996
6	Discrimination and Classi cation - Hand - 1981
5	Computer Systems that Learn: Classification and - Weiss, Kulikowski - 1991
5	A Toolbox for the Analysis and Visualisation of Sensor Data in Supervision," Universidade Nova de Lisboa, Intelligent Robots Group Technical report - Rauber, Barata, et al. - 1993
2	Pickets, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory - Swets, M - 1982
2	Decision Estimation and Classi cation: An Introduction to Pattern Recognition and related Topics - Therrien - 1989
2	Longsta , \Classi cation of Cervical Cell Nuclei Using Morphological Segmentation and Textural Feature Extraction - Walker, Lovell, et al. - 1994
2	The Multiscale Classi er - Lovell, Bradley - 1996
2	Power Curves for Pattern Classi cation Networks - Twomey, Smith - 1993
1	Relationship of Platelet Aggregation to Bleeding after Cardiopulmonary Bypass," Annals Thoractic Surgery 57 - Ray, Just, et al. - 1994
1	International application of a new probability algorithm for the diagnosis of coronary artery disease - Schmid, Sandu, et al. - 1989
1	Models o1" incremental conceot formation, Artif lntell - Gennari, Langley, et al. - 1989
1	A Toolbox for the Analysis and Visualisation - Rauber, Barata, et al. - 1993
1	l)iscrimination andClassi[~cation - Hand - 1981
1	The use of the area under the ROC curve in the evaluation 1159 - Minsky, Papert, et al. - 1969
1	Principles qfNeurodynamies - Rosenblatt - 1961
1	Power curves lot pattern classification networks - Twomey, Smith - 1993
1	Fundamental Concepts in the Design O/ E~periments - Hicks - 1993
1	On the Methodology for Comparing Learning Algorithms: A Case Study - Bradley, Lovell, et al. - 1994
1	Overfitting aw)idance as bias - Schaffer - 1993
1	On deflection as a performance criterion in dection, IEEE Trans. Aerospace Electronic Systems 31. 1072 1081 - PicinBono - 1995
1	Models of Incremental Conceot Formation - Gennari, Langley, et al. - 1989
1	ROC Curves for Classi cation Trees," Medical Decision Making 14 - Raubertas, Humiston, et al. - 1994
1	On De ection as a Performance Criterion in Dection," IEEE Transactions on Aerospace and Electronic Systems 31 - PicinBono - 1995