Abstract

Receiver Operating Characteristics (ROC) graphs are a useful technique for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been increasingly adopted in the machine learning and data mining research communities. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. This article serves both as a tutorial introduction to ROC graphs and as a practical guide for using them in research.

Citations

3143	Classification and Regression Trees - Breiman, Friedman, et al. - 1984
349	The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve - HANLEY, MCNEIL - 1982
325	AP: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn - Bradley - 1997
320	The quickhull algorithm for convex hulls - Barber, Dobkin, et al. - 1996
269	The case against accuracy estimation for comparing induction algorithms - Provost, Fawcett, et al. - 1998
224	MetaCost: A general method for making classifiers cost sensitive. The - Domingos - 1999
192	Measuring the accuracy of diagnostic systems - SWETS - 1988
179	The foundations of cost-sensitive learning - Elkan - 2001
142	Adaptive fraud detection - Fawcett - 1997
77	Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers - Zadrozny, Elkan
76	Evaluating text categorization - Lewis - 1991
74	Machine learning for the detection of oil spills in satellite radar images - Kubat, Holte, et al. - 1998
49	Fawcett T: Robust classification systems for imprecise environments - Provost - 1998
32	Using rule sets to maximize roc performance - Fawcett - 2001
30	Well-trained pets: Improving probability estimation trees - Provost, Domingos - 2000
28	Note on the location of optimal classifiers in n-dimensional ROC space - Srinivasan - 1999
22	Comparing classifiers when the misallocation costs are uncertain - Adams, Hand - 1999
22	Signal Detection Theory and ROC - Egan - 1975
13	Repairing Concavities in ROC Curves - Flach, Wu - 2005
12	Signal detection theory: Valuable tools for evaluating inductive learning - Spackman - 1989
9	A rulelearning program in high-energy physics event classification - Clearwater, Stern - 1991
7	Combining data mining and machine learning for e↑ective user pro↓ling - Fawcett, Provost - 1996
7	Representation quality in text classification: An introduction and experiment - Lewis - 1990
7	2001, ‘Robust Classification for Imprecise Environments - Provost, Fawcett
6	A Method for Discovering the Insignificance of One's Best Classifier and the Unlearnability of a Classification Task - Forman - 2002
5	Better decisions through science,” Scientific American - Swets, Monahan - 2000
4	Confidence bands for ROC curves - Macskassy, Provost, et al. - 2004
4	Learning in the "Real World - Saitta, Neri - 1998
3	Receiver operating characteristic (ROC) literature research’. Online bibliography available from http://splweb.bwh.harvard.edu:8000/pages/ ppl/zou/roc.html. ROC101.tex; 16/03/2004; 12:56; p.37 38 Tom Fawcett Algorithm 7 Generating an ROC curve from a deci - Zou - 2004
1	Holte: 2000, `Explicitly Representing Expected Cost: An alternative to ROC representation - Drummond, C
1	Holte: 2002, `Classifier cost curves: Making performance evaluation easier and more informative'. Unpublished manuscript available from the authors - Drummond, C
1	Till: 2001, `A simple generalization of the area under the ROC curve to multiple class classification problems - Hand, J
1	2002, `Personal communication - Holte
1	Someren: 2000, `CoIL Challenge 2000: The Insurance Company Case - Putten, van