## AUC optimization vs. error rate minimization (2004)

### Cached

### Download Links

- [books.nips.cc]
- [books.nips.cc]
- [books.nips.cc]
- DBLP

### Other Repositories/Bibliography

Venue: | in Advances in Neural Information Processing Systems |

Citations: | 102 - 2 self |

### BibTeX

@INPROCEEDINGS{Cortes04aucoptimization,

author = {Corinna Cortes and Mehryar Mohri},

title = {AUC optimization vs. error rate minimization},

booktitle = {in Advances in Neural Information Processing Systems},

year = {2004},

publisher = {MIT Press}

}

### Years of Citing Articles

### OpenURL

### Abstract

The area under an ROC curve (AUC) is a criterion used in many applications to measure the quality of a classification algorithm. However, the objective function optimized in most of these algorithms is the error rate and not the AUC value. We give a detailed statistical analysis of the relationship between the AUC and the error rate, including the first exact expression of the expected value and the variance of the AUC for a fixed error rate. Our results show that the average AUC is monotonically increasing as a function of the classification accuracy, but that the standard deviation for uneven distributions and higher error rates is noticeable. Thus, algorithms designed to minimize the error rate may not lead to the best possible AUC values. We show that, under certain conditions, the global function optimized by the RankBoost algorithm is exactly the AUC. We report the results of our experiments with RankBoost in several datasets demonstrating the benefits of an algorithm specifically designed to globally optimize the AUC over other existing algorithms optimizing an approximation of the AUC or only locally optimizing the AUC. 1

### Citations

3928 |
Classification and Regression Trees
- Breiman, Friedman, et al.
- 1984
(Show Context)
Citation Context ...n ROC curve [8]. The AUC value for a reasonable classifier is above 0.5 and closer to 1.0. The AUC value is equivalent to the Wilcoxon-Mann-Whitney statistic [8] and closely related to the Gini index =-=[1]-=-. It has been re-invented under the name of L-measure by [11], as correctly pointed out by [2], and slightly modified under the name of Linear Ranking by [13, 14]. Recently, algorithms have been propo... |

869 |
Signal detection theory and psychophysics
- Green, Swets
- 1966
(Show Context)
Citation Context ...lue is exactly the probability P(X > Y ) where X is the random variable corresponding to the distribution of the outputs for the positive examples and Y the one corresponding to the negative examples =-=[7]-=-. The Wilcoxon-Mann-Whitney statistic is clearly the expression of that probability in the discrete case, which proves the lemma [8]. Thus, the AUC can be viewed as a measure based on pairwise compari... |

543 |
The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology
- Hanley, McNeil
- 1982
(Show Context)
Citation Context ...le of an ROC curve. The line connecting (0,0) and (1,1), corresponding to random classification, is drawn for reference. A convenient summary statistic of the ROC curve is the area under an ROC curve =-=[8]-=-. The AUC value for a reasonable classifier is above 0.5 and closer to 1.0. The AUC value is equivalent to the Wilcoxon-Mann-Whitney statistic [8] and closely related to the Gini index [1]. It has bee... |

524 | An efficient boosting algorithm for combining preferences - Freund, Iyer, et al. - 2003 |

260 | Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions
- Provost, Fawcett
- 1997
(Show Context)
Citation Context ...r many years especially for medical decision-making. Over the last few years, they have found increased interest in the machine learning and data mining communities for model evaluation and selection =-=[12, 10, 4, 9, 15, 2]-=-. The ROC curve for a binary classification problem is based on plotting the true-positive (TP) rate as a function of the false-positive (FP) rate, where the rates are defined as follows: TP rate = co... |

229 |
Signal Detection Theory and ROC Analysis
- Egan
- 1975
(Show Context)
Citation Context ...d. A criterion closely related to ranking is that of the area under an ROC curve (AUC). The Receiver Operating Characteristics curves (ROC curves) were originally developed in signal detection theory =-=[3]-=- in connection with radio signals, and have been used for many years especially for medical decision-making. Over the last few years, they have found increased interest in the machine learning and dat... |

123 |
A decision theoretic generalization of on-line learning and an application to boosting
- Freund, Schapire
- 1995
(Show Context)
Citation Context ...optimization is based combining a number of weak rankings. For our experiments, we chose as weak rankings threshold rankers with the range {0, 1}, similar to the boosted stumps often used by AdaBoost =-=[6]-=-. We used the so-called Third Method of RankBoost for selecting the best weak ranker. According to this method, at each step, the weak threshold ranker is selected so as to maximize the AUC of the wei... |

93 | Learning decision trees using the area under the ROC curve
- Ferri, Flach, et al.
(Show Context)
Citation Context ...r many years especially for medical decision-making. Over the last few years, they have found increased interest in the machine learning and data mining communities for model evaluation and selection =-=[12, 10, 4, 9, 15, 2]-=-. The ROC curve for a binary classification problem is based on plotting the true-positive (TP) rate as a function of the false-positive (FP) rate, where the rates are defined as follows: TP rate = co... |

64 | Simonoff “Tree Induction vs. Logistic Regression: A Learning-Curve Analysis
- Perlich, Provost, et al.
(Show Context)
Citation Context ...r many years especially for medical decision-making. Over the last few years, they have found increased interest in the machine learning and data mining communities for model evaluation and selection =-=[12, 10, 4, 9, 15, 2]-=-. The ROC curve for a binary classification problem is based on plotting the true-positive (TP) rate as a function of the false-positive (FP) rate, where the rates are defined as follows: TP rate = co... |

18 |
Measuring lift quality in database marketing”, ACM SIGKDD Explorations Newsletter 2(2) (2000), p. 76–80. tel-00482649, version 1
- Piatetsky-Shapiro, Steingold
- 1998
(Show Context)
Citation Context ...s above 0.5 and closer to 1.0. The AUC value is equivalent to the Wilcoxon-Mann-Whitney statistic [8] and closely related to the Gini index [1]. It has been re-invented under the name of L-measure by =-=[11]-=-, as correctly pointed out by [2], and slightly modified under the name of Linear Ranking by [13, 14]. Recently, algorithms have been proposed for maximizing the AUC value locally [4] or maximizing ap... |

12 | Optimizing Classifier Performance Via the Wilcoxon-Mann-Whitney Statistics
- Yan, Dodier, et al.
- 2003
(Show Context)
Citation Context |

10 | Evaluation of prediction models for marketing campaigns
- Rosset, Neumann, et al.
- 2001
(Show Context)
Citation Context ... [8] and closely related to the Gini index [1]. It has been re-invented under the name of L-measure by [11], as correctly pointed out by [2], and slightly modified under the name of Linear Ranking by =-=[13, 14]-=-. Recently, algorithms have been proposed for maximizing the AUC value locally [4] or maximizing approximations of the global AUC value [9, 15], but, in general, these algorithms do not obtain AUC val... |

6 | Targeting Customer Groups using Gain and Cost Matrix : a Marketing Application
- Chauchat, Rakotomalala, et al.
- 2001
(Show Context)
Citation Context |

1 |
Prodding the ROC curve
- Mozer, Dodier, et al.
- 2002
(Show Context)
Citation Context |

1 |
Ranking-methods for flexible evaluation and efficient comparison of 2-class models
- Rosset
- 1999
(Show Context)
Citation Context ... [8] and closely related to the Gini index [1]. It has been re-invented under the name of L-measure by [11], as correctly pointed out by [2], and slightly modified under the name of Linear Ranking by =-=[13, 14]-=-. Recently, algorithms have been proposed for maximizing the AUC value locally [4] or maximizing approximations of the global AUC value [9, 15], but, in general, these algorithms do not obtain AUC val... |