We study generalization properties of the area under an ROC curve (AUC), a quantity that has been advocated as an evaluation criterion for bipartite ranking problems. The AUC is a different and more complex term than the error rate used for evaluation in classification problems; consequently, existing generalization bounds for the classification error rate cannot be used to draw conclusions about the AUC. In this paper, we define a precise notion of the expected accuracy of a ranking function (analogous to the expected error rate of a classification function), and derive distribution-free probabilistic bounds on the deviation of the empirical AUC of a ranking function (observed on a finite data sequence) from its expected accuracy. We derive both a large deviation bound, which serves to bound the expected accuracy of a ranking function in terms of its empirical AUC on a test sequence, and a uniform convergence bound, which serves to bound the expected accuracy of a learned ranking function in terms of its empirical AUC on a training sequence. Our uniform convergence bound is expressed in terms of a new set of combinatorial parameters that we term the bipartite rank-shatter coefficients; these play the same role in our result as do the standard shatter coefficients (also known variously as the counting numbers or growth function) in uniform convergence results for the classification error rate. We also compare our result with a recent uniform convergence result derived by Freund et al. (2003) for a quantity closely related to the AUC; as we show, the bound provided by our result is considerably tighter. 1 1

Contract No. NSF 0-5824 For the problem of classification into one of two normal populations (univariate and also multivariate), minimax, admissible, unbiased, consistent rules are obtained under the homoscedasticity assumption and with a loss function based on Mahalanobis-distancej lower bounds are obtained for the probability of correct classification under the maximum likelihood. rule. Admissible rules are obtained for classification into several normal populations, in particular, when the populations are identified as different cells of a statistical design. Non-parametric classification rules, based on distance functions between distribution functions and also on Wilcoxon-statistic, are proposed and their consistencies are shown. ·,·e

A criterion is proposed for classifying multivariate "observations" according to their populations of origin when the observable data are the distances between pairs of "observations, " with these distances themselves subject to further variation, such as measurement error. The same basIc problem is investigated under s'everalassumptions on the underlying normal distributions. In each case, the criterion is shown to be a particular quadratic form in normal variables. In the. simplest case considered, a computational form for the distribution is given. An asymptotic expansion is developed which provides an·approximation to the distribution in other cases. The accuracy of the approximation is investigated numerically. The related problem of estimation of the noncentrality parameter of a noncentral chi-squared random variable is also investigated. An estimator is proposed which is based on the two-sample Wilcoxon statistic, using independent samples from the central and noncentral chisquared distributions. The estimator has the property that it is invariant under monotonic transformations of the observed data. Further properties of the estimator are derived and its asymptotic relative efficiency with respect to the maximum likelihood estimator is investigated numerically.