We show how to learn a Mahanalobis distance metric for k-nearest neighbor (kNN) classification by semidefinite programming. The metric is trained with the goal that the k-nearest neighbors always belong to the same class while examples from different classes are separated by a large margin. On seven data sets of varying size and difficulty, we find that metrics trained in this way lead to significant improvements in kNN classification—for example, achieving a test error rate of 1.3 % on the MNIST handwritten digits. As in support vector machines (SVMs), the learning problem reduces to a convex optimization based on the hinge loss. Unlike learning in SVMs, however, our framework requires no modification or extension for problems in multiway (as opposed to binary) classification. 1

In this paper we study how to improve nearest neighbor classification by learning a Mahalanobis distance metric. We build on a recently proposed framework for distance metric learning known as large margin nearest neighbor (LMNN) classification. Our paper makes three contributions. First, we describe a highly efficient solver for the particular instance of semidefinite programming that arises in LMNN classification; our solver can handle problems with billions of large margin constraints in a few hours. Second, we show how to reduce both training and testing times using metric ball trees; the speedups from ball trees are further magnified by learning low dimensional representations of the input space. Third, we show how to learn different Mahalanobis distance metrics in different parts of the input space. For large data sets, the use of locally adaptive distance metrics leads to even lower error rates. 1.

The central thesis of this work explores how necessary linear feature extraction is, with regard to leading approaches for classification and alignment in computer vision. Linear filters are frequently employed as a preprocessing step before optimizing some learning goal in vision such as classification or alignment. Often the choice of these filters involve both: (i) large computational and memory requirements due to increased feature size, and (ii) heuristic assumptions about what filters work best for specific applications (e.g., Gabor filters, edge filters, Haar filters, etc.). A central concept of our work is that if our learning goal can be expressed as an ℒ2 norm, and our feature extraction step linear, then the sequential feature extraction and optimization steps can be subsumed within a single learning goal. This alternative view of linear feature extraction with respect to an ℒ2 learning goal has a number of advantages. First, for the case of classification within the well known linear support vector machine (SVM) framework, the memory and computational overheads, typically occurring due to the high dimensionality of the feature extraction process, now disappear. From a theoretical perspective the

Abstract not found