## Manifold based local classifiers: Linear and nonlinear approaches (2007)

Venue: | In Pattern Recognition in review |

Citations: | 6 - 1 self |

### BibTeX

@INPROCEEDINGS{Cevikalp07manifoldbased,

author = {Hakan Cevikalp and Diane Larlus and Marian Neamtu and Bill Triggs and Frederic Jurie and H. Cevikalp and D. Larlus and M. Neamtu and B. Triggs and F. Jurie},

title = {Manifold based local classifiers: Linear and nonlinear approaches},

booktitle = {In Pattern Recognition in review},

year = {2007}

}

### OpenURL

### Abstract

Abstract In case of insufficient data samples in highdimensional classification problems, sparse scatters of samples tend to have many ‘holes’—regions that have few or no nearby training samples from the class. When such regions lie close to inter-class boundaries, the nearest neighbors of a query may lie in the wrong class, thus leading to errors in the Nearest Neighbor classification rule. The K-local hyperplane distance nearest neighbor (HKNN) algorithm tackles this problem by approximating each class with a smooth nonlinear manifold, which is considered to be locally linear. The method takes advantage of the local linearity assumption by using the distances from a query sample to the affine hulls of query’s nearest neighbors for

### Citations

4023 |
Convex Optimization
- Boyd, Vandenberghe
- 2004
(Show Context)
Citation Context ...the following quadratic programming problem: min a i ðÞ 1 s:t: XK 2 xq Xia i m1 ðÞ 2 a i m 1; ai m 0 ; m 1; ...; K; ð9Þ where Xi is the matrix whose columns are the nearest neighbors in V K i xq =-=[19]-=-. Once the optimal coefficient vector a i ðÞ* is found, xq Xia i ðÞ* determines the distance from xq to the local convex hull of the class i. This is repeated for each class and the query is assigned ... |

1781 |
A global geometric framework for nonlinear dimensionality reduction, Science 290 (5500
- Tenenbaum, Silva, et al.
- 2000
(Show Context)
Citation Context ... tasks and dimensionality reduction hasJ Sign Process Syst (2010) 61:61–73 63 emerged as a fruitful line of research. Significant improvements have been achieved using task specific distance metrics =-=[6, 16, 17]-=-. For instance, for Euclidean distance, HKNN achieved the best recognition rate among all of the methods discussed in [6], but it was outperformed by SVM-KNN trained using alternative distance metrics... |

1716 | Nonlinear Dimensionality Reduction by Locally Linear Embedding
- Roweis, LK
- 2000
(Show Context)
Citation Context ...ction [10–12]. Hinton et al. [10] introduced Mixtures of Principal Component Analyzers to approximate the underlying nonlinear data manifolds for classification of handwritten digits. Roweis and Saul =-=[11]-=- proposed Locally Linear Embedding, in which nonlinear structure of the highdimensional data is approximated by exploiting the linear reconstructions. Similarly, Verbeek [12] combined several locally ... |

1135 | Nonlinear component analysis as a kernel eigenvalue problem
- Scholkopf, Smola, et al.
- 1998
(Show Context)
Citation Context ... x i m m T ; ð11Þ where μ is the mean of all nearest neighbors. The kernel trick can be used to map the data into a higher-dimensional space as in the Kernel Principal Component Analysis (Kernel PCA) =-=[20]-=- approach, but the HKNN algorithm uses the null spaces of the local covariance matrices not range spaces. Therefore, we need to modify HKNN to work in terms of dot products of the mapped samples in =.... |

627 | Visual categorization with bags of keypoints
- Csurka, Dance, et al.
- 2004
(Show Context)
Citation Context ... experiments we chose 40 classes from the database. The selected categories are shown in Fig. 4. We represent the image samples using the “bag of features” representation. Introduced by Csurka et al. =-=[29]-=-, such representations have been widely applied for both object classification and localization. In our approach, salient image patches are chosen at different positions and scales using a multi-scale... |

496 | Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories
- Fei-Fei, Fergus, et al.
- 2004
(Show Context)
Citation Context ...tion set is not available. We search the range of values between 2 and 20 for K in the methods employing affine and convex hulls, and between 10 and 100 for SVM-KNN except for the Caltech101 database =-=[23]-=- where K is respectively set to 300 and 200 for the linear and nonlinear kernel functions. 4.1 Experiments on the United States Postal Service (USPS) Database Figure 3 Some samples from the USPS datab... |

409 | Local features and kernels for classification of texture and object categories: A comprehensive study
- Zhang, lek, et al.
(Show Context)
Citation Context ... tasks and dimensionality reduction hasJ Sign Process Syst (2010) 61:61–73 63 emerged as a fruitful line of research. Significant improvements have been achieved using task specific distance metrics =-=[6, 16, 17]-=-. For instance, for Euclidean distance, HKNN achieved the best recognition rate among all of the methods discussed in [6], but it was outperformed by SVM-KNN trained using alternative distance metrics... |

276 | Think globally, fit locally: unsupervised learning of low dimensional manifolds
- Saul, Roweis
- 2003
(Show Context)
Citation Context ...manifold can be approximated by locally linear structures such as affine or convex hulls. However this requires the manifold to be well sampled for modeling local structures correctly. As reported in =-=[32]-=-, manifolds can be considered as well sampled if each sample in a class has on the order of 2l neighbors where l is the dimensionality of the underlying manifold of the class. The local approaches yie... |

258 | Discriminant adaptive nearest neighbor classification and regression
- Hastie, Tibshirani
- 1996
(Show Context)
Citation Context ... surface owing to random variations in sampling density, and this reduces the generalization performance [2–4]. Various methods have been proposed in the literature to overcome this pitfall [2–6]. In =-=[3]-=-, the authors perform a local linear discriminant analysis on the nearest neighbors to deform the Euclidean distance. However, this method is not suitable for high-dimensional classification tasks as ... |

226 | SVM-KNN: Discriminative nearest neighbor classification for visual category recognition
- Zhang, Berg, et al.
- 2006
(Show Context)
Citation Context ...ly and it assumes the classes to be locally linearly separable, which is not always true in practice. Similarly, Zhang et al. proposed a local classifier for the high-dimensional case, called SVM-KNN =-=[6]-=-, without deforming the distance metric. This method trains SVMs on the nearest samples using various distance functions and uses them to classify the query sample. While using various distance metric... |

207 | Spectral grouping using the nystrom method
- Fowlkes, Belongie, et al.
(Show Context)
Citation Context ...n histograms where the χ 2 distance between two histograms I1 ðu1; ...; udÞ and I2 ðv1; ...; vdÞ is CSDðI1; I2Þ 1 P 2 d ðumvmÞ m1 2. h i ðumþvmÞ .Thisfunction satisfies the Mercer’s conditions =-=[31]-=-, so we do not need to perturb the kernel matrix. The nearest neighbors around the query points are also found using χ 2 distance for all nonlinear local approaches. To assess the recognition accuracy... |

155 | Modeling the Manifolds of Images of Handwritten Digits
- Hinton, Dayan, et al.
- 1997
(Show Context)
Citation Context ...rse training data giving significant improvements in recognition accuracies. Linearity assumptions for local regions are also widely used for nonlinear dimensionality reduction [10–12]. Hinton et al. =-=[10]-=- introduced Mixtures of Principal Component Analyzers to approximate the underlying nonlinear data manifolds for classification of handwritten digits. Roweis and Saul [11] proposed Locally Linear Embe... |

136 | Transformation invariance in pattern recognition-tangent distance and tangent propogation
- Simard, Cun, et al.
- 1998
(Show Context)
Citation Context ... set is added to the training set, and the classification accuracies are assessed on the allocated test set. Tangent distance has been shown to work well for the classification of hand-written digits =-=[1, 25]-=-. It improves the classification rate by compensating for small spatial affine transformations and changes in the thickness of the pen stroke. We use both the two-sided tangent distance and the Euclid... |

129 | A sparse texture representation using local affine regions
- Lazebnik, Schmid, et al.
(Show Context)
Citation Context ...ches are chosen at different positions and scales using a multi-scale Harris Laplace detector. Then, the chosen patches are described by using the scale invariant feature transform (SIFT) descriptors =-=[30]-=-. Following this process, all descriptors extracted from images are clustered by k-means clustering method and cluster means are considered as visual words forming the visual vocabulary. The size of t... |

103 | Maximum likelihood estimation of intrinsic dimension - Levina, Bickel - 2004 |

78 | Locally adaptive metric nearest neighbour classification
- Domeniconi, Peng, et al.
- 2002
(Show Context)
Citation Context ... high-dimensional spaces when there is a limited amount of data. As a result, the Mahalanobis distance cannot be computed in these situations, which in turn limits application areas of the method. In =-=[8]-=-, the authors introduced the ADAMENN algorithm, in which the Euclidean distance is deformed in a neighborhood of the query points by means of a local feature relevance factor. However, the method requ... |

70 | An algorithm for finding intrinsic dimensionality of data - Fukunaga, Olsen - 1971 |

68 | Discriminative common vectors for face recognition
- Cevikalp, Neamtu, et al.
- 2005
(Show Context)
Citation Context ...ear mapping between the high-dimensional sample space and the low-dimensional manifold. Another application for the local linear manifolds is the identification and matching of faces. Cevikalp et al. =-=[13]-=- projected face images onto a linear manifold removing the within-class variations in the process. Kim and Kittler [14] used K-means clustering to partition each face class into several disjoint sets,... |

50 | Locally linear discriminant analysis for multimodally distributed classes for face recognition with a single model image - Kim, Kittler - 2005 |

46 | Joint Manifold Distance: A New Approach to Appearance-Based Clustering
- Fitzgibbon, Zisserman
- 2003
(Show Context)
Citation Context ...basis vectors are extracted by treating each set as a linear manifold. Finally, locally linear manifolds are used to approximate the nonlinear global structure of each class. Fitzgibbon and Zisserman =-=[15]-=- used linear manifolds in the context of face matching to discover the principal cast of a movie. The many applications of linear manifolds in the context of classification can be attributed in part t... |

43 | Estimating the intrinsic dimension of data with a fractal-based method - Camastra, Vinciarelli |

42 | Zikatanov, “The method of alternating projections and the method of subspace corrections in Hilbert space - Xu, T |

40 | Experiments with an extended tangent distance
- Keysers, Dahmen, et al.
(Show Context)
Citation Context ... set is added to the training set, and the classification accuracies are assessed on the allocated test set. Tangent distance has been shown to work well for the classification of hand-written digits =-=[1, 25]-=-. It improves the classification rate by compensating for small spatial affine transformations and changes in the thickness of the pen stroke. We use both the two-sided tangent distance and the Euclid... |

33 | K-local hyperplane and convex distance nearest neighbor algorithms
- Vincent, Bengio
- 2002
(Show Context)
Citation Context ...spect to the query sample, which may yield unreasonable assignments. Another local method, HKNN, was proposed for high-dimensional data and has been shown to work well in several classification tasks =-=[4, 9]-=-. In this method, each class is modeled as a smooth lowdimensional manifold embedded in the high-dimensional space. In this setting it is reasonable to assume that the manifolds are locally linear. If... |

20 | Lda/svm driven nearest neighbor classification
- Peng, Heisterkamp, et al.
- 2001
(Show Context)
Citation Context ...time high-dimensional classification applications. Furthermore, derivations are done for 2 classes and the method is not generalized for multi-class case, which limits its practical applicability. In =-=[2]-=-, Peng et al. train local linear SVMs, rather than a single global nonlinear one, using the nearest samples, and weigh the distance metric based on the linear separating hyperplanes obtained from thes... |

17 |
The common vector approach and its relation to principal component analysis
- Gulmezoglu, Dzhafarov, et al.
(Show Context)
Citation Context ...ð1=KÞ PK x m1 i m , and rewrite Eq. 1 as: LHK i xq pjp mi þ Pli b m1 i mzim ; bim 2< ; ð2Þ i 1; ::; C; where the set zi 1 ; zi2 ; ...; zi n o is any linear basis of the li difference subspace =-=[18]-=- spanned by the difference vectors xi 1 mi; xi 2 mi; ...; xi K mi , and li K 1 is the dimension of the basis. Note that there is no constraint on the sum of the new coefficients b i m . The difference... |

14 |
Adaptive quasiconformal kernel nearest neighbor classification
- Peng, Heisterkamp, et al.
- 2004
(Show Context)
Citation Context ...deform the Euclidean distance. However, this method is not suitable for high-dimensional classification tasks as there are not enough nearest neighbors for a linear discriminant analysis. Peng et al. =-=[7]-=- proposed the Adaptive Quasiconformal Kernel Nearest Neighbors algorithm which warps the input space based on the local posterior probability estimates and weighted Mahalanobis distance. However, the ... |

13 |
Discriminative common vector method with kernels
- Cevikalp, Neamtu, et al.
- 2006
(Show Context)
Citation Context ...ace of each of the individual-class local scatters, so the projection along this subspace is the same for all classes: only projections within the span of the local total scatter differ among classes =-=[21]-=-. The total squared distance from a query to a class is the sum of the squared distance to the span of the local total scatter (a constant for all classes) and of the squared distance to the span of t... |

11 | Learning Nonlinear Image Manifolds by Global Alignment of Local Linear Models
- Verbeek
- 2006
(Show Context)
Citation Context ...n digits. Roweis and Saul [11] proposed Locally Linear Embedding, in which nonlinear structure of the highdimensional data is approximated by exploiting the linear reconstructions. Similarly, Verbeek =-=[12]-=- combined several locally valid linear manifolds to obtain a global nonlinear mapping between the high-dimensional sample space and the low-dimensional manifold. Another application for the local line... |

4 | Efficient local flexible nearest neighbor classification - Domeniconi, Gunopulos - 2002 |

3 |
Protein fold recognition with K-local hyperplane distance nearest neighbor algorithm
- Olkun
- 2004
(Show Context)
Citation Context ...spect to the query sample, which may yield unreasonable assignments. Another local method, HKNN, was proposed for high-dimensional data and has been shown to work well in several classification tasks =-=[4, 9]-=-. In this method, each class is modeled as a smooth lowdimensional manifold embedded in the high-dimensional space. In this setting it is reasonable to assume that the manifolds are locally linear. If... |

2 | Cevikalp received the B.S. and M.S. degrees from the Electrical and Electronics Engineering Department of Eskisehir Sign Process Syst (2010) 61:61–73 73 Osmangazi University, Eskisehir, Turkey in 1999 and 2001, respectively. He received a Ph.D. degree in - Hakan |