## A fast iterative nearest point algorithm for support vector machine classifier design (2000)

### Cached

### Download Links

Venue: | IEEE Transactions on Neural Networks |

Citations: | 68 - 3 self |

### BibTeX

@ARTICLE{Keerthi00afast,

author = {S. S. Keerthi and S. K. Shevade and C. Bhattacharyya and K. R. K. Murthy},

title = {A fast iterative nearest point algorithm for support vector machine classifier design},

journal = {IEEE Transactions on Neural Networks},

year = {2000},

volume = {11},

pages = {124--136}

}

### Years of Citing Articles

### OpenURL

### Abstract

Abstract—In this paper we give a new fast iterative algorithm for support vector machine (SVM) classifier design. The basic problem treated is one that does not allow classification violations. The problem is converted to a problem of computing the nearest point between two convex polytopes. The suitability of two classical nearest point algorithms, due to Gilbert, and Mitchell et al., is studied. Ideas from both these algorithms are combined and modified to derive our fast algorithm. For problems which require classification violations to be allowed, the violations are quadratically penalized and an idea due to Cortes and Vapnik and Frieß is used to convert it to a problem in which there are no classification violations. Comparative computational evaluation of our algorithm against powerful SVM methods such as Platt's sequential minimal optimization shows that our algorithm is very competitive. Index Terms—Classification, nearest point algorithm, quadratic programming, support vector machine. I.

### Citations

9092 | Statistical Learning Theory
- Vapnik
- 1998
(Show Context)
Citation Context ...erful SVM methods such as Platt's Sequential Minimal Optimization shows that our algorithm is very competitive. 1. Introduction. The last few years have seen the rise of Support Vector Machines (SVMs)=-=[25]-=- as powerful tools mpessk@guppy.mpe.nus.edu.sg y shirish@csa.iisc.ernet.in z cbchiru@csa.iisc.ernet.in x murthy@csa.iisc.ernet.in 1 for solving classification and regression problems[5]. Recently, fas... |

2313 | A tutorial on support vector machines for pattern recognition
- Burges
- 1998
(Show Context)
Citation Context ...achines (SVMs)[25] as powerful tools mpessk@guppy.mpe.nus.edu.sg y shirish@csa.iisc.ernet.in z cbchiru@csa.iisc.ernet.in x murthy@csa.iisc.ernet.in 1 for solving classification and regression problems=-=[5]-=-. Recently, fast iterative algorithms that are also easy to implement have been suggested[20, 12, 19, 8, 23]; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely in... |

2206 | Support-vector networks
- Cortes, Vapnik
- 1995
(Show Context)
Citation Context ...VL-DUAL, with set to . Therefore, any algorithm designed for solving SVM-VL-DUAL can be easily used to solve the dual of SVM-NV, and thereby solve SVM-NV. Using a suggestion made by Cortes and Vapnik =-=[6]-=-, in a recent paper Frieß [9] has explored the use of a sum of squared violations in the cost function s.t. (SVM-VQ) Preliminary experiments by Frieß have shown this formulation to be promising. Unlik... |

1088 |
Practical Methods of Optimization
- FLETCHER
- 1980
(Show Context)
Citation Context ...I [ J.) Here C is a positive, inverse regularization constant that is chosen to give the correct relative weighting between margin maximization and classification violation. Using Wolfe duality theory=-=[5,7]-=- SVM-VL can be transformed to the following equivalent dual problem: max P k ff k \Gamma 1 2 P k P l ff k ff l y k y l z k \Delta z l s:t: 0sff ksC 8k; P k ff k y k = 0; (SVM \Gamma VL \Gamma DUAL) wh... |

1021 |
Fast Training of Support Vector Machines using Sequential Minimal Optimization
- Platt
- 1998
(Show Context)
Citation Context ...These algorithms require enormous matrix storage and do expensive matrix operations. To overcome these problems, recently fast iterative algorithms that are also easy to implement have been suggested =-=[20]-=-, [12], [19], [8], [23]; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVM’s among practitioners. This paper makes another contribu... |

810 |
Estimation of dependences based on empirical data
- Vapnik
- 1982
(Show Context)
Citation Context ...ication and regression problems [5]. A variety of algorithms for solving these problems has emerged. Traditional quadratic programming algorithms [13] and modifications such as the chunking algorithm =-=[26]-=- that make use of the fact that the number of support vectors is usually a small percentage of the total training set have been tried. These algorithms require enormous matrix storage and do expensive... |

492 |
Solving Least Square Problems
- Lawson
- 1974
(Show Context)
Citation Context ...sed to effectively terminate a numerical algorithm for solving NPP. 5. Iterative Algorithms for NPP. NPP has been well studied in the literature, and a number of good algorithms have been given for it=-=[10, 18, 27, 2, 15]-=-. Best general purpose algorithms for NPP such as Wolfe's algorithm[27] terminate within a finite number of steps; however they require expensive matrix storage and matrix operations in each step that... |

472 | Making large-scale support vector machine learning practical
- Joachims
- 1999
(Show Context)
Citation Context ... z cbchiru@csa.iisc.ernet.in x murthy@csa.iisc.ernet.in 1 for solving classification and regression problems[5]. Recently, fast iterative algorithms that are also easy to implement have been suggested=-=[20, 12, 19, 8, 23]-=-; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVMs among practitioners. This paper makes another contribution in this direction. ... |

279 | A fast procedure for computing the distance between complex objects in three-dimensional space
- Gilbert, Johnson, et al.
- 1988
(Show Context)
Citation Context ... of the first algorithms suggested for solving NPP. It was originally devised to solve certain optimal control problems. Later its modifications have found good use in pattern recognition and robotics=-=[27,11]-=-. In this section we briefly describe the algorithm and point to how it can be adapted for SVM classifier design. Let fz k g, I, J , U and V be as in the definition of NPP. Let Z denote the Minkowski ... |

210 | Robust linear programming discrimination of two linearly inseparable sets
- Bennett, Mangasarian
- 1992
(Show Context)
Citation Context ...he g function, i.e., (15). For the sake of uniform comparison, we have used the tighter criteria mentioned above. The following standard problems were used in our testing: Wisconsin Breast Cancer data=-=[26,4]-=-; Two Spirals data[24]; Checkers data[13]; UCI Adult data[21]; and Web page classification data[21, 12]. Except for Checkers data, for which we created a random set of points on a 4 \Theta 4 checkers ... |

163 |
Practical methods of optimization 2nd ed
- Fletcher
- 1991
(Show Context)
Citation Context ... al.: A FAST ITERATIVE NEAREST POINT ALGORITHM FOR SUPPORT VECTOR MACHINE CLASSIFIER DESIGN 125 weighting between margin maximization and classification violation. Using the Wolfe duality theory [5], =-=[7]-=- SVM-VL can be transformed to the following equivalent dual problem: s.t. (SVM-VL-DUAL) where and . It is computationally easy to handle this problem since it is directly based on (kernel) calculation... |

98 |
A Tutorial on Support Vector Regression. NeuroCOLT
- Smola, Schölkopf
- 1998
(Show Context)
Citation Context ... z cbchiru@csa.iisc.ernet.in x murthy@csa.iisc.ernet.in 1 for solving classification and regression problems[5]. Recently, fast iterative algorithms that are also easy to implement have been suggested=-=[20, 12, 19, 8, 23]-=-; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVMs among practitioners. This paper makes another contribution in this direction. ... |

96 |
The Kernel-Adatron Algorithm: A fast and simple learning procedure for Support Vector Machines
- Friess, Cristianini, et al.
- 1998
(Show Context)
Citation Context ... z cbchiru@csa.iisc.ernet.in x murthy@csa.iisc.ernet.in 1 for solving classification and regression problems[5]. Recently, fast iterative algorithms that are also easy to implement have been suggested=-=[20, 12, 19, 8, 23]-=-; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVMs among practitioners. This paper makes another contribution in this direction. ... |

66 | Successive overrelaxation for support vector machines
- Mangasarian, Musicant
- 1999
(Show Context)
Citation Context |

54 |
The AdaTron: an adaptive perceptron algorithm
- Anlauf, Biehl
- 1989
(Show Context)
Citation Context ...compared with that of solving SVM-VQ by our algorithm, we find that our algorithm is competitive. Recently Friess et.al., [8] (deriving inspiration from the Adatron algorithm given by Anlauf and Biehl=-=[1]-=- for designing Hopfield nets) and Mangasarian and Musicant[19] (using a successive overrelaxation idea) independently suggested the inclusion of the extra term, b 2 =2 in the objective functions of th... |

47 |
Convex Sets and Their Applications
- Lay
- 1982
(Show Context)
Citation Context ...any one solution of (9), i.e., s P (j) satisfies h P (j) = s P (j) \Delta j and s P (j) 2 P (10) Now consider the case where P is a convex polytope: Z = fz 1 ; :::;sz r g and P = coZ. It is wellknown =-=[16]-=- that the maximum of a linear function over a convex polytope is attained by an extreme point. This means that h P = hZ and s P = s Z . Therefore h P and s P can be determined by a simple enumeration ... |

46 |
Solving the quadratic programming problem arising in support vector classification
- Kaufman
- 1998
(Show Context)
Citation Context ...ensive matrix storage and matrix operations in each step that makes them unsuitable for use in large SVM design. 1 Iterative 1 It is interesting to point out that the active set method used by Kaufman=-=[13]-=- to solve SVMs is identical to Wolfe's algorithm when both are used to solve SVM-NV. 12 algorithms that need minimal memory (i.e., memory size needed is linear in m, the number of training vectors), b... |

25 |
Finding the nearest point in a polytope
- Wolfe
- 1976
(Show Context)
Citation Context ...sed to effectively terminate a numerical algorithm for solving NPP. 5. Iterative Algorithms for NPP. NPP has been well studied in the literature, and a number of good algorithms have been given for it=-=[10, 18, 27, 2, 15]-=-. Best general purpose algorithms for NPP such as Wolfe's algorithm[27] terminate within a finite number of steps; however they require expensive matrix storage and matrix operations in each step that... |

19 | Geometry in learning
- Bennett, Bredensteiner
- 1998
(Show Context)
Citation Context ... for ffi chosen such that ku ? \Gamma v ? k = 2=kw ? k. A direct, geometrically intuitive proof is given by Sancheti and Keerthi[22] with reference to a geometrical problem in Robotics. Later, Bennett=-=[3]-=- proved a somewhat close result in the context of learning algorithms. Here we only give a discussion that follows the traditional Wolfe-Dual approach employed in the SVM literature. The main reason f... |

9 |
Finding the point of a polyhedron closest to the origin
- Mitchell, Dem’yanov, et al.
- 1974
(Show Context)
Citation Context ...d accuracy for the solution of SVM-NV. The main algorithm of the paper is derived in sections 5 and 6. Two classical algorithms for doing nearest point solution, due to Gilbert[10] and Mitchell et.al.=-=[18]-=-, are combined and modified to derive our fast algorithm. Section 7 concerns the simplification of this algorithm to treat the case corresponding to the inclusion of b 2 =2 in the objective functions ... |

9 | The kernel adatron algorithm: A fast and simple learning procedure for support vector machines
- Frieß, Cristianini, et al.
- 1998
(Show Context)
Citation Context ...equire enormous matrix storage and do expensive matrix operations. To overcome these problems, recently fast iterative algorithms that are also easy to implement have been suggested [20], [12], [19], =-=[8]-=-, [23]; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVM’s among practitioners. This paper makes another contribution in this dire... |

8 |
Minimizing the quadratic form on a convex set
- Gilbert
- 1966
(Show Context)
Citation Context ...s so as to get guaranteed accuracy for the solution of SVM-NV. The main algorithm of the paper is derived in sections 5 and 6. Two classical algorithms for doing nearest point solution, due to Gilbert=-=[10]-=- and Mitchell et.al.[18], are combined and modified to derive our fast algorithm. Section 7 concerns the simplification of this algorithm to treat the case corresponding to the inclusion of b 2 =2 in ... |

7 |
On fast computation of distance between line segments
- Lumelsky
- 1985
(Show Context)
Citation Context ...2 and 4, B = z k . This defines the basic operation of the algorithm. Efficient algorithms for doing nearest point computations involving a triangle and line segments can be found in the appendix and =-=[17]-=-. Tremendous improvement in efficiency can be achieved by doing two things: (1) maintaining and updating caches for some variables; and (2) interchanging operations between the support vector and non-... |

6 |
Computation of certain measures of proximity between convex polytopes: A complexity viewpoint
- Sancheti, Keerthi
- 1992
(Show Context)
Citation Context ...pair of closest points of U and V . Note that w ? = ffi(u ? \Gamma v ? ) for ffi chosen such that ku ? \Gamma v ? k = 2=kw ? k. A direct, geometrically intuitive proof is given by Sancheti and Keerthi=-=[22]-=- with reference to a geometrical problem in Robotics. Later, Bennett[3] proved a somewhat close result in the context of learning algorithms. Here we only give a discussion that follows the traditiona... |

5 |
A tutorial on support vector regression,” Royal-Holloway
- Smola, Schölkopf
- 1998
(Show Context)
Citation Context ...e enormous matrix storage and do expensive matrix operations. To overcome these problems, recently fast iterative algorithms that are also easy to implement have been suggested [20], [12], [19], [8], =-=[23]-=-; Platt's SMO algorithm [20] is an important example. Such algorithms are bound to widely increase the popularity of SVM’s among practitioners. This paper makes another contribution in this direction.... |

3 |
An efficient computational procedure for a generalized quadratic programming problem
- Barr
- 1969
(Show Context)
Citation Context ...sed to effectively terminate a numerical algorithm for solving NPP. 5. Iterative Algorithms for NPP. NPP has been well studied in the literature, and a number of good algorithms have been given for it=-=[10, 18, 27, 2, 15]-=-. Best general purpose algorithms for NPP such as Wolfe's algorithm[27] terminate within a finite number of steps; however they require expensive matrix storage and matrix operations in each step that... |

3 | Support vector networks: The kernel adatron with bias and soft-margin - Friess - 1998 |

1 |
Adult and Web Datasets. http://www.research.microsoft.com/~jplatt
- Platt
(Show Context)
Citation Context ...we have used the tighter criteria mentioned above. The following standard problems were used in our testing: Wisconsin Breast Cancer data[26,4]; Two Spirals data[24]; Checkers data[13]; UCI Adult data=-=[21]-=-; and Web page classification data[21, 12]. Except for Checkers data, for which we created a random set of points on a 4 \Theta 4 checkers grid, all other data sets were downloaded from the internet s... |

1 |
Support Vector Networks: The Kernel Adatron with Bias and Soft-Margin
- Frieß
- 1998
(Show Context)
Citation Context ...ore, any algorithm designed for solving SVM-VL-DUAL can be easily used to solve the dual of SVM-NV, and thereby solve SVM-NV. Using a suggestion made by Cortes and Vapnik [6], in a recent paper Frieß =-=[9]-=- has explored the use of a sum of squared violations in the cost function s.t. (SVM-VQ) Preliminary experiments by Frieß have shown this formulation to be promising. Unlike SVM-VL, here there is no ne... |