## The Forgetron: A kernel-based perceptron on a fixed budget (2005)

### Cached

### Download Links

- [www.cs.huji.ac.il]
- [www.cs.huji.ac.il]
- [www.cs.huji.ac.il]
- [ttic.uchicago.edu]
- [books.nips.cc]
- [www.cs.huji.ac.il]
- DBLP

### Other Repositories/Bibliography

Venue: | In Advances in Neural Information Processing Systems 18 |

Citations: | 37 - 5 self |

### BibTeX

@INPROCEEDINGS{Dekel05theforgetron:,

author = {Ofer Dekel and Shai Shalev-shwartz and Yoram Singer},

title = {The Forgetron: A kernel-based perceptron on a fixed budget},

booktitle = {In Advances in Neural Information Processing Systems 18},

year = {2005},

pages = {259--266},

publisher = {MIT Press}

}

### Years of Citing Articles

### OpenURL

### Abstract

The Perceptron algorithm, despite its simplicity, often performs well on online classification problems. The Perceptron becomes especially effective when it is used in conjunction with kernels. However, a common difficulty encountered when implementing kernel-based online algorithms is the amount of memory required to store the online hypothesis, which may grow unboundedly. In this paper we describe and analyze a new infrastructure for kernel-based learning with the Perceptron while adhering to a strict limit on the number of examples that can be stored. We first describe a template algorithm, called the Forgetron, for online learning on a fixed budget. We then provide specific algorithms and derive a unified mistake bound for all of them. To our knowledge, this is the first online learning paradigm which, on one hand, maintains a strict limit on the number of examples it can store and, on the other hand, entertains a relative mistake bound. We also present experiments with real datasets which underscore the merits of our approach. 1

### Citations

10096 | Statistical Learning Theory
- Vapnik
- 1998
(Show Context)
Citation Context ...und. In addition to the formal results, we also present experiments with real datasets which underscore the merits of our approach. 1 Introduction The introduction of the Support Vector Machine (SVM) =-=[8]-=- sparked a widespread interest in kernel methods as a means of solving (binary) classification problems. Although SVM was initially stated as a batch-learning technique, it significantly influenced th... |

2265 | Learning With Kernels
- Sholkopf, Smola
- 2002
(Show Context)
Citation Context ...antly influenced the development of kernel methods in the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron [6], ROMMA [5], ALMA [3], NORMA =-=[4]-=-, Ballseptron [7], and the Passive-Aggressive family of algorithms [1]. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification function incrementally, by... |

897 | The perceptron: a probabilistic model for information storage and organization
- Rosenblatt
- 1958
(Show Context)
Citation Context ...-learning technique, it significantly influenced the development of kernel methods in the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron =-=[6]-=-, ROMMA [5], ALMA [3], NORMA [4], Ballseptron [7], and the Passive-Aggressive family of algorithms [1]. Each of these algorithms observes examples in a sequence of rounds, and constructs its classific... |

91 | A new approximate maximal margin classification algorithm
- Gentile
- 2001
(Show Context)
Citation Context ...it significantly influenced the development of kernel methods in the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron [6], ROMMA [5], ALMA =-=[3]-=-, NORMA [4], Ballseptron [7], and the Passive-Aggressive family of algorithms [1]. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification function increm... |

75 | The relaxed online maximum margin algorithm
- Li, Long
(Show Context)
Citation Context ...echnique, it significantly influenced the development of kernel methods in the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron [6], ROMMA =-=[5]-=-, ALMA [3], NORMA [4], Ballseptron [7], and the Passive-Aggressive family of algorithms [1]. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification funct... |

65 |
Online passive aggressive algorithms
- Crammer, Dekel, et al.
- 2006
(Show Context)
Citation Context ...ing setting. Online classification algorithms that can incorporate kernels include the Perceptron [6], ROMMA [5], ALMA [3], NORMA [4], Ballseptron [7], and the Passive-Aggressive family of algorithms =-=[1]-=-. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification function incrementally, by storing a subset of the observed examples in its internal memory. The... |

48 | Online classification on a budget
- Crammer, Kandola, et al.
- 2003
(Show Context)
Citation Context ... an excessively large active set can lead to unacceptably long running times, as the time-complexity of each online round scales linearly with the size of the active set. Crammer, Kandola, and Singer =-=[2]-=- first addressed this problem by describing an online kernel-based modification of the Perceptron algorithm in which the active set does not exceed a predefined budget. Their algorithm removes redunda... |

19 | Online (and offline) on an even tighter budget
- Weston, Bordes, et al.
- 2005
(Show Context)
Citation Context ...the active set does not exceed a predefined budget. Their algorithm removes redundant examples from the active set so as to make the best use of the limited memory resource. Weston, Bordes and Bottou =-=[9]-=- followed with their own online kernel machine on a budget. Both techniques work relatively well in practice, however they both lack a theoretical guarantee on their prediction accuracy. In this paper... |

15 | A new perspective on an old perceptron algorithm
- Shalev-Shwartz, Singer
- 2005
(Show Context)
Citation Context ...the development of kernel methods in the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron [6], ROMMA [5], ALMA [3], NORMA [4], Ballseptron =-=[7]-=-, and the Passive-Aggressive family of algorithms [1]. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification function incrementally, by storing a subset... |

4 |
Online (and offline) on an even tighter budget. AISTATS
- Weston, Bordes, et al.
- 2005
(Show Context)
Citation Context ...the active set does not exceed a predefined budget. Their algorithm removes redundant examples from the active set so as to make the best use of the limited memory resource. Weston, Bordes and Bottou =-=[9]-=- followed with their own online kernel machine on a budget. Both techniques work relatively well in practice, however they both lack a theoretical guarantee on their prediction accuracy. In this paper... |

3 |
Online classification on a budget. NIPS
- Crammer, Kandola, et al.
- 2003
(Show Context)
Citation Context ... an excessively large active set can lead to unacceptably long running times, as the time-complexity of each online round scales linearly with the size of the active set. Crammer, Kandola, and Singer =-=[2]-=- first addressed this problem by describing an online kernel-based modification of the Perceptron algorithm in which the active set does not exceed a predefined budget. Their algorithm removes redunda... |

2 |
Online passive aggressive algorithms. Leibniz TR 2005-29. Available from http://leibniz.cs.huji.ac.il/tr/774.pdf
- Crammer, Dekel, et al.
(Show Context)
Citation Context ...n the online-learning setting. Online classification algorithms that can incorporate kernels include the Perceptron [7], ROMMA [6], ALMA [4], NORMA [5] and the Passive-Aggressive family of algorithms =-=[1]-=-. Each of these algorithms observes examples in a sequence of rounds, and constructs its classification function incrementally by storing a subset of the observed examples in its internal memory. The ... |