DMCA
Active evaluation of classifiers on large datasets (2012)
| Venue: | In ICDM (Runner-up for Best paper award |
| Citations: | 3 - 1 self |
BibTeX
@INPROCEEDINGS{Katariya12activeevaluation,
author = {Namit Katariya and Arun Iyer and Sunita Sarawagi},
title = {Active evaluation of classifiers on large datasets},
booktitle = {In ICDM (Runner-up for Best paper award},
year = {2012}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract—The goal of this work is to estimate the accuracy of a classifier on a large unlabeled dataset based on a small labeled set and a human labeler. We seek to estimate accuracy and select instances for labeling in a loop via a continuously refined stratified sampling strategy. For stratifying data we develop a novel strategy of learning r bit hash functions to preserve similarity in accuracy values. We show that our algorithm provides better accuracy estimates than existing methods for learning distance preserving hash functions. Experiments on a wide spectrum of real datasets show that our estimates achieve between 15 % and 62 % relative reduction in error compared to existing approaches. We show how to perform stratified sampling on unlabeled data that is so large that in an interactive setting even a single sequential scan is impractical. We present an optimal algorithm for performing importance sampling on a static index over the data that achieves close to exact estimates while reading three orders of magnitude less data. Keywords-Accuracy estimation, active evaluation, learning hash functions. I.
Keyphrases
active evaluation large datasets hash function accuracy value wide spectrum unlabeled data optimal algorithm real datasets large unlabeled dataset static index relative reduction interactive setting single sequential scan accuracy estimate novel strategy bit hash function human labeler select instance keywords-accuracy estimation
