Cached Sufficient Statistics for Efficient Machine Learning with Large Datasets

Cached Sufficient Statistics for Efficient Machine Learning with Large Datasets (1997)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Andrew Moore , Mary Soon Lee

Venue:	Journal of Artificial Intelligence Research
Citations:	146 - 20 self

BibTeX

@ARTICLE{Moore97cachedsufficient,
    author = {Andrew Moore and Mary Soon Lee},
    title = {Cached Sufficient Statistics for Efficient Machine Learning with Large Datasets},
    journal = {Journal of Artificial Intelligence Research},
    year = {1997},
    volume = {8},
    pages = {67--91}
}

OpenURL

Abstract

This paper introduces new algorithms and data structures for quick counting for machine learning datasets. We focus on the counting task of constructing contingency tables, but our approach is also applicable to counting the number of records in a dataset that match conjunctive queries. Subject to certain assumptions, the costs of these operations can be shown to be independent of the number of records in the dataset and loglinear in the number of non-zero entries in the contingency table. We provide a very sparse data structure, the ADtree, to minimize memory use. We provide analytical worst-case bounds for this structure for several models of data distribution. We empirically demonstrate that tractably-sized data structures can be produced for large real-world datasets by (a) using a sparse tree structure that never allocates memory for counts of zero, (b) never allocating memory for counts that can be deduced from other counts, and (c) not bothering to expand the tree fully near its...

Keyphrases

efficient machine learning large datasets sufficient statistic contingency table data structure non-zero entry conjunctive query memory use analytical worst-case bound several model new algorithm quick counting sparse data structure certain assumption sparse tree structure tractably-sized data structure large real-world datasets data distribution