BibTeX
@MISC{Williams05regularpapers,
author = {Hugh E. Williams and Justin Zobel},
title = {Regular papers Searchable words on the Web},
year = {2005}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract. In designing data structures for text databases, it is valuable to know how many different words are likely to be encountered in a particular collection. For example, vocabulary accumulation is central to index construction for text database systems; it is useful to be able to estimate the space requirements and performance characteristics of the main-memory data structures used for this task. However, it is not clear how many distinct words will be found in a text collection or whether new words will continue to appear after inspecting large volumes of data. We propose practical definitions of a word and investigate new word occurrences under these models in a large text collection. We inspected around two billion word occurrences in 45 GB of World Wide Web documents and found just over 9.74 million different words in 5.5 million documents; overall, 1 word in 200 was new. We observe that new words continue to occur, even in very large datasets, and that choosing stricter definitions of what constitutes a word has only limited impact on the number of new words found.
Keyphrases
regular paper searchable word new word word occurrence many different word text database system many distinct word large text collection world wide web document text collection large datasets performance characteristic main-memory data structure data structure different word new word occurrence text database practical definition particular collection large volume vocabulary accumulation index construction space requirement limited impact
