Searchable Words on the Web

Searchable Words on the Web (2001)

Cached

Download Links

[www.cs.rmit.edu.au]

Other Repositories/Bibliography

DBLP

by Hugh E. Williams , Justin Zobel

Venue:	International Journal of Digital Libraries
Citations:	14 - 6 self

Versions

Version History

Metadata Version 3

User correction supplied by SystemCorrections

Datum	Value	Source
TITLE	Searchable Words on the Web	user correction - Legacy Corrections
AUTHOR NAME	Hugh E. Williams	SVM HeaderParse 0.1
AUTHOR AFFIL	Department of Computer Science, RMIT University,; GPO Box 2476V, Melbourne 3001	SVM HeaderParse 0.2
AUTHOR ADDR	Australia	SVM HeaderParse 0.1
AUTHOR NAME	Justin Zobel	SVM HeaderParse 0.1
AUTHOR AFFIL	Department of Computer Science, RMIT University,; GPO Box 2476V, Melbourne 3001	SVM HeaderParse 0.2
AUTHOR ADDR	Australia	SVM HeaderParse 0.1
ABSTRACT	In designing data structures for text databases, it is valuable to know how many different words are likely to be encountered in a particular collection. For example, vocabulary accumulation is central to index construction for text database systems; it is useful to be able to estimate the space requirements and performance characteristics of the main-memory data structures used for this task. However, it is not clear how many distinct words will be found in a text collection or whether new words will continue to appear after inspecting large volumes of data. We propose practical definitions of a word, and investigate new word occurrences under these models in a large text collection. We inspected around two billion word occurrences in 45 gigabytes of world-wide web documents, and found just over 9.74 million different words in 5.5 million documents; overall, 1 word in 200 was new. We observe that new words continue to occur, even in very large data sets, and that choosing stricter definitions of what constitutes a word has only limited impact on the number of new words found.	user correction - Legacy Corrections
YEAR	2001	INFERENCE
VENUE	International Journal of Digital Libraries	INFERENCE
VENUE TYPE	JOURNAL	INFERENCE
PAGES	2005	INFERENCE
VOLUME	5	INFERENCE
CITATIONS	17 found	ParsCit 1.0