Abstract

: Two problems arise when semi-static word-based compression methods are applied to large texts, such as those stored in information retrieval systems. First, the space required for the model during decoding can become very large. Second, the need to handle document insertions means that the collection must be periodically recompressed if compression efficiency is to be maintained. Here we show that with careful management the impact of both of these drawbacks can be minimised. Experiments with a word-based model and over 500 Mb of text show that compression rates can be retained even in the face of severe memory limitations on the decoder, and in the face of significant expansion in the size of the text itself. 1 Word-Based Compression The use of a word-based zero-order compression model to represent English text has been considered by several authors [2, 6, 7, 15]. It is particularly appropriate for compressing full-text document collections, an application in which very large quant...

Citations