Abstract

This paper argues that lossless compression, operating within the standard training/testing paradigm of machine learning, is a key technology for text mining. Researchincompression has always taken the pragmatic view that les need to be processed whatever they maycontain, rather than the normative approach of classical language analysis whichgenerally assumes idealized input: a sequence of sentences, comprising words all of which appear in the dictionary, delimited by single spaces, with punctuation and perhaps numbers but no other extraneous symbols. In practice text|particularly text gathered from the Web, the principal source of material used today|is messy, and many useful clues come from the messiness

Citations