Abstract

The Problem: All classifiers require a set of features that can be used to distinguish between different examples. In some cases, such as determining whether a chess position is a winning position, the features are clear (the positions of the chess pieces). In other cases, such as text, they are less clear. A document is simply a string of characters. Standard practice dictates that documents should be broken into words, common, uninformative words should be removed and a stemmer is to be applied for removal of common endings (such as “-s ” and “-ing”). However, this processing is insufficient for many languages and many modern-day classification problems. For example, documents about the XWindow system include an unusually large number of words that begin with the letter “x, ” but traditional processing obscures this fact. Unsolicitied commercial e-mail (“spam”) can often be identified by the use of foreign languages, a disproportionate number of exclamation points or capital letters. Again, such information will often be discarded. Motivation: A fundamental evaluation of a system’s understanding of a language is its ability to compress examples of that language. A system with no understanding of the English language (such as most computer compression programs) will compress based on statistical regularities in character occurrences and short words, such as common pairs of letters and short words such as “to ” or “and. ” In

Citations