Abstract

This thesis describes research leading to an improved word hyphenation algo-rithm for the TjrjX82 typesetting system. Hyphenation is viewed primarily as a data compression problem, where we are given a dictionary of words with allowable divi-sion points, and try to devise methods that take advantage of the large amount of redundancy present. The new hyphenation algorithm is based on the idea of hyphenating and in-hibiting patterns. These are simply strings of letters that, when they match hi a word, give us information about hyphenation at some point in the pattern. For example, '-tion ' and *c-c ' are good hyphenating patterns. An important feature of this method is that a suitable set of patterns can be extracted automatical! / from the dictionary. In order to represent the set of patterns in a compart form that is also reasonably efficient for searching, the author has developed a new data structure called a packed trie. This data structure allows the very fast search times characteristic of indexed tries, but in many cases it entirely eliminates the wasted space for null links usually present in such tries. We demonstrate the versatility and practical advantages of this data structure by using a variant of it as the critical component of the program that generates the patterns from the dictionary. The resulting hyphenation algorithm uses about 4500 patterns that compile into a packed trie occupying 25K bytes of storage. These patterns find 89 % of the hyphens in a pocket dictionary word list, with essentially no error. By comparison, the uncompressed dictionary occupies over 500K bytes.

Citations