Abstract

A statistically based program has been written which learns to distinguish between languages. The amount of training text that such a program needs is surprisingly small, and the amount of text needed to make an identification is also quite small. The program incorporates no linguistic presuppositions other than the assumption that text can be encoded as a string of bytes. Such a program can be used to determine which language small bits of text are in. It also shows a potential for what might be called `statistical philology' in that it may be applied directly to phonetic transcriptions to help elucidate family trees among language dialects. A variant of this program has been shown to be useful as a quality control in biochemistry. In this application, genetic sequences are assumed to be expressions in a language peculiar to the organism from which the sequence is taken. Thus language identification becomes species identification.

Citations

202	N-Gram-Based Text Categorization - Cavnar, Trenkle
8	A quality control algorithm for DNA sequencing projects - White, Dunning, et al. - 1993
2	Bigraph and trigraph models for language identification and character recognition - Churcher, Hayes, et al. - 1994
2	Language recognition using two-and three-letter clusters - Hayes - 1993
1	Automatic clustering of languages - Batagelj, Kerzic, et al. - 1992
1	Distinctive character sequences - Churcher - 1994