Results 1 
1 of
1
Study of Some Distance Measures for Language and Encoding Identification
 in Proceedings of ACL 2006 Workshop on Linguistic Distance,Sydney
, 2006
"... To determine how close two language models (e.g., ngrams models) are, we can use several distance measures. If we can represent the models as distributions, then the similarity is basically the similarity of distributions. And a number of measures are based on information theoretic approach. In thi ..."
Abstract

Cited by 6 (2 self)
 Add to MetaCart
(Show Context)
To determine how close two language models (e.g., ngrams models) are, we can use several distance measures. If we can represent the models as distributions, then the similarity is basically the similarity of distributions. And a number of measures are based on information theoretic approach. In this paper we present some experiments on using such similarity measures for an old Natural Language Processing (NLP) problem. One of the measures considered is perhaps a novel one, which we have called mutual cross entropy. Other measures are either well known or based on well known measures, but the results obtained with them visavis oneanother might help in gaining an insight into how similarity measures work in practice. The first step in processing a text is to identify the language and encoding of its contents. This is a practical problem since for many languages, there are no universally followed text encoding standards. The method we have used in this paper for language and encoding identification uses pruned character ngrams, alone as well augmented with word ngrams. This method seems to give results comparable to other methods. 1