Abstract

Much work has been done on the statistical analysis of text. In some cases reported in the literature, inappropriate statistical methods have been used, and statistical significance of results have not been addressed. In particular, asymptotic normality assumptions have often been used unjustifiably, leading to flawed results.This assumption of normal distribution limits the ability to analyze rare events. Unfortunately rare events do make up a large fraction of real text.However, more applicable methods based on likelihood ratio tests are available that yield good results with relatively small samples. These tests can be implemented efficiently, and have been used for the detection of composite terms and for the determination of domain-specific terms. In some cases, these measures perform much better than the methods previously used. In cases where traditional contingency table methods work well, the likelihood ratio tests described here are nearly identical.This paper describes the basis of a measure based on likelihood ratios that can be applied to the analysis of text.

Citations

502	A statistical approach to machine translation - Brown, Cocke, et al. - 1990
168	in Introduction to the Theory of Statistics - MOOD - 1950
141	KW: Identifying word correspondences in parallel texts - WA, Church - 1991
84	Using latent semantic analysis to improve access to textual information - Dumais, Furnas, et al. - 1988
32	Parsing, word associations, and typical predicate-argument relations.” Speech and Natural Language - Church, Gale, et al. - 1989
31	Distribution-free Statistical Tests - BRADLEY - 1968
1	kji Pji - ~j kji = ~kjlogpj J - Gale, Church - 1993