Random texts exhibit Zipf's-law-like word frequency distribution (1992)
Cached
Download Links
- [www.nslij-genetics.org]
- [www.santafe.edu]
- [www.newasiacsr.com]
- DBLP
Other Repositories/Bibliography
| Venue: | IEEE Transactions on Information Theory |
| Citations: | 131 - 3 self |
BibTeX
@ARTICLE{Li92randomtexts,
author = {Wentian Li},
title = {Random texts exhibit Zipf's-law-like word frequency distribution},
journal = {IEEE Transactions on Information Theory},
year = {1992},
pages = {1842--1845}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
are scanned from a copy of the paper (apologize for the poor quality). It is shown that the distribution of word frequencies for randomly generated texts is very similar to Zipf’s law observed in natural languages such as the English. The facts that the frequency of occurrence of a word is almost an inverse power law function of its rank and the exponent of this inverse power law is very close to 1 are largely due to the transformation from the word’s length to its rank, which stretches an exponential function to a power law function. key words: statistical linguistics, Zipf’s law, power-law distribution, random texts. Zipf observed long time ago [1] that the distribution of word frequencies in English, if the words are aligned according to their ranks, is an inverse power law with the exponent very close to 1. In other words, if the most frequently occurring word appears in the text with the frequency P(1), the next most frequently occurring word has the frequency P(2), and the rank-r word has the frequency P(r), the frequency distribution is P(r) = C rα, (1)
Keyphrases
s-law-like word frequency distribution random text exhibit zipf word frequency inverse power law zipf law statistical linguistics random text poor quality frequency distribution long time inverse power law function rank-r word key word power law function exponential function power-law distribution natural language word length
