DMCA
LETTER Communicated by Liam Paninski Maximum Likelihood Set for Estimating a Probability Mass Function
BibTeX
@MISC{Jedynak_lettercommunicated,
author = {Bruno M. Jedynak and Département De Mathématiques and Sanjeev Khudanpur},
title = {LETTER Communicated by Liam Paninski Maximum Likelihood Set for Estimating a Probability Mass Function},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a new method for estimating the probability mass function (pmf) of a discrete and finite random variable from a small sample. We focus on the observed counts—the number of times each value appears in the sample—and define the maximum likelihood set (MLS) as the set of pmfs that put more mass on the observed counts than on any other set of counts possible for the same sample size. We characterize the MLS in detail in this article. We show that the MLS is a diamond-shaped subset of the probability simplex [0, 1] k bounded by at most k × (k − 1) hyperplanes, where k is the number of possible values of the random variable. The MLS always contains the empirical distribution, as well as a family of Bayesian estimators based on a Dirichlet prior, particularly the wellknown Laplace estimator. We propose to select from the MLS the pmf that is closest to a fixed pmf that encodes prior knowledge. When using Kullback-Leibler distance for this selection, the optimization problem comprises finding the minimum of a convex function over a domain defined by linear inequalities, for which standard numerical procedures are available. We apply this estimate to language modeling using Zipf’s law to encode prior knowledge and show that this method permits obtaining state-of-the-art results while being conceptually simpler than most competing methods. 1
Keyphrases
probability mass function liam paninski maximum likelihood set finite random variable empirical distribution linear inequality sample size new method possible value maximum likelihood zipf law standard numerical procedure probability simplex bayesian estimator small sample diamond-shaped subset optimization problem prior knowledge random variable wellknown laplace estimator kullback-leibler distance convex function state-of-the-art result observed count
