Abstract

If two translation systems differ differ in performance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling methods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.

Citations

1783	An Introduction to the Bootstrap - Tibshirani, R - 1993
992	BLEU: A Method for Automatic Evaluation of Machine Translation - Papineni, Roukos, et al. - 2002
282	Lattice-based minimum error rate training for statistical machine translation - Macherey, Och, et al. - 2008
90	Pharaoh: a beam search decoder for phrasebased statistical machine translation models - Koehn - 2004
78	Minimum Bayes-risk decoding for statistical machine translation - Kumar, Byrne - 2004
77	Europarl: A multilingual corpus for evaluation of machine translation - Koehn - 2002
63	R.L.: The mathematics of statistical machine translation: Parameter estimation - Brown, Pietra, et al. - 1993
44	Statistical machine translation: from single-word models to alignment templates - Och
35	Precision and recall of machine translation - Melamed, Green, et al. - 2003
30	Statistical Phrase based Translation - Koehn, Och, et al. - 2003
28	A projection extension algorithm for statistical machine translation - Tillmann - 2003
20	Greedy Decoding for Statistical Machine Translation in Almost Linear Time - Germann
20	The CMU Statistical Machine Translation System - Vogel, Zhang, et al. - 2003