Better hypothesis testing for statistical machine translation: Controlling for optimizer instability (2011)
Cached
Download Links
| Venue: | In Proc. of ACL |
| Citations: | 8 - 3 self |
BibTeX
@INPROCEEDINGS{Clark11betterhypothesis,
author = {Jonathan H. Clark and Chris Dyer and Alon Lavie and Noah A. Smith},
title = {Better hypothesis testing for statistical machine translation: Controlling for optimizer instability},
booktitle = {In Proc. of ACL},
year = {2011}
}
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
In statistical machine translation, a researcher seeks to determine whether some innovation (e.g., a new feature, model, or inference algorithm) improves translation quality in comparison to a baseline system. To answer this question, he runs an experiment to evaluate the behavior of the two systems on held-out data. In this paper, we consider how to make such experiments more statistically reliable. We provide a systematic analysis of the effects of optimizer instability—an extraneous variable that is seldom controlled for—on experimental outcomes, and make recommendations for reporting results more accurately. 1
