Corroborating Text Evaluation Results with Heterogeneous Measures

Corroborating Text Evaluation Results with Heterogeneous Measures

Cached

Download Links

by Enrique Amigó , Julio Gonzalo , Jesús Giménez , Felisa Verdejo

BibTeX

@MISC{Amigó_corroboratingtext,
    author = {Enrique Amigó and Julio Gonzalo and Jesús Giménez and Felisa Verdejo},
    title = {Corroborating Text Evaluation Results with Heterogeneous Measures},
    year = {}
}

Bookmark

OpenURL

Abstract

Automatically produced texts (e.g. translations or summaries) are usually evaluated with n-gram based measures such as BLEU or ROUGE, while the wide set of more sophisticated measures that have been proposed in the last years remains largely ignored for practical purposes. In this paper we first present an indepth analysis of the state of the art in order to clarify this issue. After this, we formalize and verify empirically a set of properties that every text evaluation measure based on similarity to human-produced references satisfies. These properties imply that corroborating system improvements with additional measures always increases the overall reliability of the evaluation process. In addition, the greater the heterogeneity of the measures (which is measurable) the higher their combined reliability. These results support the use of heterogeneous measures in order to consolidate text evaluation results. 1

Citations

992	BLEU: A Method for Automatic Evaluation of Machine Translation - Papineni, Roukos, et al. - 2002
204	A Study of Translation Edit Rate with Targeted Human Annotation - Snover, Dorr, et al. - 2006
102	Dependency treelet translation: Syntactically informed phrasal smt - Quirk, Menezes, et al. - 2005
92	A Evaluation Tool for Machine Translation: Fast Evaluation for MT Research - Nießen, Och, et al. - 2000
51	Further meta-evaluation of machine translation - Callison-Burch, Fordyce, et al. - 2008
40	Evaluation of Machine Translation and its Evaluation - Turian, Shen, et al. - 2003
35	A learning approach to improving sentence-level mt evaluation - Kulesza, Shieber - 2004
29	A package for automatic evaluation of summaries - Rouge
29	H.: Accelerated dp based search for statistical translation - Tillmann, Vogel, et al. - 1997
24	CDer: Efficient MT Evaluation Using Block Movements - Leusch, Ueffing, et al. - 2006
17	2009. Findings of the 2009 workshop on statistical machine translation - Callison-Burch, Koehn, et al.
17	A Machine Learning Approach to the Automatic Evaluation of Machine Translation - Corston-Oliver, Gamon, et al. - 2001
16	2007b. Regression for sentencelevel MT evaluation with pseudo references - Albrecht, Hwa
16	Robust Machine Translation Evaluation with Entailment Features - Pado, Galley, et al. - 2009
14	of the 2010 joint workshop on statistical machine translation and metrics for machine translation - Findings
13	Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statics - 2004a
13	ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation - 2004b
12	BLANC: Learning Evaluation Metrics for MT - Lita, Rogati, et al. - 2005
11	NIST 2005 machine translation evaluation official results. In Official release of automatic evaluation scores for all submission - Audrey, Przybocki - 2005
9	The Limits of N-gram Translation Evaluation Metrics - Culy, Riehemann - 2003
8	Evaluation: HumanLike vs. Human Acceptable - MT
6	Summarization Evaluation Using Transformed Basic Elements - Tratz, Hovy - 2008
5	dependencies in machine translation evaluation - Labelled
5	The Naming of Things and the Confusion of Tongues: an MT Metric - Reeder, Miller, et al. - 2001
4	Anselmo Pe nas, and Felisa Verdejo. 2005. QARLA: a Framework for the Evaluation of Automatic Summarization - Amigó, Gonzalo
4	Stochastic iterative alignment for machine translation evaluation - Liu, Gildea - 2006
4	error rates: decomposition over POS classes and applications for error analysis - Word
3	A Reexamination of Machine Learning Approaches for Sentence-Level MT Evaluation - 2007a
3	MT evaluation without reference translations: beyond language modeling - Sentence-Level
2	Empirical Machine Translation and its Evaluation - Giménez - 2008
2	Dependency Pair Match: Predicting translation quality with expected syntactic structure. Machine Translation - Expected
2	Dependency-Based Automatic Evaluation for Machine Translation - 2007a
2	Reducing Human Assessments of Machine Translation Quality to Binary Classifiers - Paul, Finch, et al. - 2007
1	approach for automatic evaluation of natural language generation technologies: Application to automatic summarization - Kernel-based