Many automatic evaluation metrics for machine translation (MT) rely on making comparisons to human translations, a resource that may not always be available. In this work, we present a method for developing sentence-level MT evaluation metrics that do not directly rely on human reference translations. Our metrics are developed using regression learning and are based on a set of weaker indicators of fluency and adequacy (pseudo references). Experimental results suggest that they rival standard reference-based metrics in terms of correlations with human judgments on new test instances. 1

Automatic evaluation of machine translation (MT) systems is an important research topic for the advancement of MT technology. Most automatic evaluation methods proposed to date are score-based: they compute scores that represent translation quality, and MT systems are compared on the basis of these scores. We advocate an alternative perspective of automatic MT evaluation based on ranking. Instead of producing scores, we directly produce a ranking over the set of MT systems to be compared. This perspective is often simpler when the evaluation goal is system comparison. We argue that it is easier to elicit human judgments of ranking and develop a machine learning approach to train on rank data. We compare this ranking method to a score-based regression method on WMT07 data. Results indicate that ranking achieves higher correlation to human judgments, especially in cases where ranking-specific features are used. 1

Machine learning methods have been extensively employed in developing MT evaluation metrics and several studies show that it can help to achieve a better correlation with human assessments. Adopting the regression SVM framework, this paper discusses the linguistic motivated feature formulation strategy. We argue that “blind ” combination of available features does not yield a general metrics with high correlation rate with human assessments. Instead, certain simple intuitive features serve better in establishing the regression SVM evaluation model. With six features selected, we show evidences to support our view through a few experiments in this paper. 1

Many automatic evaluation metrics for machine translation (MT) rely on making comparisons to human translations, a resource that may not always be available. We present a method for developing sentence-level MT evaluation metrics that do not directly rely on human reference translations. Our metrics are developed using regression learning and are based on a set of weaker indicators of fluency and adequacy (pseudo references). Experimental results suggest that they rival standard reference-based metrics in terms of correlations with human judgments on new test instances. 1

Combining different metrics into a single measure of quality seems the most direct and natural way to improve over the quality of individual metrics. Recently, several approaches have been suggested (Kulesza and

Abstract not found

This paper describes the automatic MT evaluation

We conduct a set of experiments exploring issues of multiple kernels and kernel learning with respect to a previous work on automatic machine translation evaluation that used SVMs. By applying the techniques of model tampering and anchored learning to the learned SVM(s), we demonstrate both their practicality and utility, especially when used for error analysis and to better understand a given trained SVM. The results are a number of insights about how to improve the SVM for this task, generation of possibilities for future work, and evidence that these two techniques are useful for opening the black box of SVMs. 1

We present three modifications of wellestablished automatic machine translation evaluation measures, to improve correlation between those measures and human evaluation. Following Lin & Och, we present an improved version of the BLEU score, which uses a smoothed geometric mean for combining different n-gram precisions. We use segment boundary markers to increase the weight of words near the segment boundaries in the BLEU score. Our second MT evaluation measure is a variant of the WER which allows for block movements, but does not demand complete and disjoint coverage of the source sentence. As this might be problematic if MT systems are tuned on this score, we later investigate a linear combination of this measure with PER. Finally, we describe an edit distance similar to TER, which also allows for block reordering. Our measure uses a full search, but with the constraint that block operations must be bracketed. We describe this measure using a Bracketing Transduction Grammar, and sketch a polynomial-time algorithm for its calculation. We also modify the WER-like measures such that they use word-dependent substitution costs instead of fixed ones to model the similarity between words. Experimental comparison of these measures show that our new measures correlate significantly better with human judgment than the original measures. 1

Statistical Machine Translation (SMT) is an approach to automatic text translation based on the use of statistical models and examples of translations. Although Machine Translation (MT) systems developed according to other paradigms are still in use, mainly rule-based or example-based MT, SMT dominates academic