## Word-level confidence estimation for machine translation using phrase-based translation models (2005)

### Cached

### Download Links

Venue: | Computational Linguistics |

Citations: | 36 - 4 self |

### BibTeX

@INPROCEEDINGS{Ueffing05word-levelconfidence,

author = {Nicola Ueffing and Hermann Ney},

title = {Word-level confidence estimation for machine translation using phrase-based translation models},

booktitle = {Computational Linguistics},

year = {2005},

pages = {763--770}

}

### Years of Citing Articles

### OpenURL

### Abstract

Confidence measures for machine translation is a method for labeling each word in an automatically generated translation as correct or incorrect. In this paper, we will present a new approach to confidence estimation which has the advantage that it does not rely on system output such as Nbest lists or word graphs as many other confidence measures do. It is, thus, applicable to any kind of machine translation system. Experimental evaluation has been performed on translation of technical manuals in three different language pairs. Results will be presented for different machine translation systems to show that the new approach is independent of the underlying machine translation system which generated the translations. To the best of our knowledge, the performance of the new confidence measure is better than that of any existing confidence measure. 1

### Citations

3915 | Pattern Classification and Scene Analysis - Duda, Hart - 1973 |

1197 |
Binary codes capable of correcting deletions, insertions and reversals
- Levenshtein
- 1966
(Show Context)
Citation Context ... is performed in an additional step. 4.3 Approach Based on the Levenshtein Alignment Another way of accounting for variations in the target position of a word is to perform the Levenshtein alignment (=-=Levenshtein 1966-=-)between sentence eI 1 under consideration and the other possible target sentences. The summation in Equation (5)is then performed over all sentences containing e in a position Levenshtein-aligned to ... |

1173 | The mathematics of statistical machine translation: Parameter estimation
- Brown, Pietra, et al.
- 1994
(Show Context)
Citation Context ...m. Therefore, we determine the maximal translation probability of the target word e over the source sentence words: pIBM−1(e|f J 1 ) = max j=0,...,J p(e|fj) , (9) where f0 is the “empty” source word (=-=Brown et al., 1993-=-). The probabilities p(e|fj) are word-based lexicon probabilities. Investigations on the use of the IBM-1 model for word confidence measures showed promising results (Blatz et al., 2003; Blatz et al.,... |

633 | Statistical phrase-based translation - Koehn, Och, et al. - 2003 |

451 | Minimum error rate training in statistical machine translation - Och |

348 |
The alignment template approach to statistical machine translation
- Och, Ney
- 2004
(Show Context)
Citation Context ...lary 223K 351K Dictionary entries 82K DEV (2002 evaluation set)Sentences 878 Running words 25K 105K TEST (2004 evaluation set)Sentences 1,788 Running words 52K 239K � � The alignment template system (=-=Och and Ney 2004-=-)(denoted as AT in the tables), which is also a state-of-the art phrase-based translation system. The Systran version available at http://babelfish.altavista.com/tr in June 2005. These hypotheses were... |

269 | Improved alignment models for statistical machine translation - Och, Tillmann, et al. - 1999 |

118 | An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research
- Nießen, Och, et al.
- 2000
(Show Context)
Citation Context ... second variant considers as correct only those words that are contained in the nearest reference (with this count). The latter corresponds to the procedure used for m-WER and m-PER in MT evaluation (=-=Nießen et al. 2000-=-). Table 5 shows the percentage of words that are labeled as correct according to the different error measures on the development and test corpora of the EPPS task. It can be seen that WER is the stri... |

80 | Confidence Estimation for Machine Translation
- Blatz, Fitzgerald, et al.
- 2004
(Show Context)
Citation Context ...easures have been extensively studied for speech recognition, but are not well known in other areas. Only recently have researchers started to investigate confidence measures for machine translation (=-=Blatz et al., 2004-=-; Gandrabur and Foster, 2003; Quirk, 2004; Ueffing et al., 2003). We apply word confidence measures in MT as follows: For a given translation generated by a machine translation system, we determine a ... |

58 | Improvements in phrase-based statistical machine translation
- Zens, Ney
- 2004
(Show Context)
Citation Context ... language. Nowadays, most state-of-the-art SMT systems are based on bilingual phrases (Och, Tillmann, and Ney 1999; Koehn, Och, and Marcu 2003; Tillmann 2003; Bertoldi et al. 2004; Vogel et al. 2004; =-=Zens and Ney 2004-=-; Chiang 2005). A more detailed description of (1) 10Ueffing and Ney Word-Level Confidence Estimation for MT a phrase-based approach to statistical machine translation will be given in the following ... |

48 | The mathematics of statistical machine translation: Parameter estimation - Mercer - 1993 |

33 | Generation of word graphs in statistical machine translation - Ueffing, Och, et al. - 2002 |

16 | 2003. Confidence estimation for text prediction - Gandrabur, Foster |

14 | Statistical machine translation of European parliamentary speeches - Vilar, Matusov, et al. - 2005 |

10 |
Confidence estimation for machine translation. final report, jhu/clsp summer workshop
- Blatz, Fitzgerald, et al.
- 2003
(Show Context)
Citation Context ...the word confidence scores can be used for generating new hypotheses from the output of different systems (Jayaraman and Lavie, 2005), or the sentence confidence value can be employed for re-ranking (=-=Blatz et al., 2003-=-). In this paper, we will present several approaches to word-level confidence estimation and develop a new phrase-based confidence measure which is independent of the machine translation system whichs... |

6 | sentencelevel confidence measures for machine translation - Word- |

4 | Using a mixture of n-best lists from multiple MT systems in rank-sum-based confidence measure for MT outputs
- Akiba, Sumita, et al.
- 2004
(Show Context)
Citation Context ...d Foster, 2003; Ueffing and Ney, 2005), • combining output from different machine translation systems: hypotheses with low confidence can be discarded before selecting one of the system translations (=-=Akiba et al., 2004-=-), or the word confidence scores can be used for generating new hypotheses from the output of different systems (Jayaraman and Lavie, 2005), or the sentence confidence value can be employed for re-ran... |

4 | The ITC-irst statistical machine translation system for IWSLT-2004
- Bertoldi, Cattoni, et al.
- 2004
(Show Context)
Citation Context ...ll-formedness of the syntax in the target language. Nowadays, most state-of-the-art SMT systems are based on bilingual phrases (Och, Tillmann, and Ney 1999; Koehn, Och, and Marcu 2003; Tillmann 2003; =-=Bertoldi et al. 2004-=-; Vogel et al. 2004; Zens and Ney 2004; Chiang 2005). A more detailed description of (1) 10Ueffing and Ney Word-Level Confidence Estimation for MT a phrase-based approach to statistical machine trans... |

4 |
The ISL statistical translation system for spoken language translation
- Vogel, Hewavitharana, et al.
- 2004
(Show Context)
Citation Context ...yntax in the target language. Nowadays, most state-of-the-art SMT systems are based on bilingual phrases (Och, Tillmann, and Ney 1999; Koehn, Och, and Marcu 2003; Tillmann 2003; Bertoldi et al. 2004; =-=Vogel et al. 2004-=-; Zens and Ney 2004; Chiang 2005). A more detailed description of (1) 10Ueffing and Ney Word-Level Confidence Estimation for MT a phrase-based approach to statistical machine translation will be give... |

3 | decision rules and confidence measures for statistical machine translation - Bayes |

2 | TransType2—Computer assisted translation. RTD project TransType2 (IST–2001–32091) funded 39 Linguistics Volume 33, Number 1 by the European Commission. http://tt2.atosorigin.es - TransType2 |

1 | estimates for confidence intervals in ASR performance evaluation - Bootstrap |

1 |
TC-STAR deliverable no. D5: SLT progress report. Technical report, Integrated project TC-STAR (IST-2002-FP6-506738) funded by the European Commission. http://www.tc-star.org
- Ney, Steinbiss, et al.
- 2005
(Show Context)
Citation Context ...PS data using the direct phrase-based confidence measures. Within the project TC-STAR, an MT evaluation campaign was performed in March 2005 to compare the research systems of the consortium members (=-=Ney et al. 2005-=-). Different conditions concerning the input data were defined. In the following, rescoring results on the verbatim transcriptions will be presented. The translations that RWTH submitted to this evalu... |

1 | Confidence Measures for Machine Translation.Ph.D - Word |

1 |
Application of word-level confidence measures in interactive statistical machine translation
- 2005a
(Show Context)
Citation Context ...hese models and for the sentence probability assigned by the SMT system were optimized with respect to BLEU score on the development corpus. For a detailed description of the system, see Vilar et al. =-=(2005)-=-. This system was ranked first in the evaluation round according to all evaluation criteria (Ney et al. 2005). Two different sets of rescoring experiments were performed. They differ only in their sta... |

1 |
Word-level confidence estimation for machine translation using phrase-based translation models
- 2005b
(Show Context)
Citation Context ...hese models and for the sentence probability assigned by the SMT system were optimized with respect to BLEU score on the development corpus. For a detailed description of the system, see Vilar et al. =-=(2005)-=-. This system was ranked first in the evaluation round according to all evaluation criteria (Ney et al. 2005). Two different sets of rescoring experiments were performed. They differ only in their sta... |