## Computing Lattice BLEU Oracle Scores for Machine Translation

Citations: | 1 - 0 self |

### BibTeX

@MISC{Sokolov_computinglattice,

author = {Artem Sokolov and Guillaume Wisniewski and François Yvon},

title = {Computing Lattice BLEU Oracle Scores for Machine Translation},

year = {}

}

### OpenURL

### Abstract

The search space of Phrase-Based Statistical Machine Translation (PBSMT) systems can be represented under the form of a directed acyclic graph (lattice). The quality of this search space can thus be evaluated by computing the best achievable hypothesis in the lattice, the so-called oracle hypothesis. For common SMT metrics, this problem is however NP-hard and can only be solved using heuristics. In this work, we present two new methods for efficiently computing BLEU oracles on lattices: the first one is based on a linear approximation of the corpus BLEU score and is solved using the FST formalism; the second one relies on integer linear programming formulation and is solved directly and using the Lagrangian relaxation framework. These new decoders are positively evaluated and compared with several alternatives from the literature for three language pairs, using lattices produced by two PBSMT systems. 1

### Citations

1624 | BLEU: a Method for Automatic Evaluation of Machine Translation
- Papineni, Roukos, et al.
- 2001
(Show Context)
Citation Context .... 4 Oracle decoding assumes the definition of a measure of the similarity between a reference and a hypothesis. In this paper we will consider sentence-level approximations of the popular BLEU score (=-=Papineni et al., 2002-=-). BLEU is formally defined for two parallel corpora, E = {ej} J j=1 and R = {rj} J j=1 , each containing J sentences as: ( ∏n ) 1/n n-BLEU(E, R) = BP · , (1) m=1 pm where BP = min(1, e 1−c1(R)/c1(E) ... |

403 | A study of translation edit rate with targeted human annotation
- Snover, Dorr, et al.
- 2006
(Show Context)
Citation Context ...for Computational Linguistics, pages 120–129, Avignon, France, April 23 - 27 2012. c○2012 Association for Computational Linguisticsfor other metrics such as METEOR (Banerjee and Lavie, 2005) or TER (=-=Snover et al., 2006-=-). The exact computation of oracles under corpus level metrics, such as BLEU, poses supplementary combinatorial problems that will not be addressed in this work. In this paper, we present two original... |

138 | Minimum Bayes-Risk Decoding for Statistical Machine Translation
- Kumar, Byrne
- 2004
(Show Context)
Citation Context ... approximate oracle hypotheses on lattices. The first one is based on a linear approximation of the corpus BLEU, that was originally designed for efficient Minimum Bayesian Risk decoding on lattices (=-=Tromble et al., 2008-=-). The second one, based on Integer Linear Programming, is an extension to lattices of a recent work on failure analysis for phrase-based decoders (Wisniewski et al., 2010). In this framework, we stud... |

124 | An end-to-end discriminative approach to machine translation
- Liang, Bouchard-Côté, et al.
- 2006
(Show Context)
Citation Context ...le, insufficient expressiveness of reordering models, inadequate scoring function, non-literal references, over-pruned lattices, etc. Oracle decoding has several other applications: for instance, in (=-=Liang et al., 2006-=-; Chiang et al., 2008) it is used as a work-around to the problem of non-reachability of the reference in discriminative training of MT systems. Lattice reranking (Li and Khudanpur, 2009), a promising... |

50 | On dual decomposition and linear programming relaxations for natural language processing
- Rush, Sontag, et al.
(Show Context)
Citation Context ...dard Bellman algorithm. 5.3 Oracle Decoding through Lagrangian Relaxation (RLX) In this section, we introduce another method to solve problem (7) without relying on an external ILP solver. Following (=-=Rush et al., 2010-=-; Chang and Collins, 2011), we propose an original method for oracle decoding based on Lagrangian relaxation. This method relies on the idea of relaxing the clipping constraints: starting from an unco... |

31 | Wojciech Skut, and Mehryar Mohri. 2007. OpenFST: a general and efficient weighted finite-state transducer library - Allauzen, Riley, et al. |

19 | Weighted automata algorithms - Mohri - 2009 |

14 | A systematic analysis of translation model search spaces
- Auli, Lopez, et al.
- 2009
(Show Context)
Citation Context ... better approximate the search space. Exploring the PBSMT search space is one of the few means to perform diagnostic analysis and to better understand the behavior of the system (Turchi et al., 2008; =-=Auli et al., 2009-=-). Useful diagnostics are, for instance, provided by looking at the best (oracle) hypotheses contained in the search space, i.e, those hypotheses that have the highest quality score with respect to on... |

4 | Efficient extraction of oracle-best translations from hypergraphs
- Li, Khudanpur
- 2009
(Show Context)
Citation Context ...for instance, in (Liang et al., 2006; Chiang et al., 2008) it is used as a work-around to the problem of non-reachability of the reference in discriminative training of MT systems. Lattice reranking (=-=Li and Khudanpur, 2009-=-), a promising way to improve MT systems, also relies on oracle decoding to build the training data for a reranking algorithm. For sentence level metrics, finding oracle hypotheses in n-best lists is ... |

3 | Xpress optimizer
- Optimization
- 2007
(Show Context)
Citation Context ..., as long as the reward of an edge can be computed locally. The constrained optimization problem (7) can be solved efficiently using off-the-shelf ILP solvers 7 . 7 In our experiments we used Gurobi (=-=Optimization, 2010-=-) a commercial ILP solver that offers free academic license. 5.2 Shortest Path Oracle (SP) As a trivial special class of the above formulation, we also define a Shortest Path Oracle (SP) that solves t... |

3 | Assessing phrase-based translation models with oracle decoding
- Wisniewski, Allauzen, et al.
- 2010
(Show Context)
Citation Context ... the highest quality score with respect to one or several references. Such oracle hypotheses can be used for failure analysis and to better understand the bottlenecks of existing translation systems (=-=Wisniewski et al., 2010-=-). Indeed, the inability to faithfully reproduce reference translations can have many causes, such as scantiness of the translation table, insufficient expressiveness of reordering models, inadequate ... |

1 | Online large-margin training of syntactic and structural translation features - Dreyer, Khudanpur - 2008 |

1 | Comparing reordering constraints for SMT using efficient BLEU oracle computation - Leusch, Matusov, et al. - 2007 |