A Statistical Approach to Extract Chinese Chunk Candidates From Large Corpora (2003)
| Citations: | 1 - 0 self |
BibTeX
@MISC{Zhang03astatistical,
author = {Le Zhang and Xue-qiang Lü and U Xue-qiang and Yan-na Shen and Yao Tian-shun},
title = {A Statistical Approach to Extract Chinese Chunk Candidates From Large Corpora},
year = {2003}
}
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
The extraction of Chunk candidates from real corpora is one of the fundamental tasks of building example-based machine translation model. This paper presents a statistical approach to extract Chinese chunk candidates from large monolingual corpora. The first step is to extract large N-grams (up to 20-gram) from raw corpus. Then two newly proposed Fast Statistical Substring Reduction (FSSR) algorithms can be applied to the initial N-gram set to remove some unnecessary N-grams using their frequency information. The two algorithms are effcient (both have a time complexity of O(n)) and can e#ectively reduce the size of N-gram set up to 50%. Finally, mutual information is used to obtain chunk candidates from reduced N-gram set.
