BibTeX
@MISC{_xiaofeilu,
author = {},
title = {Xiaofei Lu A Hybrid Model for Chinese Word Segmentation},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This paper describes a hybrid model that combines machine learning with linguistic and statistical heuristics for integrating unknown word identification with Chinese word segmentation. The model consists of two major components: a tagging component that annotates each character in a Chinese sentence with a position-of-character (POC) tag that indicates its position in a word, and a merging component that transforms a POC-tagged character sequence into a word-segmented sentence. The tagging component uses a support vector machine (Vapnik, 1995) based tagger to produce an initial tagging of the text and a transformation-based tagger (Brill, 1995) to improve the initial tagging. In addition to the POC tags assigned to the characters, the merging component incorporates a number of linguistic and statistical heuristics to detect words with regular internal structures, recognize long words, and filter non-words. Experiments show that, without resorting to a separate unknown word identification mechanism, the model achieves an F-score of 95.0 % for word segmentation and a competitive recall of 74.8 % for unknown word identification. 1
Keyphrases
chinese word segmentation xiaofei lu hybrid model initial tagging unknown word identification statistical heuristic hybrid model support vector machine major component merging component poc-tagged character sequence regular internal structure poc tag word-segmented sentence separate unknown word identification mechanism long word chinese sentence competitive recall transformation-based tagger word segmentation
