Citation Context

...orithm is widely adopted for unsupervised training is that it is guaranteed to converge to a good probability model that locally maximizes the likelihood or posterior probability of the training data =-=[6]-=-. For Chinese segmentation, EM is usually applied bysrst extracting a lexicon which contains the candidate multi-grams from a given training corpus, initializing a probability distribution over lexico...

Citation Context

...wo-chunk segmentation, say s 1 =\abcd" and s 2 =\efghijk". Let the probabilities of the original string and the two chunks be p(s), p(s 1 ), and p(s 2 ) respectively. The pointwise mutual in=-=formation [10-=-] between s 1 and s 2 is MI(s 1 ; s 2 ) = log p(s) p(s 1 ) p(s 2 ) : (6) To apply this measure to pruning, we set two thresholdss1 >s2 . If the mutual information is higher than the thresholds1 , we ...

Citation Context

...rter primitives (once the EM optimization has stabilized). Not only does this have the advantage of producing a smaller core lexicon, it also has the side eect of driving EM out of poor local maxima [=-=6, 2]-=- and yielding better segmentation performance. The remainder of the paper describes the self-supervised training procedure in detail, followed by the mutual information lexicon pruning criterion, expe...

Citation Context

... (4) therefore are weighted frequency counts. Thus, the updates can be eciently calculated using the forward and backward algorithm, or eciently approximated using the Viterbi algorithm; see [13] and =-=[5]-=- for detailed algorithms. 2.2 Self-supervised training The main diculty with applying EM to this problem is that the probability distributions are complex and typically cause EM to get trapped in poor...

Citation Context

...63 are most commonly used, building a complete lexicon by hand is impractical. Therefore a number of unsupervised segmentation methods have been proposed recently to segment Chinese and Japanese text =-=[1, 3, 8, 12, 9]-=-. Most of these approaches use some form of EM to learn a probabilistic model of character sequences and then employ Viterbi-decoding-like procedures to segment new text into words. One reason that EM...

Citation Context

...63 are most commonly used, building a complete lexicon by hand is impractical. Therefore a number of unsupervised segmentation methods have been proposed recently to segment Chinese and Japanese text =-=[1, 3, 8, 12, 9]-=-. Most of these approaches use some form of EM to learn a probabilistic model of character sequences and then employ Viterbi-decoding-like procedures to segment new text into words. One reason that EM...

Citation Context

...ease the posterior probability of the training data. One advantage of unsupervised lexicon construction is that it can automatically discover new words once other words have acquired high probability =-=[4]. For example, -=-if one knows the word \computer" then upon seeing \computerscience" it is natural to segment \science" as a new word. Based on this observation, we propose a new word discovery method t...

Citation Context

...63 are most commonly used, building a complete lexicon by hand is impractical. Therefore a number of unsupervised segmentation methods have been proposed recently to segment Chinese and Japanese text =-=[1, 3, 8, 12, 9]-=-. Most of these approaches use some form of EM to learn a probabilistic model of character sequences and then employ Viterbi-decoding-like procedures to segment new text into words. One reason that EM...

Citation Context

...sts the context information is very important. However, because of a dierent test set (our test set is the 1M ChineseTreebank from LDC, whereas their test data is 61K pre-segmented by NMSU segmenter [=-=9]-=- and corrected by hand), the comparison is not fully calibrated. In the perfect lexicon experiments, [12] achieves higher performance (94.7% F-measure), whereas only 91.9% is achieved in our experimen...

Citation Context

... our technique to previous results, we follow [8, 12] and measure performance by precision, recall, and F-measure on detecting word boundaries. Here, a word is considered to be correctly recovered i [=-=11]-=-: 1. a boundary is correctly placed in front of thesrst character of the word, 2. a boundary is correctly placed at the end of the last character of the word, 3. and there is no boundary between thesr...

Citation Context

...tion 2 to recover C 3 0 from C 3 00 . The validation corpus, C 2 , consists of 2000 sentences randomly selected from the test corpus. According to the 1980 Frequency Dictionary of Modern Chinese (see =-=[7]-=-), the top 9000 most frequent words in Chinese consist of 26.7% unigrams, 69.8% 1 http://www.ldc.upenn.edu/ctb/ bigrams, 2.7% trigrams, 0.007% 4-grams, and 0.002% 5-grams. So in our model, we limit th...

Citation Context

...63 are most commonly used, building a complete lexicon by hand is impractical. Therefore a number of unsupervised segmentation methods have been proposed recently to segment Chinese and Japanese text =-=[1, 3, 8, 12, 9]-=-. Most of these approaches use some form of EM to learn a probabilistic model of character sequences and then employ Viterbi-decoding-like procedures to segment new text into words. One reason that EM...

Citation Context

...gmenting an input sentence into words is a nontrivial task in such cases. For Chinese, there has been a signicant amount of research on techniques for discovering word segmentations; see for example [=-=14]-=-. The main idea behind most of these techniques is to start with a lexicon that contains the set of possible Chinese words and then segment a concatenated Chinese character string by optimizing a heur...