Mostly-Unsupervised Statistical Segmentation of Japanese: Applications to Kanji

Mostly-Unsupervised Statistical Segmentation of Japanese: Applications to Kanji (2000)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Rie Kubota Ando , Lillian Lee

Citations:

32 - 1 self

BibTeX

@MISC{Ando00mostly-unsupervisedstatistical,
    author = {Rie Kubota Ando and Lillian Lee},
    title = {Mostly-Unsupervised Statistical Segmentation of Japanese: Applications to Kanji},
    year = {2000}
}

Years of Citing Articles

Bookmark

OpenURL

Abstract

Given the lack of word delimiters in written Japanese, word segmentation is generally considered a crucial first step in processing Japanese texts. Typical Japanese segmentation algorithms rely either on a lexicon and grammar or on pre-segmented data. In contrast, we introduce a novel statistical method utilizing unsegmented training data, with performance on kanji sequences comparable to and sometimes surpassing that of morphological analyzers over a variety of error metrics.

Citations

627	Word association norms, mutual information, and lexicography - Church, Hanks - 1990
560	G: Suffix arrays: a new method for on-line string searches - Manber, Myers - 1993
246	Statistical Learning by 8-Month-Old Infants - Saffran, Aslin, et al. - 1996
201	Unsupervised Learning of the Morphology of a Natural Language - Goldsmith - 2001
191	Elements of Information Theory. Wiley series in telecommunications. Wiley-Interscience - Cover, Thomas - 1991
89	Japanese morphological analysis system ChaSen manual - Matsumoto, Kitauchi, et al. - 1997
48	A Compression-based Algorithm for Chinese Word Segmentation - Teahan, McNab, et al. - 2000
46	Japanese morphological analysis system JUMAN manual, version 2.0 (in Japanese - Matsumoto, Kurohashi, et al. - 1994
46	A trainable rule-based algorithm for word segmentation - Palmer - 1997
43	Chinese Text Segmentation for Text Retrieval: Achievements and Problems - Wu, Tseng - 1993
41	A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of Japanese - Nagao, Mori - 1994
41	Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus - Yamamoto, Church - 2001
37	A stochastic japanese morphological analyzer using a forward-DP backward-A* n-best search algorithm - Nagata - 1994
26	Discovering Chinese words from unsegmented text - Ge, Pratt, et al. - 1999
24	A statistical method for finding word boundaries - Sproat, Shih - 1990
22	Selfsupervised Chinese word segmentation - Peng, Schuurmans - 2001
21	Chinese word segmentation without using lexicon and hand-crafted training data - Sun, Shen, et al. - 1998
20	Evaluating parsing strategies using standardized parse files - Grishman, Macleod, et al. - 1992
19	The use of predictive dependencies in language learning - Saffran - 2001
13	A new statistical formula for Chinese text segmentation incorporating contextual information. 82--89 - Dai, Loh, et al.
12	Context-based spelling correction for Japanese OCR - Nagata - 1996
11	An application of Information Theory in Chinese Word Segmentation, Computer Processing of Chinese - Lua, Gan - 1994
9	Japanese morphological analyzer using word co-occurrence—JTAG - Fuchi, Takagi - 1998
9	Extracting key terms from chinese and japanese texts - Fung - 1998
8	Corpus-based automatic compound extraction with mutual information and relative frequency count - Wu, Su - 1993
7	Improvements of japanese morphological analyzer JUMAN - Matsumoto, Nagao - 1994
7	Japanese word segmentation by hidden markov model - Papageorgiou - 1994
6	HMM Parameter Learning for Japanese Morphological Analyzer”, IPSJ - Takeuchi, Matsumoto - 1997
6	Example-based correction of word segmentation and part of speech labelling - Matsukawa, Miller, et al. - 1993
5	Unsupervised statistical segmentation of japanese kanji strings - Ando, Lee - 1999
4	Language Modeling by String Pattern N-gram for Japanese Speech Recogniton - Ito, Kohda - 1996
4	Automatic decomposition of kanji compound words using stochastic estimation - Takeda, Fujisaki - 1987
4	A probabilistic algorithm for segmenting non-kanji Japanese strings - Teller, Batchelder - 1994
3	Use of mutual information based character clusters in dictionary-less morphological analysis of Japanese - Kashioka, Kawata, et al. - 1998
3	Unknown word extraction from corpora using n-gram statistics - Mori, Nagao - 1998
3	Automatic extraction of new words from Japanese texts using generalized forwardbackward search - Nagata - 1996
3	A self-organizing Japanese word segmenter using heuristic word identification and re-estimation - Nagata - 1997
3	What makes a word: learning base units in Japanese for speech recognition - Tomokiyo, Ries - 1997
3	A re-estimation method for stochastic language modeling from ambiguous observations - Yamamoto - 1996
3	Overlapping statistical segmentation for effective indexing of Japanese text - Ogawa, Matsuda - 1999
2	A position statement on Chinese segmentation. http://www.cs.ust.hk/#dekai/- papers/segmentation.html. Presented at the Chinese Language Processing Workshop - Wu - 1998
2	Experiments on the use of bigram mutual information for Chinese natural language processing - Lua - 1995
2	Using sux arrays to compute term frequency and document frequency for all substrings in a corpus - Yanamoto, Church - 2001
1	A stochastic finitesate word-segmentation algorithm for Chinese - Sproat, Shih, et al. - 1996
1	Chinese word segmentation without using lexicon and hand-crafted training data - Tsou - 1998
1	Xiaopeng Tao. 2001. Chinese text segmentation with MBDP-1: Making the most of training corpora - Brent
1	Parsing a natural language using information statistics - Magerman, Marcus - 1990
1	Mostly-Unsupervised Statistical Segmentation of Japanese 23 Matsumoto, Yuji and Makoto - Nagao - 1994
1	Overlapping statistical segmentation for e#ective indexing of Japanese text - Ogawa, Matsuda - 1999
1	The use of predictive dependencies in language learning - Saran