DMCA
Applying Machine Learning to Text Segmentation for Information Retrieval (2002)
| Citations: | 2 - 0 self |
BibTeX
@MISC{Huang02applyingmachine,
author = {Xiangji Huang and Fuchun Peng and Dale Schuurmans and Nick Cercone and Stephen Robertson},
title = {Applying Machine Learning to Text Segmentation for Information Retrieval},
year = {2002}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a self-supervised word segmentation technique for text segmentation in Chinese information retrieval. This method combines the advantages of traditional dictionary based, character based and mutual information based approaches, while overcoming many of their shortcomings. Experiments on TREC data show this method is promising. Our method is completely language independent and unsupervised, which provides a promising avenue for constructing accurate multi-lingual or cross-lingual information retrieval systems that are exible and adaptive. We nd that although the segmentation accuracy of self-supervised segmentation is not as high as some other segmentation methods, it is enough to give comparable (in some cases even better) retrieval performance. It is commonly believed that word segmentation accuracy is monotonically related to retrieval performance in Chinese information retrieval. However, for Chinese, we nd that the relationship between segmentation and retrieval performance is in fact nonmonotonic; that is, at around 70% word segmentation accuracy an over-segmentation phenomenon begins to occur which leads to a reduction in information retrieval performance. We demonstrate this eect by presenting an empirical investigation of information retrieval on Chinese TREC data, using a wide variety of word segmentation algorithms with word segmentation accuracies ranging from 44% to 95%, including 70% word segmentation accuracy from our self-supervised word-segmentation approach.
Keyphrases
information retrieval text segmentation retrieval performance word segmentation accuracy machine learning chinese information retrieval cross-lingual information retrieval system self-supervised word segmentation technique self-supervised word-segmentation approach traditional dictionary trec data show chinese trec data word segmentation algorithm information retrieval performance segmentation accuracy self-supervised segmentation word segmentation mutual information empirical investigation over-segmentation phenomenon wide variety segmentation method promising avenue
