A Comparative Study of Bing Web N-gram Language Models for Web Search and Natural Language Processing

A Comparative Study of Bing Web N-gram Language Models for Web Search and Natural Language Processing

Cached

Download Links

by Jianfeng Gao , Patrick Nguyen , Xiaolong Li , Chris Thrasher , Mu Li , Kuansan Wang

Versions

Version 0

Version History

Metadata Version 0

Datum	Value	Source
TITLE	A Comparative Study of Bing Web N-gram Language Models for Web Search and Natural Language Processing	SVM HeaderParse 0.2
AUTHOR NAME	Jianfeng Gao	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
AUTHOR NAME	Patrick Nguyen	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
AUTHOR NAME	Xiaolong Li	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
AUTHOR NAME	Chris Thrasher	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
AUTHOR NAME	Mu Li	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
AUTHOR NAME	Kuansan Wang	SVM HeaderParse 0.2
AUTHOR AFFIL	Microsoft Research; One Microsoft Way	SVM HeaderParse 0.2
AUTHOR ADDR	Redmond, WA 98052 USA	SVM HeaderParse 0.2
ABSTRACT	This paper presents a comparative study of the recently released Microsoft Web N-gram Language Models (MWNLM) 1 on three web search and natural language processing tasks: search query spelling correction, query reformulation, and statistical machine translation. MWNLM, as well as the corresponding web services, called Microsoft Web N-gram Services, are much more accessible and easier to use than the previously released text corpora used for large language model training, including the LDC English Gigaword corpus and the Google Web 1T N-gram corpus, because the Microsoft Web N-gram Services provide the access to the smoothed n-gram probabilities based on a set of language models trained from the different text fields from the web documents as well as search queries. Our results show that MWNLM outperform the n-gram models trained on the Gigaword corpus and the Google Web 1T N-gram corpus on all the three tasks. In particular, the significant improvements on search query spelling correction and search query reformulation, resulting from MWNLM, demonstrate the benefit of training multiple language models on different portions of web data and search queries in a principled way with zero count cutoff.	SVM HeaderParse 0.2
CITATIONS	16 found	ParsCit 1.0