## Improve latent semantic analysis based language model by integrating multiple level knowledge (2002)

Venue: | In Proc. of ICSLP |

Citations: | 4 - 0 self |

### BibTeX

@INPROCEEDINGS{Zhang02improvelatent,

author = {Rong Zhang and Er I. Rudnicky},

title = {Improve latent semantic analysis based language model by integrating multiple level knowledge},

booktitle = {In Proc. of ICSLP},

year = {2002},

pages = {893--896}

}

### OpenURL

### Abstract

We describe an extension to the use of Latent Semantic Analysis (LSA) for language modeling. This technique makes it easier to exploit long distance relationships in natural language for which the traditional n-gram is unsuited. However, with the growth of length, the semantic representation of the history may be contaminated by irrelevant information, increasing the uncertainty in predicting the next word. To address this problem, we propose a multilevel framework dividing the history into three levels corresponding to document, paragraph and sentence. To combine the three levels of information with the n-gram, a Softmax network is used. We further present a statistical scheme that dynamically determines the unit scope in the generalization stage. The combination of all the techniques leads to a 14 % perplexity reduction on a subset of Wall Street Journal, compared with the trigram model. 1.

### Citations

163 | Two decades of statistical language modeling: Where do we go from here
- Rosenfeld
- 2000
(Show Context)
Citation Context ... models lose a great deal of useful language information, such as long distance word correlations, syntactic constraints and semantic consistence. Many attempts have been made in the last two decades =-=[1]-=- to solve these problems. Some successful examples include the class model, lattice model, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the mo... |

160 | A neural probabilistic language model
- Bengio, Ducharme, et al.
- 2003
(Show Context)
Citation Context ...del, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the model based on Latent Semantic Analysis (LSA). The LSA based language model [2][3][4][5]=-=[6]-=- aims to use the semantic relationship between words to increase the accuracy of prediction. Different from the trigger modeling which has the same motivation, the LSA approach maps the words and hist... |

142 |
Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval
- Ricardo
- 2011
(Show Context)
Citation Context ...al compared with a trigram model. 2. LANGUAGE MODELING BASED ON LATENT SEMANTIC ANALYSIS 2.1 Latent Semantic Analysis LSA is a widely used statistical technique in the information retrieval community =-=[7]-=-. The primary assumption is that there exists some underlying or latent structure in the occurrence pattern of words across documents, and LSA can be used to estimate this latent structure. This is ac... |

78 | Exploiting latent semantic information in statistical language modeling
- Bellegarda
- 2000
(Show Context)
Citation Context ...ice model, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the model based on Latent Semantic Analysis (LSA). The LSA based language model [2][3]=-=[4]-=-[5][6] aims to use the semantic relationship between words to increase the accuracy of prediction. Different from the trigger modeling which has the same motivation, the LSA approach maps the words an... |

49 | Towards better integration of semantic predictors in statistical language modeling
- Coccaro, Jurafsky
- 1998
(Show Context)
Citation Context ... model, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the model based on Latent Semantic Analysis (LSA). The LSA based language model [2][3][4]=-=[5]-=-[6] aims to use the semantic relationship between words to increase the accuracy of prediction. Different from the trigger modeling which has the same motivation, the LSA approach maps the words and h... |

32 | A multi-span language modeling framework for large vocabulary speech recognition
- Bellegarda
- 1998
(Show Context)
Citation Context ..., lattice model, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the model based on Latent Semantic Analysis (LSA). The LSA based language model =-=[2]-=-[3][4][5][6] aims to use the semantic relationship between words to increase the accuracy of prediction. Different from the trigger modeling which has the same motivation, the LSA approach maps the wo... |

31 |
Large Vocabulary Speech Recognition with Multispan Statistical Language Models
- Bellegarda
- 2000
(Show Context)
Citation Context ...attice model, caching model, decision tree model, maximum entropy model, whole sentence model, structured model, and the model based on Latent Semantic Analysis (LSA). The LSA based language model [2]=-=[3]-=-[4][5][6] aims to use the semantic relationship between words to increase the accuracy of prediction. Different from the trigger modeling which has the same motivation, the LSA approach maps the words... |

5 |
Can artificial neural networks learn language models
- Xu, Rudnicky
- 2000
(Show Context)
Citation Context ...0 10 20 30 40 50 60 70 n (sentence) Figure 3 F(n) and approximation for sentence 5. SOFTMAX COMBINATION The Softmax network is a powerful tool to learn probabilistic events, e.g. language modeling [6]=-=[8]-=-. In a Softmax network, the activation function for the output node i is gi exp O i = (16) g j exp where i ∑ j g is the net value for node i. Apparently, the value of O is within (0, 1), and the sum o... |