Language Models as Representations for Weakly-Supervised NLP Tasks

Language Models as Representations for Weakly-Supervised NLP Tasks

Cached

Download Links

by Fei Huang , Er Yates , Arun Ahuja , Doug Downey

BibTeX

@MISC{Huang_languagemodels,
    author = {Fei Huang and Er Yates and Arun Ahuja and Doug Downey},
    title = {Language Models as Representations for Weakly-Supervised NLP Tasks},
    year = {}
}

Bookmark

OpenURL

Abstract

Finding the right representation for words is critical for building accurate NLP systems when domain-specific labeled data for the task is scarce. This paper investigates language model representations, in which language models trained on unlabeled corpora are used to generate real-valued feature vectors for words. We investigate ngram models and probabilistic graphical models, including a novel lattice-structured Markov Random Field. Experiments indicate that language model representations outperform traditional representations, and that graphical model representations outperform ngram models, especially on sparse and polysemous words. 1

Citations

6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
3116	A tutorial on hidden Markov models and selected applications in speech recognition - Rabiner - 1989
2699	MJ: Introduction to Modern Information Retrieval - Salton, McGill - 1986
2168	Indexing by latent semantic analysis - Deerwester, Dumais, et al. - 1990
1654	B.: Building a large annotated corpus of english: The Penn treebank - Marcus, Marcinkiewicz, et al. - 1993
540	Class-based n-gram models of natural language - Brown, Pietra, et al. - 1992
478	L: Distributional Clustering of English Words - Pereira, Tishby, et al. - 1993
375	Factorial hidden markov models - Ghahramani, Jordan - 1997
182	Distributional structure - Harris - 1954
93	Dimensionality reduction by random mapping: Fast similarity computation for clustering - Kaski - 1998
86	Frustratingly easy domain adaptation - Daume - 2007
85	Domain adaptation for statistical classifiers - Daume, Marcu
54	Simple semi-supervised dependency parsing - Koo, Carreras, et al. - 2008
52	2008. A unified architecture for natural language processing: deep neural networks with multitask learning - Collobert, Weston
46	Algorithms for bigram and trigram word clustering - Martin, Liermann, et al. - 1995
42	on the limited memory method for large scale optimization - Liu, Nocedal - 1989
41	Learning bounds for domain adaptation - Blitzer, Crammer, et al.
41	Dynamic programming on graphs with bounded treewidth - Bodlaender - 1988
34	2010) From Frequency to Meaning: Vector Space Models of Semantics - Turney, Pantel
31	Guided learning for bidirectional sequence classification - Shen, Satta, et al. - 2007
23	Curriculum learning - Bengio, Louradour, et al. - 2009
20	Hierarchical Bayesian Domain Adaptation - Finkel, Manning - 2009
18	Domain adaptation with multiple sources - Mansour, Mohri, et al. - 2009
17	A theory of learning from different domains - Ben-David, Blitzer, et al. - 2010
17	Distributional representations for handling sparsity in supervised sequence labeling - Huang, Yates - 2009
16	A scalable hierarchical distributed language model - Mnih, Hinton - 2008
14	Phrase clustering for discriminative learning - Lin, Wu - 2009
13	Web-scale distributional similarity and entity set expansion - Pantel, Crestan, et al. - 2009
11	dirichlet allocation. The - Latent
10	Improving generative statistical parsing with semi-supervised word clustering - Candito, Crabbé - 2009
8	representations: A simple and general method for semi-supervised learning - Word
6	Neural net language models - Bengio - 2001
6	Online methods for multi-domain learning and adaptation - Dredze, Crammer - 2008
5	Self-organizing maps of words for natural language processing applications - Honkela - 1997
4	Noun classification from predicageargument structures - Hindle - 1990
4	Open-domain semantic role labeling by modeling word spans - Huang, Yates - 2010
3	Locating complex named entities in web text - 2007a
3	Sparse information extraction: Unsupervised language models to the rescue - 2007b
2	Exploring representation-learning approaches to domain adaptation - Huang, Yates - 2010
1	Multi-domain learning by confidence weighted parameter combination - Dredze, Kulesza, et al. - 2010
1	Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data - Res - 2007
1	Towards explicit semantic features using independent component analysis - Väyrynen, Honkela, et al. - 2007
1	learning via semi-supervised embedding - Deep
1	Open-domain semantic role labeling by modeling word spans - 2010b - 2007