## Domain Adaptation of Natural Language Processing Systems (2007)

Citations: | 17 - 1 self |

### BibTeX

@TECHREPORT{Blitzer07domainadaptation,

author = {John Blitzer and Fernando Pereira and Rajeev Alur},

title = {Domain Adaptation of Natural Language Processing Systems},

institution = {},

year = {2007}

}

### OpenURL

### Abstract

My first thanks must go to Fernando Pereira. He was a wonderful advisor, and every aspect of this thesis has benefitted from his insight. At times I was a difficult, even unruly graduate student, and Fernando had patience with all my ideas, whether good or bad. What I’ll miss most, though, is the quick trip to Fernando’s office, coming away with new insights on everything from numerical underflow to the state of the academic community in machine learning and NLP. In addition to Fernando, this thesis was shaped by a great committee. Having Ben Taskar as committee chairman has given me the perfect excuse to interrupt his workday with new, ostensibly-thesis-related machine learning ideas. Mark Liberman and Mitch Marcus brought a much-needed linguistic perspective to a thesis on language, and many of the techniques described are based on work by Tong Zhang, who kindly served as my external committee member. Although he didn’t directly serve on my committee, Shai Ben-David got me started on the theoretical aspects of this work, and chapter 4 grew out of work I co-authored with him. I was also fortunate to have a great academic family. With brothers (and one sister!)

### Citations

9946 | Statistical Learning Theory
- Vapnik
- 1998
(Show Context)
Citation Context ...ability at least 1 − δ (over the choice of the samples), for every h ∈ H |ˆɛα(h) − ɛα(h)| < √ α 2 β √ (1 − α)2 d log(2m) − log δ + 1 − β 2m The proof is similar to standard uniform convergence proofs =-=[64, 6]-=-, but it uses Hoeffding’s inequality in a different way because the bound on the range of the random variables underlying the inequality varies with α and β. The lemma shows that as α moves away from ... |

2507 | Conditional random fields: probabilistic modeling for segmenting and labeling sequence data
- Lafferty, McCallum, et al.
- 2001
(Show Context)
Citation Context ...labels themselves have internal structure that we can take advantage of in designing the mapping ζ(x,y). Because of this, models which solve these tasks are often referred to as structured predictors =-=[42, 57, 60]-=-. Methods for structured prediction must factor problems so as to be able to perform computationally efficient inference and to be able to make accurate predictions. When we investigate adapting part ... |

2330 |
The Elements of Statistical Learning
- Hastie, Tibshirani, et al.
- 2001
(Show Context)
Citation Context ...either positive or negative) and returns the top-scoring label for this input (right). decades. A complete discussion is well beyond the scope of this thesis, but we refer the reader to Hastie et al. =-=[34]-=- for an introduction to supervised learning and to Manning and Schütze [45] and Jelinek [38] for overviews of its use in natural language processing. Shawe-Taylor and Cristianini [58] is a good refere... |

2235 | Building a Large Annotated Corpus for English: The Penn Treebank
- Marcus, Santorini, et al.
- 1993
(Show Context)
Citation Context ...speech tagging systems must be deployed in a variety of domains. In this section, we show how to use SCL to adapt a tagger from a standard resource, the Penn treebank Wall Street Journal (WSJ) corpus =-=[46]-=- to a new corpus of MEDLINE abstracts [52]. The Penn BioIE project [52] focuses on building information extraction and natural language processing systems for biomedical text. We obtained a corpus fro... |

1574 | Probability inequalities for sums of bounded random variables
- Hoeffding
- 1963
(Show Context)
Citation Context ...ion D ɛD(h) = E(x,y)∼D [h(x) ̸= y] . Suppose that we choose a hypothesis from a class of finite cardinality H. In this case, we may relate training and generalization error via Hoeffding’s inequality =-=[35]-=- and the union bound. For a training sample {xi,yi} N i=1 drawn from D, with probability 1 − δ, for every h ∈ H, ɛD(h) ≤ 1 n n∑ √ 2 log(2|H|) − log δ [h(xi) ̸= yi] + . (1.2) n i=1 This result is a sli... |

976 |
On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications
- Vapnik, Chervonenkis
- 1971
(Show Context)
Citation Context ... the bound from equation 1.2 does not apply. We may still state a uniform convergence result, however, through a measure of hypothesis class complexity known as the Vapnik-Chervonenkis (VC) dimension =-=[65]-=-. For a given training sample S of size N, the number of possible unique partitions of the points into 9two classes is 2 N . But note that for a given dimension d and number of points N, not all part... |

798 |
Statistical Methods for Speech Recognition”, The
- Jelinek
- 1997
(Show Context)
Citation Context ...es. A complete discussion is well beyond the scope of this thesis, but we refer the reader to Hastie et al. [34] for an introduction to supervised learning and to Manning and Schütze [45] and Jelinek =-=[38]-=- for overviews of its use in natural language processing. Shawe-Taylor and Cristianini [58] is a good reference for support vector machines, a particularly popular method for training and representing... |

703 | Thumbs up? Sentiment Classification Using Machine Learning Techniques
- Pang, Lee, et al.
- 2002
(Show Context)
Citation Context ...iment classification system receives as input a document and outputs a label indicating the sentiment (positive or negative) of the document. This problem has received considerable attention recently =-=[51, 63, 32]-=-. While movie reviews have been the most studied domain, sentiment analysis has been extended to a number of new domains, ranging from stock message boards to congressional floor debates [25, 61]. Res... |

533 | Semi-supervised learning using Gaussian field and harmonic functions
- Zhu, Ghahramani, et al.
- 2003
(Show Context)
Citation Context ...omplete survey of semi-supervised learning methods. 2.4.1 Manifold regularization Procedurally, the most similar methods to structural learning are those which learn a regularizer from unlabeled data =-=[68, 9, 71]-=-. Like structural learning, these methods regularize parameters by enforcing smoothness in some underlying subspace. The assumptions on the structure of the subspace are quite different, though. The s... |

517 | Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms
- Collins
(Show Context)
Citation Context ...orrect instance by a margin. Crammer et al. [22] give a more complete description of the MIRA algorithm. The application of MIRA to structured prediction is strongly influenced by the work of Collins =-=[19]-=-, who described an application of the perceptron to structured prediction. For a more general discussion and comparison of optimization techniques for structured predictors, we again refer to Taskar [... |

517 | Unsupervised Word Sense Disambiguation Rivaling Supervised Methods
- Yarowsky
- 1995
(Show Context)
Citation Context ... “processing” are more similar than either one is to “accomplishments” is necessary to give correct distances here. 2.4.2 Bootstrapping Another paradigm for exploiting unlabeled data is bootstrapping =-=[67, 17, 49, 21, 1, 47]-=-. Bootstrapping methods begin with an initial classifier. They label unlabeled instances with this classifier. Then they choose some subset of the newly-labeled instances to create a new training set ... |

509 | Semi-supervised learning literature survey
- Zhu
- 2005
(Show Context)
Citation Context ...though. In this section, we briefly review semi-supervised and unsupervised methods in text, with an emphasis on applicability for domain adaptation. Our survey here is necessarily brief, but see Zhu =-=[70]-=- for a more complete survey of semi-supervised learning methods. 2.4.1 Manifold regularization Procedurally, the most similar methods to structural learning are those which learn a regularizer from un... |

501 | Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. ACL’02
- Turney
(Show Context)
Citation Context ...iment classification system receives as input a document and outputs a label indicating the sentiment (positive or negative) of the document. This problem has received considerable attention recently =-=[51, 63, 32]-=-. While movie reviews have been the most studied domain, sentiment analysis has been extended to a number of new domains, ranging from stock message boards to congressional floor debates [25, 61]. Res... |

473 | A Maximum Entropy Model for Part-Of-Speech Tagging
- Ratnaparkhi
- 1996
(Show Context)
Citation Context ...features. Given a sentence, the task of a part of speech tagger is to label each word with its grammatical function. The best part of speech taggers encode a sentence label as a chainstructured graph =-=[53, 20, 62]-=-. In this formulation, the part of speech label factors along the cliques of the graph. We will design pivot features for individual cliques and the input features associated with them. Consider the e... |

473 | Shallow Parsing with Conditional Random Fields
- Sha, Pereira
- 2003
(Show Context)
Citation Context ...labels themselves have internal structure that we can take advantage of in designing the mapping ζ(x,y). Because of this, models which solve these tasks are often referred to as structured predictors =-=[42, 57, 60]-=-. Methods for structured prediction must factor problems so as to be able to perform computationally efficient inference and to be able to make accurate predictions. When we investigate adapting part ... |

463 | Unsupervised models for named entity classification
- Collins, Singer
- 1999
(Show Context)
Citation Context ... “processing” are more similar than either one is to “accomplishments” is necessary to give correct distances here. 2.4.2 Bootstrapping Another paradigm for exploiting unlabeled data is bootstrapping =-=[67, 17, 49, 21, 1, 47]-=-. Bootstrapping methods begin with an initial classifier. They label unlabeled instances with this classifier. Then they choose some subset of the newly-labeled instances to create a new training set ... |

378 | Feature-rich part-of-speech tagging with a cyclic dependency network
- Toutanova, Klein, et al.
- 2003
(Show Context)
Citation Context ...features. Given a sentence, the task of a part of speech tagger is to label each word with its grammatical function. The best part of speech taggers encode a sentence label as a chainstructured graph =-=[53, 20, 62]-=-. In this formulation, the part of speech label factors along the cliques of the graph. We will design pivot features for individual cliques and the input features associated with them. Consider the e... |

335 | A framework for learning predictive structures from multiple tasks and unlabeled data
- Ando, Zhang
- 2005
(Show Context)
Citation Context ...s from different domains for both tasks. . . . . . . 13 2.1 ASO algorithm as it is implemented in practice . . . . . . . . . . . . . . 23 2.2 An illustration of block SVD by type, from Ando and Zhang =-=[3]-=-. . . . . . 25 2.3 SCL algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.4 An example of how feature vector hamming distance can be misleading. . 38 3.1 A discriminating pro... |

328 |
Neural Network Learning: Theoretical Foundations
- Anthony, Bartlett
- 1999
(Show Context)
Citation Context ...ive theoretical results for generalization to new domains. Once again, we focus on binary classification and are necessarily brief, but we refer to 8Kearns and Vazirani [41] and Anthony and Bartlett =-=[6]-=- for excellent introductions to the concepts of learning theory. As before, we denote by x ∈ X a feature vector in feature space. Formally, suppose that instances are drawn from a probability distribu... |

311 | Online passiveaggressive algorithms
- Crammer, Dekel, et al.
- 2006
(Show Context)
Citation Context ...ore complex than for binary classification, but many of the basic aspects are similar. In this thesis, when we solve structured prediction problems, we use the margin infused relaxed algorithm (MIRA) =-=[22]-=-. MIRA is an online algorithm which updates the parameter vector each instance to give the minimum change to the weight vector (as measured by the L2 norm) to separate the correct instance from the to... |

284 | A Syntax-based Statistical Translation Model
- Yamada, Knight
- 2001
(Show Context)
Citation Context ...cal problem in text processing, and it serves as a first step in many pipelined systems, including higher-level syntactic processing [21, 47], information extraction [56, 52], and machine translation =-=[66]-=-. Because of their fundamental role, part of speech tagging systems must be deployed in a variety of domains. In this section, we show how to use SCL to adapt a tagger from a standard resource, the Pe... |

242 | 2005a. Online large-margin training of dependency parsers
- McDonald, Crammer, et al.
(Show Context)
Citation Context ... text processing systems. Here we show that improving a part of speech tagger in a new domain can improve a dependency parser in the new domain as well. We use the parser described by McDonald et al. =-=[48]-=-. That parser assumes that a sentence has been PoS-tagged before parsing, so it is a straightforward match for our experiments here. Accuracy 82 78 74 70 66 62 58 Dependency Parsing for 561 Test Sente... |

209 | Some statistical issues in the comparison of speech recognition algorithms
- Gillick, Cox
- 1989
(Show Context)
Citation Context ...s. Figure 3.7(b) gives results for 40,000 sentences, and Figure 3.7(c) shows corresponding significance tests, with p < 0.05 being significant. We use a McNemar paired test for labeling disagreements =-=[31]-=-. Even when we use all the 57WSJ training data available, the SCL model significantly improves accuracy over both the supervised and ASO baselines. SCL is designed to improve the accuracies for unkno... |

195 | Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
- Pang, Lee
- 2005
(Show Context)
Citation Context ...the work of Pang et al. [51], which we use as our baseline. Thomas et al. [61] use discourse structure present in congressional records to perform more accurate sentiment classification. Pang and Lee =-=[50]-=- treat sentiment analysis as an ordinal ranking problem. In our work we only show improvement for the basic model, but all of these new techniques also make use of lexical features. Thus we believe th... |

188 | Canonical correlation analysis: An overview with application to learning methods
- Hardoon, Szedmak, et al.
(Show Context)
Citation Context ...oint distribution on both views. Canonical correlation analysis finds two sets of basis vectors such that for all k, the projections of X (1) and X (2) onto the first k bases are maximally correlated =-=[36, 33]-=-. Let C be the joint covariance matrix for ( X (1),X(2)) C = Ex∼D [xx ′ ] . We may write C in the block form C = ⎡ ⎣ C11 C21 C12 C22 ⎤ ⎦ , [ (1) (1) where C11 = Ex (1) ∼D x x ′] and likewise for the o... |

177 | Learning structured prediction models: A large margin approach
- Taskar
(Show Context)
Citation Context ...labels themselves have internal structure that we can take advantage of in designing the mapping ζ(x,y). Because of this, models which solve these tasks are often referred to as structured predictors =-=[42, 57, 60]-=-. Methods for structured prediction must factor problems so as to be able to perform computationally efficient inference and to be able to make accurate predictions. When we investigate adapting part ... |

171 | Learning to classify text from labeled and unlabeled documents
- Nigam, McCallum, et al.
- 1998
(Show Context)
Citation Context ... “processing” are more similar than either one is to “accomplishments” is necessary to give correct distances here. 2.4.2 Bootstrapping Another paradigm for exploiting unlabeled data is bootstrapping =-=[67, 17, 49, 21, 1, 47]-=-. Bootstrapping methods begin with an initial classifier. They label unlabeled instances with this classifier. Then they choose some subset of the newly-labeled instances to create a new training set ... |

162 | Domain adaptation with structural correspondence learning
- Blitzer, McDonald, et al.
- 2006
(Show Context)
Citation Context ...t combining SCL with these methods yields still greater improvements, reducing error due to adaptation by as much as forty percent. The results in this chapter are drawn primarily from Blitzer et al. =-=[16]-=- and Blitzer et al. [15]. 423.1 Adapting a sentiment classification system A sentiment classification system receives as input a document and outputs a label indicating the sentiment (positive or neg... |

153 |
Domain adaptation for statistical classifiers
- Daume, Marcu
(Show Context)
Citation Context ...mial parameters of a generative parsing model to combine a large amount of training data from a source corpus (WSJ), and small amount of training data from a target corpus (Brown). Daume 61and Marcu =-=[28]-=- use an empirical Bayes model to estimate a latent variable model grouping instances into domain-specific or common across both domains. They also jointly estimate the parameters of the common classif... |

138 | Correcting sample selection bias by unlabeled data
- Huang, Smola, et al.
- 2007
(Show Context)
Citation Context ...blem that is very closely-related to domain adaptation is the problem of covariate shift (also called sample selection bias), which has been studied in the machine learning and statistics communities =-=[59, 37]-=-. Here we assume the conditional distributions PrDS [y|x] and PrDT [y|x] are identical, but the instance marginal distributions PrDS [x] and PrDT [x] are different. Several researchers have studied al... |

118 |
bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification
- Biographies
(Show Context)
Citation Context ...se methods yields still greater improvements, reducing error due to adaptation by as much as forty percent. The results in this chapter are drawn primarily from Blitzer et al. [16] and Blitzer et al. =-=[15]-=-. 423.1 Adapting a sentiment classification system A sentiment classification system receives as input a document and outputs a label indicating the sentiment (positive or negative) of the document. ... |

114 | Analysis of representations for domain adaptation
- Ben-David, Blitzer, et al.
- 2007
(Show Context)
Citation Context ...f when adaptation techniques work, as well as how to best exploit the resources we have. This chapter develops a theoretical framework for domain adaptation and comprises the work of Ben-David et al. =-=[10]-=- and Blitzer et al. [14]. We first show how to use this framework to prove bounds on the target error for classifiers which are trained in a source domain. We then demonstrate how to use the bound to ... |

105 | Detecting change in data streams
- Kifer, Ben-David, et al.
- 2004
(Show Context)
Citation Context ...vantage over other methods for comparing distributions such as L1 distance or the KL divergence: we can compute dH using finite samples from the distributions D and D ′ when H has finite VC dimension =-=[12]-=-. Furthermore, as the following theorem shows, we can compute a finite-sample approximation to dH by finding a classifier h ∈ H that maximally discriminates between instances from D and D ′. Theorem 2... |

102 | Get out the vote: Determining support or opposition from Congressional floor-debate transcripts
- Thomas, Pang, et al.
(Show Context)
Citation Context ... [51, 63, 32]. While movie reviews have been the most studied domain, sentiment analysis has been extended to a number of new domains, ranging from stock message boards to congressional floor debates =-=[25, 61]-=-. Research results have been deployed industrially in systems that gauge market reaction and summarize opinion from web pages, discussion boards, and blogs. With such widely-varying domains, researche... |

80 | Discriminative learning for differing training and test distributions
- Bickel, Brückner, et al.
- 2007
(Show Context)
Citation Context ... authors have empirically studied a special case of this in which each instance is weighted separately in the loss function, and instance weights are set to approximate the target domain distribution =-=[37, 13, 24, 39]-=-. We give a uniform convergence bound for algorithms that minimize a convex combination of multiple empirical source errors and we show that these algorithms can outperform standard empirical error mi... |

80 |
PAC generalization bounds for co-training
- Dasgupta, Littman, et al.
- 2001
(Show Context)
Citation Context ...to split the feature space into multiple 30“views” [17]. Learning in the two views model proceeds by training separate classifiers for each view and requiring that they “agree” on the unlabeled data =-=[26, 1, 2, 29, 55]-=-. In this section, we show how to relate ASO and SCL to new theoretical work on using canonical correlation analysis for multiple view learning [36, 40]. We show that a variant of the ASO optimization... |

74 |
Adaptation of Maximum Entropy Capitalizer: Little data can help a lot
- Chelba, Acero
- 2006
(Show Context)
Citation Context ... idea of using the regularizer of a linear model to encourage the target parameters to be close to the source parameters has been used previously in domain adaptation. In particular, Chelba and Acero =-=[18]-=- showed how this technique can be effective for capitalization adaptation. The major difference between our approach and theirs is that we penalize deviation from the source parameters for the weights... |

74 | A Statistical Model for Multilingual Entity Detection and Tracking
- Florian, Hassan, et al.
- 2004
(Show Context)
Citation Context ... is probably insufficient to significantly change w, but we 50can correct v, which is much smaller. We augment each labeled target instance xj with the label assigned by the source domain classifier =-=[30, 16]-=-. Then we solve minw,v ∑ j L (w′ xj + v ′ θxj,yj) + λ||w|| 2 +µ||v − vs|| 2 . Since we don’t want to deviate significantly from the source parameters, we set λ = µ = 10 −1 . Figure 3.4 shows the corre... |

71 |
Yahoo! for Amazon: Extracting market sentiment from stock message boards. APFA’01
- Das, Chen
- 2001
(Show Context)
Citation Context ... [51, 63, 32]. While movie reviews have been the most studied domain, sentiment analysis has been extended to a number of new domains, ranging from stock message boards to congressional floor debates =-=[25, 61]-=-. Research results have been deployed industrially in systems that gauge market reaction and summarize opinion from web pages, discussion boards, and blogs. With such widely-varying domains, researche... |

68 |
Seeing stars when there aren’t many stars: Graph-based semi-supervised learning for sentiment categorization
- Goldberg, Zhu
- 2006
(Show Context)
Citation Context ...iment classification system receives as input a document and outputs a label indicating the sentiment (positive or negative) of the document. This problem has received considerable attention recently =-=[51, 63, 32]-=-. While movie reviews have been the most studied domain, sentiment analysis has been extended to a number of new domains, ranging from stock message boards to congressional floor debates [25, 61]. Res... |

65 | Customizing Sentiment Classifiers to New Domains: a Case Study - Aue, Gamon - 2005 |

63 | Learning bounds for domain adaptation
- Blitzer, Crammer, et al.
- 2007
(Show Context)
Citation Context ...ques work, as well as how to best exploit the resources we have. This chapter develops a theoretical framework for domain adaptation and comprises the work of Ben-David et al. [10] and Blitzer et al. =-=[14]-=-. We first show how to use this framework to prove bounds on the target error for classifiers which are trained in a source domain. We then demonstrate how to use the bound to estimate the adaptation ... |

58 | Parsing biomedical literature
- Lease, M, et al.
- 2005
(Show Context)
Citation Context ...h tagging While the literature on unsupervised part of speech tagging is quite large, to the best of our knowledge, we are the first to adapt part of speech taggers to new domains. Lease and Charniak =-=[43]-=- adapt a WSJ parser to biomedical text without any biomedical treebanked data. However, they assume other labeled resources in the target domain. In section 3.2.3 we give similar parsing results, but ... |

50 | Nonparametric transforms of graph kernels for semi-supervised learning
- Zhu, Kandola, et al.
(Show Context)
Citation Context ..., for a real-valued predictor, we require that the predictions be close for points that are close. How can we decide which points are close, though? One way to proceed is the data manifold assumption =-=[9, 72]-=-: We assume that the input instances x are sampled from a low dimensional manifold. The neighborhood graph on the unlabeled data can provide us with an indication of the structure of that manifold. Su... |

47 | Input-dependent estimation of generalization error under covariate shift
- Sugiyama, Müller
- 2005
(Show Context)
Citation Context ...blem that is very closely-related to domain adaptation is the problem of covariate shift (also called sample selection bias), which has been studied in the machine learning and statistics communities =-=[59, 37]-=-. Here we assume the conditional distributions PrDS [y|x] and PrDT [y|x] are identical, but the instance marginal distributions PrDS [x] and PrDT [x] are different. Several researchers have studied al... |

46 |
Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled
- Belkin
(Show Context)
Citation Context ...omplete survey of semi-supervised learning methods. 2.4.1 Manifold regularization Procedurally, the most similar methods to structural learning are those which learn a regularizer from unlabeled data =-=[68, 9, 71]-=-. Like structural learning, these methods regularize parameters by enforcing smoothness in some underlying subspace. The assumptions on the structure of the subspace are quite different, though. The s... |

45 | Understanding the Yarowsky Algorithm
- Abney
- 2004
(Show Context)
Citation Context ...to split the feature space into multiple 30“views” [17]. Learning in the two views model proceeds by training separate classifiers for each view and requiring that they “agree” on the unlabeled data =-=[26, 1, 2, 29, 55]-=-. In this section, we show how to relate ASO and SCL to new theoretical work on using canonical correlation analysis for multiple view learning [36, 40]. We show that a variant of the ASO optimization... |

41 | Learning from multiple sources
- Crammer, Kearns, et al.
- 2008
(Show Context)
Citation Context ...s result can be found Appendix A.2. The main step in the proof is a variant of the triangle inequality in which the sides of the triangle represent errors of one decision rule with respect to another =-=[10, 23]-=-. The bound is relative to λ. When the combined error of the ideal hypothesis is large, there is no classifier that performs well on both the source and target domains, so we cannot hope to find a goo... |

41 | Two view learning: SVM-2K, theory and practice
- Farquhar, Hardoon, et al.
- 2006
(Show Context)
Citation Context ...to split the feature space into multiple 30“views” [17]. Learning in the two views model proceeds by training separate classifiers for each view and requiring that they “agree” on the unlabeled data =-=[26, 1, 2, 29, 55]-=-. In this section, we show how to relate ASO and SCL to new theoretical work on using canonical correlation analysis for multiple view learning [36, 40]. We show that a variant of the ASO optimization... |

35 | On the difficulty of approximately maximizing agreements - Ben-David, Eiron, et al. - 2003 |