This thesis is about estimating probabilistic models to uncover useful hidden structure in data; specifically, we address the problem of discovering syntactic structure in natural language text. We present three new parameter estimation techniques that generalize the standard approach, maximum likelihood estimation, in different ways. Contrastive estimation maximizes the conditional probability of the observed data given a “neighborhood” of implicit negative examples. Skewed deterministic annealing locally maximizes likelihood using a cautious parameter search strategy that starts with an easier optimization problem than likelihood, and iteratively moves to harder problems, culminating in likelihood. Structural annealing is similar, but starts with a heavy bias toward simple syntactic structures and gradually relaxes the bias. Our estimation methods do not make use of annotated examples. We consider their performance in both an unsupervised model selection setting, where models trained under different initialization and regularization settings are compared by evaluating the training objective on a small set of unseen, unannotated development data, and supervised model selection, where the most accurate model on the development set (now with annotations)

Abstract not found

We introduce two ways to detect entailment using distributional semantic representations of phrases. Our first experiment shows that the entailment relation between adjective-noun constructions and their head nouns (big cat | = cat), once represented as semantic vector pairs, generalizes to lexical entailment among nouns (dog | = animal). Our second experiment shows that a classifier fed semantic vector pairs can similarly generalize the entailment relation among quantifier phrases (many dogs|=some dogs) to entailment involving unseen quantifiers (all cats|=several cats). Moreover, nominal and quantifier phrase entailment appears to be cued by different distributional correlates, as predicted by the type-based view of entailment in formal semantics. 1

We describe a open-domain information extraction method for extracting concept-instance pairs from an HTML corpus. Most earlier approaches to this problem rely on combining clusters of distributionally similar terms and conceptinstance pairs obtained with Hearst patterns. In contrast, our method relies on a novel approach for clustering terms found in HTML tables, and then assigning concept names to these clusters using Hearst patterns. The method can be efficiently applied to a large corpus, and experimental results on several datasets show that our method can accurately extract large numbers of concept-instance pairs. 1.

Can a system that “learns from reading” figure out on it’s own the semantic classes of arbitrary noun phrases? This is essential for text understanding, given the limited coverage of proper nouns in lexical resources such as WordNet. Previous methods that use lexical patterns to discover hypernyms suffer from limited precision and recall. We present methods based on lexical patterns that find hypernyms of arbitrary noun phrases with high precision. This more than doubles the recall of proper noun hypernyms provided by WordNet at a modest cost to precision. We also present a novel method using a Hidden Markov Model (HMM) to extend recall further.

The Semantic Web’s need for machine understandable content has led researchers to attempt to automatically acquire such content from a number of sources, including the web. To date, such research has focused on “document-driven” systems that individually process a small set of documents, annotating each with respect to a given ontology. This paper introduces OntoSyphon, an alternative that strives to more fully leverage existing ontological content while scaling to extract comparatively shallow content from millions of documents. OntoSyphon operates in an “ontology-driven” manner: taking any ontology as input, OntoSyphon uses the ontology to specify web searches that identify possible semantic instances, relations, and taxonomic information. Redundancy in the web, together with information from the ontology, is then used to automatically verify these candidate instances and relations, enabling OntoSyphon to operate in a fully automated, unsupervised manner. A prototype of OntoSyphon is fully implemented and we present experimental results that demonstrate substantial instance learning in a variety of domains based on independently constructed ontologies. We also introduce new methods for improving instance verification, and demonstrate that they improve upon previously known techniques.

In this thesis we develop a discriminative learning method for dependency parsing using online large-margin training combined with spanning tree inference algorithms. We will show that this method provides state-of-the-art accuracy, is extensible through the feature set and can be implemented efficiently. Furthermore, we display the language independent nature of the method by evaluating it on over a dozen diverse languages as well as show its practical applicability through integration into a sentence compression system. We start by presenting an online large-margin learning framework that is a generaliza- tion of the work of Crammer and Singer [34, 37] to structured outputs, such as sequences and parse trees. This will lead to the heart of this thesis – discriminative dependency pars- ing. Here we will formulate dependency parsing in a spanning tree framework, yielding efficient parsing algorithms for both projective and non-projective tree structures. We will then extend the parsing algorithm to incorporate features over larger substructures with- out an increase in computational complexity for the projective case. Unfortunately, the non-projective problem then becomes NP-hard so we provide structurally motivated ap- proximate algorithms. Having defined a set of parsing algorithms, we will also define a rich feature set and train various parsers using the online large-margin learning framework. We then compare our trained dependency parsers to other state-of-the-art parsers on 14 diverse languages: Arabic, Bulgarian, Chinese, Czech, Danish, Dutch, English, German, Japanese, Portuguese, Slovene, Spanish, Swedish and Turkish. Having built an efficient and accurate discriminative dependency parser, this thesis will then turn to improving and applying the parser. First we will show how additional re- sources can provide useful features to increase parsing accuracy and to adapt parsers to new domains. We will also argue that the robustness of discriminative inference-based learning algorithms lend themselves well to dependency parsing when feature representa- tions or structural constraints do not allow for tractable parsing algorithms. Finally, we integrate our parsing models into a state-of-the-art sentence compression system to show its applicability to a real world problem.

We study how to extend a large knowledge base (Freebase) by reading relational information from a large Web text corpus. Previous studies on extracting relational knowledge from text show the potential of syntactic patterns for extraction, but they do not exploit background knowledge of other relations in the knowledge base. We describe a distributed, Web-scale implementation of a path-constrained random walk model that learns syntactic-semantic inference rules for binary relations from a graph representation of the parsed text and the knowledge base. Experiments show significant accuracy improvements in binary relation prediction over methods that consider only text, or only the existing knowledge base. 1

Web search engines can greatly benefit from knowledge about attributes of entities present in search queries. In this paper, we introduce lightly-supervised methods for extracting entity attributes from natural language text. Using these methods, we are able to extract large numbers of attributes of different entities at fairly high precision from a large natural language corpus. We compare our methods against a previously proposed pattern-based relation extractor, showing that the new methods give considerable improvements over that baseline. We also demonstrate that query expansion using extracted attributes improves retrieval performance on underspecified information-seeking queries. 1 Attributes in Web Search Web search engines receive numerous queries requesting information, often focused on a specific entity, such as a person, place or organization. These queries are sometimes general requests, such as “bio of George Bush, ” or specific requests, such as “new york mayor. ” Accurately identifying the entity (new york) or related attributes (mayor) can improve search results in several ways [1]. For

The ability to aggregate huge volumes of queries over a large population of users allows search engines to build precise models for a variety of query-assistance features such as query recommendation, correction, etc. Yet, no matter how much data is aggregated, the long-tail distribution implies that a large fraction of queries are rare. As a result, most query assistance services perform poorly or are not even triggered on long-tail queries. We propose a method to extend the reach of query assistance techniques (and in particular query recommendation) to long-tail queries by reasoning about rules between query templates rather than individual query transitions, as currently done in query-flow graph models. As a simple example, if we recognize that ‘Montezuma’ is a city in the rare query “Montezuma surf”andif therule‘<city> surf → <city> beach ’ has been observed, we are able to offer “Montezuma beach ” as a recommendation, even if the two queries were never observed in a same session. We conducted experiments to validate our hypothesis, first via traditional small-scale editorial assessments but more interestingly via a novel automated large scale evaluation methodology. Our experiments show that general coverage can be relatively increased by 24 % using templates without penalizing quality. Furthermore, for 36 % of the 95M queries in our query flow graph, which have no out edges and thus could not be served recommendations, we can now offer at least one recommendation in 98 % of the cases.