Results 1  10
of
47
Adaptive Intersection and tThreshold Problems
, 2002
"... Consider the problem of computing the intersection of k sorted sets. In the comparison model, we prove a new lower bound which depends on the nondeterministic complexity of the instance, and implies that the algorithm of Demaine, LopezOrtiz and Munro [2] is usually optimal in this \adaptive" ..."
Abstract

Cited by 43 (12 self)
 Add to MetaCart
Consider the problem of computing the intersection of k sorted sets. In the comparison model, we prove a new lower bound which depends on the nondeterministic complexity of the instance, and implies that the algorithm of Demaine, LopezOrtiz and Munro [2] is usually optimal in this \adaptive" sense. We extend the lower bound and the algorithm to the tThreshold Problem, which consists in nding the elements which are in at least t of the k sets. These problems are motivated by boolean queries in text database systems.
Adaptive searching in succinctly encoded binary relations and treestructured documents (Extended Abstract)
 THEORETICAL COMPUTER SCIENCE
, 2005
"... This paper deals with succinct representations of data types motivated by applications in posting lists for search engines, in querying XML documents, and in the more general setting (which extends XML) of multilabeled trees, where several labels can be assigned to each node of a tree. To find th ..."
Abstract

Cited by 42 (14 self)
 Add to MetaCart
(Show Context)
This paper deals with succinct representations of data types motivated by applications in posting lists for search engines, in querying XML documents, and in the more general setting (which extends XML) of multilabeled trees, where several labels can be assigned to each node of a tree. To find the set of references corresponding to a set of keywords, one typically intersects the list of references associated with each keyword. We view this instead as having a single list of objects [n] = {1,..., n} (the references), each of which has a subset of the labels [σ] = {1,..., σ} (the keywords) associated with it. We are able to find the objects associated with an arbitrary set of keywords in time O(δk lg lg σ) using a data structure requiring only t(lg σ +o(lg σ)) bits, where δ is the number of steps required by a nondeterministic algorithm to check the answer, k is the number of keywords in the query, σ is the size of the set from which the keywords are chosen, and t is the number of associations between references and keywords. The data structure is succinct in that it differs from the space needed to write down all t occurrences of keywords by only a lower order term. An XML document is, for our purpose, a labeled rooted tree. We deal primarily with “nonrecursive labeled trees”, where no label occurs more than once on any root to leaf path. We find the set of nodes which path from the root include a set of keywords in the same time, O(δk lg lg σ), on a representation of the tree using essentially minimum space, 2n + n(lg σ + o(lg σ)) bits, where n is the number of nodes in the tree. If we permit nodes to have multiple
Faster adaptive set intersections for text searching
 EXPERIMENTAL ALGORITHMS: 5TH INTERNATIONAL WORKSHOP, WEA 2006, CALA GALDANA, MENORCA
, 2006
"... The intersection of large ordered sets is a common problem in the context of the evaluation of boolean queries to a search engine. In this paper we engineer a better algorithm for this task, which improves over those proposed by Demaine, Munro and LópezOrtiz [SODA 2000/ALENEX 2001], by using a va ..."
Abstract

Cited by 34 (4 self)
 Add to MetaCart
(Show Context)
The intersection of large ordered sets is a common problem in the context of the evaluation of boolean queries to a search engine. In this paper we engineer a better algorithm for this task, which improves over those proposed by Demaine, Munro and LópezOrtiz [SODA 2000/ALENEX 2001], by using a variant of interpolation search. More specifically, our contributions are threefold. First, we corroborate and complete the practical study from Demaine et al. on comparison based intersection algorithms. Second, we show that in practice replacing binary search and galloping (onesided binary) search [4] by interpolation search improves the performance of each main intersection algorithms. Third, we introduce and test variants of interpolation search: this results in an even better intersection algorithm.
Experimental Analysis of a Fast Intersection Algorithm for Sorted Sequences
 In Proceedings of 12th International Conference on String Processing and Information Retrieval (SPIRE
, 2005
"... Abstract. This work presents an experimental comparison of intersection algorithms for sorted sequences, including the recent algorithm of BaezaYates. This algorithm performs on average less comparisons than the total number of elements of both inputs (n and m respectively) when n = αm (α> 1). W ..."
Abstract

Cited by 28 (1 self)
 Add to MetaCart
Abstract. This work presents an experimental comparison of intersection algorithms for sorted sequences, including the recent algorithm of BaezaYates. This algorithm performs on average less comparisons than the total number of elements of both inputs (n and m respectively) when n = αm (α> 1). We can find applications of this algorithm on query processing in Web search engines, where large intersections, or differences, must be performed fast. In this work we concentrate in studying the behavior of the algorithm in practice, using for the experiments test data that is close to the actual conditions of its applications. We compare the efficiency of the algorithm with other intersection algorithm and we study different optimizations, showing that the algorithm is more efficient than the alternatives in most cases, especially when one of the sequences is much larger than the other. 1
An Experimental Investigation of Set Intersection Algorithms for Text Searching
, 2009
"... The intersection of large ordered sets is a common problem in the context of the evaluation of boolean queries to a search engine. In this paper we propose several improved algorithms for computing the intersection of sorted arrays, and in particular for searching sorted arrays in the intersection ..."
Abstract

Cited by 26 (2 self)
 Add to MetaCart
The intersection of large ordered sets is a common problem in the context of the evaluation of boolean queries to a search engine. In this paper we propose several improved algorithms for computing the intersection of sorted arrays, and in particular for searching sorted arrays in the intersection context. We perform an experimental comparison with the algorithms from the previous studies from Demaine, LópezOrtiz and Munro [ALENEX 2001], and from BaezaYates and Salinger [SPIRE 2005]; in addition, we implement and test the intersection algorithm from Barbay and Kenyon [SODA 2002] and its randomized variant [SAGA 2003]. We consider both the random data set from BaezaYates and Salinger, the Google queries used by Demaine et al., a corpus provided by Google and a larger corpus from the TREC Terabyte 2006 efficiency query stream, along with its own query log. We measure the performance both in terms of the number of comparisons and searches performed, and in terms of the CPU time on two different architectures. Our results confirm or improve the results from both previous studies in their respective context (comparison model on real data and CPU measures on random data), and extend them to new contexts. In particular we show that valuebased search algorithms perform well in posting lists in terms of the number of comparisons performed.
Querying Multiple Sets of Discovered Rules
 In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’02
, 2002
"... Rule mining is an important data mining task that has been applied to numerous realworld applications. Often a rule mining system generates a large number of rules and only a small subset of them is really useful in applications. Although there exist some systems allowing the user to query the disc ..."
Abstract

Cited by 22 (6 self)
 Add to MetaCart
(Show Context)
Rule mining is an important data mining task that has been applied to numerous realworld applications. Often a rule mining system generates a large number of rules and only a small subset of them is really useful in applications. Although there exist some systems allowing the user to query the discovered rules, they are less suitable for complex ad hoc querying of multiple data mining rulebases to retrieve interesting rules. In this paper, we propose a new powerful rule query language RuleQL for querying multiple rulebases that is modeled after SQL and has rigorous theoretical foundations of a rulebased calculus. In particular, we first propose a rulebased calculus RC based on the firstorder logic, and then present the language RuleQL that is at least as expressive as the safe fragment of RC. We also propose a number of efficient query evaluation techniques for RuleQL and test them experimentally on some representative queries to demonstrate the feasibility of RuleQL.
Improving the Performance of List Intersection
"... List intersection is a central operation, utilized excessively for query processing on text and databases. We present list intersection algorithms for an arbitrary number of sorted and unsorted lists tailored to the characteristics of modern hardware architectures. Two new list intersection algorith ..."
Abstract

Cited by 16 (0 self)
 Add to MetaCart
List intersection is a central operation, utilized excessively for query processing on text and databases. We present list intersection algorithms for an arbitrary number of sorted and unsorted lists tailored to the characteristics of modern hardware architectures. Two new list intersection algorithms are presented for sorted lists. The first algorithm, termed Dynamic Probes, dynamically decides the probing order on the lists exploiting information from previous probes at runtime. This information is utilized as a cacheresident microindex. The second algorithm, termed Quantilebased, deduces in advance a good probing order, thus avoiding the overhead of adaptivity and is based on detecting lists with nonuniform distribution of document identifiers. For unsorted lists, we present a novel hashbased algorithm that avoids the overhead of sorting. A detailed experimental evaluation is presented based on real and synthetic data using existing chip multiprocessor architectures with eight cores, validating the efficiency and efficacy of the proposed algorithms. 1.
Alternation and Redundancy Analysis of the Intersection Problem
, 2006
"... The intersection of sorted arrays problem has applications in search engines such as Google. Previous work propose and compare deterministic algorithms for this problem, in an adaptive analysis based on the encoding size of a certificate of the result (cost analysis). We define the alternation analy ..."
Abstract

Cited by 16 (3 self)
 Add to MetaCart
The intersection of sorted arrays problem has applications in search engines such as Google. Previous work propose and compare deterministic algorithms for this problem, in an adaptive analysis based on the encoding size of a certificate of the result (cost analysis). We define the alternation analysis, based on the nondeterministic complexity of an instance. In this analysis we prove that there is a deterministic algorithm asymptotically performing as well as any randomized algorithm in the comparison model. We define the redundancy analysis, based on a measure of the internal redundancy of the instance. In this analysis we prove that any algorithm optimal in the redundancy analysis is optimal in the alternation analysis, but that there is a randomized algorithm which performs strictly better than any deterministic algorithm in the comparison model. Finally, we describe how those results can be extended beyond the comparison model.
Fast Set Intersection in Memory
"... Set intersection is a fundamental operation in information retrieval and database systems. This paper introduces linear space data structures to represent sets such that their intersection can be computed in a worstcase efficient way. In general, given k (preprocessed) sets, with totally n elements ..."
Abstract

Cited by 15 (1 self)
 Add to MetaCart
(Show Context)
Set intersection is a fundamental operation in information retrieval and database systems. This paper introduces linear space data structures to represent sets such that their intersection can be computed in a worstcase efficient way. In general, given k (preprocessed) sets, with totally n elements, we will show how to compute their intersection in expected time O(n / √ w + kr), where r is the intersection size and w is the number of bits in a machineword. In addition,we introduce a very simple version of this algorithm that has weaker asymptotic guarantees but performs even better in practice; both algorithms outperform the state of the art techniques for both synthetic and real data sets and workloads. 1.
Compressed selfindices supporting conjunctive queries on document collections
 in: Proc. 17th SPIRE, 2010
"... Abstract. We prove that a document collection, represented as a unique sequence T of n terms over a vocabulary Σ, can be represented in nH0(T) + o(n)(H0(T) + 1) bits of space, such that a conjunctive query t1 ∧ · · · ∧ tk can be answered in O(kδ log log Σ) adaptive time, where δ is the instanc ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
(Show Context)
Abstract. We prove that a document collection, represented as a unique sequence T of n terms over a vocabulary Σ, can be represented in nH0(T) + o(n)(H0(T) + 1) bits of space, such that a conjunctive query t1 ∧ · · · ∧ tk can be answered in O(kδ log log Σ) adaptive time, where δ is the instance difficulty of the query, as defined by Barbay and Kenyon in their SODA’02 paper, and H0(T) is the empirical entropy of order 0 of T. As a comparison, using an inverted index plus the adaptive intersection algorithm by Barbay and Kenyon takes O(kδ log nM δ), where nM is the length of the shortest and longest occurrence lists, respectively, among those of the query terms. Thus, we can replace an inverted index by a more spaceefficient inmemory encoding, outperforming the query performance of inverted indices when the ratio nM δ is ω(log Σ).