Abstract

A multi-database model of distributed information retrieval is presented, in which people are assumed to have access to many searchable text databases. In such an environment, full-text information retrieval consists of discovering database contents, ranking databases by their expected ability to satisfy the query, searching a small number of databases, and merging results returned by different databases. This paper presents algorithms for each task. It also discusses how to reorganize conventional test collections into multi-database testbeds, and evaluation methodologies for multi-database experiments. A broad and diverse group of experimental results is presented to demonstrate that the algorithms are effective, efficient, robust, and scalable. 1. INTRODUCTION Wide area networks, particularly the Internet, have transformed how people interact with information. Much of the routine information access by the general public is now based on full-text information retrieval, as opposed t...

Citations

359	Searching Distributed Collections With Inference Networks - Callan, Zhihong, et al. - 1995
289	Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval - Robertson, Walker - 1994
212	Inference networks for document retrieval - Turtle - 1990
158	Generalizing GlOSS to VectorSpace Databases and Broker Hierarchies - Gravano, Garcia-Molina - 1995
139	The effectiveness of GLOSS for the text database discovery problem - Gravano, García-Molina, et al. - 1994
139	Cluster-based Language Models For Distributed Retrieval.” To Appear - Xu, Croft - 1999
134	Query-based sampling of text databases - Callan, Connell, et al. - 1999
113	A Decision-Theoretic Approach to Database Selection in Networked IR. Submitted for publication; available at http://amaunet.cs.uni-dortmund.de/ir/reports/98 - Fuhr - 1998
104	Automatic discovery of language models for text databases - Callan, Connell, et al. - 1999
103	Learning collection fusion strategies - Voorhees, Gupta, et al. - 1995
98	The Collection Fusion Problem - Voorhees, Gupta, et al. - 1994
87	Latent Semantic Indexing (LSI) and TREC-2 - Dumais - 1993
69	Methods for information server selection - Hawking, Thistlewaite - 1999
64	Evaluating database selection techniques: A testbed and experiment - French, Powell, et al. - 1998
64	Dissemination of collection wide information in a distributed information retrieval system - Viles, French - 1995
48	Recent experiments with INQUERY - Allan, Ballesteros, et al. - 1996
45	Efficient probabilistic inference for text retrieval - Turtle, Croft - 1991
40	Numerical Recipies: The art of Scientific Computing - Press, Teukolsky, et al. - 1992
34	NetSerf: Using Semantic Knowledge to Find Internet Information Archives - Chakravarthy, Haase - 1995
24	The E ects of Query-Based Sampling on Automatic Database Selection Algorithms - Callan, Powell, et al. - 2000
24	STARTS stanford protocol proposal for internet retrieval and search - Gravano, Chang, et al. - 1997
22	TREC-3 ad-hoc, routing retrieval and thresholding experiments using PIRCS - Kwok, Grunfeld, et al.
21	Document retrieval over networks wherein ranking and relevance scores are computed at the client for multiple database documents. U.S. Patent 5,659,732 - Kirsch - 1997
20	An Experimental Comparison of the Effectiveness of Computers and Humans as Search Intermediaries - Marcus - 1983
11	Effective retrieval of distributed collections - Xu, Callan - 1998
10	Facts from Figures - Moroney - 1974
9	Measures in collection ranking evaluation - Lu, Callan, et al. - 1996
6	Applying inference networks to multiple collection searching - Lu, Callan, et al. - 1996
2	Distributed IR testbed definition: trec123-100-bysourcecallan99.v2a - Callan - 1999
2	Probing a collection to discover its language model - Du, Callan - 1998
1	Distributed IR testbed definition: trecvlc1-921-bysourcecallan99 - Callan - 1999