Over the past several years, a number of information discovery and access tools have been introduced in the Internet, including Archie, Gopher, Netfind, and WAIS. These tools have become quite popular, and are helping to redefine how people think about wide-area network applications. Yet, they are not well suited to supporting the future information infrastructure, which will be characterized by enormous data volume, rapid growth in the user base, and burgeoning data diversity. In this paper we indicate trends in these three dimensions and survey problems these trends will create for current approaches. We then suggest several promising directions of future resource discovery research, along with some initial results from projects carried out by members of the Internet Research Task Force Research Group on Resource Discovery and Directory Service.

Despite blossoming computer network bandwidths and the emergence of hypertext and CD-ROM databases, little progress has been made towards uniting the world's library-style bibliographic databases. While a few advanced distributed retrieval systems can broadcast a query to hundreds of participating databases, experience shows that local users almost always clog library retrieval systems. Hence broadcast remote queries will clog nearly every system. The premise of this work is that broadcast-based systems do not scale to world-wide systems. This project describes an indexing scheme that will permit thorough yet efficient searches of millions of retrieval systems. Our architecture will work with an arbitrary number of indexing companies and information providers, and, in the market place, could provide economic incentive for cooperation between database and indexing services. We call our scheme distributed indexing, and believe it will help researchers disseminate and locate both publishe...

This paper describes the architecture and the design decisions behind a resource discovery tool that we prototyped to knit together the Internet's resource discovery fabric. We call the architecture distributed indexing or Indie for short. Indie consists of a directory of services and an unlimited number of broker databases that index their own data, data stored in other brokers, and data available from other resource discovery services. The indexing mechanism doubles as a lazily consistent data replication mechanism that can replicate the directory of services or any other broker at will. An Indie broker automatically clusters references to related objects stored in other autonomous discovery and database services. Since Indie brokers cluster related information skimmed from thousands of scattered services, efficient exhaustive search is possible. This centralization led to the success of the archie file location service. In a way, Indie is a generalized archie that locates autonomou...

Rapidly increasing global Internet connectivity offers tremendous opportunities for collaboration and information sharing. An important problem in this environment is how to discover resources of interest, such as documents, network services, and people. In this paper we discuss a number of aspects of the resource discovery problem, and summarize results from efforts to address these problems carried out in the Networked Resource Discovery Project at the University of Colorado. CR Categories and Subject Descriptors: C.2.4 [Computer-Communication Networks]: Distributed Systems - distributed applications; H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing - indexing methods; H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval - retrieval models, search process, selection process; H.3.4 [Information Storage and Retrieval]: Systems and Software - information networks; H.3.5 [Information Storage and Retrieval]: Online Information Services - data bank sharing; H.3.6 [Information Storage and Retrieval]: Library Automation - large text archives; H.4.3 [Infor- mation Systems Applications]: Communications Applications - bulletin boards, electronic mail; K.4.1 [Com- puters And Society]: Public Policy Issues - privacy. General Terms: Design, Experimentation, Measurement Additional Key Words and Phrases: resource discovery, directory service, global distribution, administrative decentralization.

This thesis addresses the challenge of locating people, resources, and other objects in the global Internet. As the Internet grows beyond a million hosts in tens of thousands of organizations, it is increasingly difficult to locate any particular object. Hierarchical name services are frustrating, because users must guess the unique names for objects or navigate the name space to find information. Descriptive (i.e. relational) name services offer the promise of simple resource location through a non-procedural query language. Users locate resources by describing resource attributes. This thesis makes the promise of descriptive name services real by providing fast query processing in large internets. The key to speed in descriptive query processing is constraining the search space using two new techniques, called an active catalog and meta-data caching. The active catalog constrains the search space for a query by returning a list of data repositories where the answer to the query is li...

This document proposes a logical and coordinated structure within the Internet, of mid-level networks offering a set of minimum technical services within the mid-level network itself and to its peer networks (the term mid-level is used as a generic term to represent all regional and similar networks, which, due to continuous evolutions and transitions, can no longer be termed "regional " [MAN]). It discusses the economics, pros and cons of offering these services, as well as areas in which these mid-level networks can work together as a team.

This document proposes a set of technical services that each Internet mid-level network can offer within the mid-level network itself and and to its peer networks. The term "mid-level" is used as a generic term to represent all regional and similar networks, which, due to continuous evolutions and transitions, can no longer be termed "regional" [MAN]. It discusses the pros and cons of offering these services, as well as areas in which mid-level networks can work together.