Over the past several years, a number of information discovery and access tools have been introduced in the Internet, including Archie, Gopher, Netfind, and WAIS. These tools have become quite popular, and are helping to redefine how people think about wide-area network applications. Yet, they are not well suited to supporting the future information infrastructure, which will be characterized by enormous data volume, rapid growth in the user base, and burgeoning data diversity. In this paper we indicate trends in these three dimensions and survey problems these trends will create for current approaches. We then suggest several promising directions of future resource discovery research, along with some initial results from projects carried out by members of the Internet Research Task Force Research Group on Resource Discovery and Directory Service.

A fundamental problem with the World Wide Web is the enormous number of resources available and the difficulty of locating and tracking everything. In this paper we will discuss two tools, GENVL and WWWW, designed to deal in different ways with resource location on the WWW. GENVL is an interactive user-driven hierarchical virtual library system for cataloguing Web resources. The real power of GENVL comes from the built-in recursion which is the key to extendibility and to avoiding the generation of massive linear lists. GENVL has been accessed 112,000 times in 4 months. WWWW - the WWW Worm - is a resource location tool. It is intended to locate almost all of the WWW-addressable resources on the Internet, and provide a powerful search interface to those resources. Searches can be performed on document titles, reference hypertext, or within the components of the URL name strings of documents - for example to locate all mpeg movies in Finland. WWWW has been accessed 60,000 times in 45 day...

Submitted inpartial ful llment of the requirements

. In many large organizations there has been a proliferation of database systems to handle ever increasing volumes of information. In order to explore a potentially huge on-line information space, we must develop an architecture which allows for the dynamic data driven construction of inter--database node relationships in an incremental manner. In this paper we introduce the FINDIT architecture which uses information meta-types to provide a basis for such an organization and, consequently, provides a platform for interoperability. A distinction is made between the information and inter-node relationship spaces to achieve scalability. Tassili language primitives are used for the incremental building of dynamic inter--node relationships based upon usage considerations. Keywords: Multidatabases, Federated Databases, Interoperability, Heterogeneous and Autonomous Databases, Query Languages. 1. Introduction Sharing information among autonomous heterogeneous databases has been researched e...

This paper describes WEBFIND, an application that discovers scientific papers made available by their authors on the web. WEBFIND uses a novel approach to performing information retrieval on the worldwide web. The approach is to use a combination of external information sources as a guide for locating where to look for information on the web. The external information sources used by WEBFIND are

WebCrawler, the first comprehensive full-text search engine for the World-Wide Web, has played a fundamental role in making the Web easier to use for millions of people. Its invention and subsequent evolution, spanning a three-year period, helped fuel the Web's growth by creating a new way of navigating hypertext. Before search engines like WebCrawler, users found Web documents by following hypertext links from one document to another. When the Web was small and its documents shared the same fundamental purpose, users could find documents with relative ease. However, the Web quickly grew to millions of pages making navigation difficult. WebCrawler assists users in their Web navigation by automating the task of link traversal, creating a searchable index of the web, and fulfilling searchers’ queries from the index. To use WebCrawler, a user issues a query to a pre-computed index, quickly retrieving a list of documents that match the query. This dissertation describes WebCrawler's scientific contributions: a method for choosing a subset of the Web to index; an approach to creating a search service that is easy to use; a new way to rank search results that can generate highly effective results for both naive and expert searchers; and an architecture for the service that has effectively handled a three-order-of-magnitude increase in load. This dissertation also describes how WebCrawler evolved to accommodate the extraordinary growth of the Web. This growth affected WebCrawler not only by increasing the size and scope of its index, but also by increasing the demand for its service. Each of WebCrawler’s components had to change to accommodate this growth: the crawler had to download more documents, the full-text index had to become more efficient at storing and finding those documents, and the service had to accommodate heavier demand. Such changes were not only related to scale, however: the evolving nature of the Web meant that functional changes were necessary, too, such as the ability to handle naive queries from searchers.

This paper describes object-oriented extensions to the Harvest information discovery and access system that allow users to define type hierarchies and associated access methods for Web objects. These extensions enable Mosaic and the Web to represent, manipulate and display arbitrarily complex data in application-specific ways. Types and access methods are extensible; new ones can be defined and exported from Web servers and imported into Mosaic on demand, without changes to Mosaic and HTML for each new data type. This paper describes our design and prototype system implementation, including an application we built using the system. 1 Merkel is currently a Leutenant in the U.S. Air Force, stationed at Maxwell Air Force Base. 1 Introduction The World Wide Web is currently used primarily for sharing text and graphics; it provides little support for applications with semantically deeper data or more complex processing requirements. For example, Mosaic cannot easily be used to share and...

. This paper describes Rama, a first generation experimental information retrieval and filtering system that attempts to recover useful information from various Internet sources including USENIX news and anonymous FTP servers. The focus of the Rama system to date has been on building a distributed query and information retrieval system, which provides an interface to heterogeneous information services. A user of Rama sends one or more asynchronous queries to a Rama server using existing SMTP e-mail clients. The server periodically searches local and remote Internet services. Searches are prefiltered with the use of timestamps. Data objects which are newer than the timestamp are then searched via a query mechanism which relies on a combination of vector-distance and pattern matching operands, and boolean operators. Results are weighted according to how closely they match queries and are posted via e-mail to the user. Input to the e-mail client can be further filtered -- one can use the ...

: Information retrieval in the worldwide web environment poses unique challenges. The most common approaches involve indexing, but indexes introduce centralization and can never be up-to-date. This paper advocates using external databases and information sources as guides for locating worldwide web information. This approach has been implemented in WEBFIND, a tool which discovers scientific papers made available on the web by their authors. The external information sources used by WEBFIND are MELVYL, the online University of California library catalog, and NETFIND, a service for finding email addresses. WEBFIND combines the information available from these services in order to find good starting points for searching for the papers that a user wants. At several stages in its operation, WEBFIND must solve instances of what we call the field matching problem. This problem is to determine whether or not two syntactic values are alternative representations of the same semantic entity. For e...

With the growth of the Internet as a whole, and the popularity of the World-Wide Web information system in particular, Uniform Resource Locators (URLs) have become the de facto standard for naming on-line resources. The usage of URLs on a large scale has highlighted their deficiences, and new technologies are being developed which attempt to address these - notably the Uniform Resource Name (URN) and Uniform Resource Characteristic (URC). This report outlines some of the problems with URLs, provides an introduction to URNs and URCs, and identifies areas where the approach adopted for the Simple Discovery Protocol may be a viable solution. 1 Preamble It is assumed that the reader is already familiar with the Internet, Uniform Resource Locators[1], and the World-Wide Web. There are now a large number of elementary texts which introduce these concepts, for example[2]. This report discusses problems arising from the current usage of URLs on the Internet, and introduces some possible so...