Over the past several years, a number of information discovery and access tools have been introduced in the Internet, including Archie, Gopher, Netfind, and WAIS. These tools have become quite popular, and are helping to redefine how people think about wide-area network applications. Yet, they are not well suited to supporting the future information infrastructure, which will be characterized by enormous data volume, rapid growth in the user base, and burgeoning data diversity. In this paper we indicate trends in these three dimensions and survey problems these trends will create for current approaches. We then suggest several promising directions of future resource discovery research, along with some initial results from projects carried out by members of the Internet Research Task Force Research Group on Resource Discovery and Directory Service.

We describe the first system that provides query based associative access to the contents of distributed information servers. Queries describe desired object attributes, and are automatically forwarded to servers that contain relevant information. In typical distributed information systems there are so many objects that underconstrained queries can produce large result sets and extraordinary processing costs. To deal with this scaling problem we use content labels to permit users to learn about available resources and to quickly formulate queries with adequate discriminatory power. We present experimental data that show that certain content label attributes can be automatically chosen. We have implemented associative access to a distributed set of information servers in the content routing system. A content routing system is organized as a network of servers called content routers that present a single query based image of a distributed information system. Experiments motivated by our video access service show that substantial performance benefits result when content routers are removed from the client-server path once an object of interest is found.

Over the past several years, a number of information discovery and access tools have been introduced in the Internet, including Archie, Gopher, Netfind, and WAIS. These tools have become quite popular, and are helping to redefine how people think about wide area network applications. Yet, they are not well suited to supporting the future information infrastructure, which will be characterized by enormous data volume, rapid growth in the user base, and burgeoning data diversity. In this paper we indicate trends in these three dimensions, and survey problems these trends will create for current approaches. We then suggest several promising directions of future resource discovery research, along with some initial results from projects carried out by members of the Internet Research Task Force Research Group on Resource Discovery and Directory Service.

An increasing number of clients use the web for applications such asweb browsing, E-commerce, and web portals. An important characteristic of data access in this environment is that clients often have different requirements for the latency and recency of objects. However, there has been little work that aims to accomodate varying client needs. My Ph.D. research introduces client profiles that allow clients to express their requirements with respect to the latency and recency of objects. Profiles are used by a cache or web portal to determine when to download a fresh object and when to use a cached copy. Profiles can be tuned to reflect the requirements of a client or application. The profile-based framework is straightforward to implement for both clients and caches, and can scale to a large number of clients and applications. Preliminary results show that profiles can reduce latency and bandwidth consumption compared to existing techniques, while still providing recent data in many cases.

Research on replication techniques to reduce traffic and minimize the latency of information retrieval in a distributed system has concentrated on client-based caching, whereby recently/frequently accessed information is cached at a client (or at a proxy thereof) in anticipation of future accesses. We believe that such myopic solutions|focussing exclusively on a particular client or set of clients|are likely to have a limited impact. Instead, we offer a solution that allows the replication of information to be done on a global supply/demand basis. We propose a hierarchical demand-based replication strategy that optimally disseminates information from its producer to servers that are closer to its consumers. The level of dissemination depends on the relative popularity of documents, and on the expected reduction in traffic that results from such dissemination. We used extensive HTTP logs to validate an analytical model of server popularity and le access profiles. Using that model we show that by disseminating the most popular documents on servers closer to clients, network traffic could be reduced considerably, while servers are load-balanced. We argue that this process could be generalized toprovide for an automated server-based information dissemination protocol that will be more effective in reducing both network bandwidth and document retrieval times than clientbased caching protocols.