It is increasingly difficult to make effective use of Internet information, given the rapid growth in data volume, user base, and data diversity. In this paper we introduce Harvest, a system that provides a scalable, customizable architecture for gathering, indexing, caching, replicating, and accessing Internet information. 1.

With huge amounts of information connected to the global information network (Internet), efficient and effective discovery of resource and knowledge from the "global information base" has become an imminent research issue, especially with the advent of the Information SuperHighway. In this article, a multiple layered database (MLDB) approach is proposed to handle the resource and knowledge discovery in global information base. A preliminary experiment using on-line technical reports, a representative subset of the Internet, shows the advantages of such an approach. A multiple layered database is a database formed by generalization and transformation of the information, layer-by-layer, starting from the original information base (treated as layer-0, the primitive layer). Information retrieval, data mining, and data analysis techniques can be used to extract and transform information from a lower layer database to a higher one. Layer-1 and higher layers of an MLDB can be modeled by an e...

With huge amounts of information connected to the global information network (Internet), efficient and effective discovery of resource and knowledge from the "global information base" has become an imminent research issue, especially with the advent of the Information SuperHighway. In

With huge amounts of information connected to the global information network (Internet), efficient and effective discovery of resources from the "global information base" has become an imminent research issue, especially with the advent of the Information Highway. This article proposes the use of novel Artificial Intelligence and Database techniques (Assumption Grammars, Concept Hierarchies, Multi-Layered Databases, Intelligent Agents) for intelligently searching information pertaining to a specific industry on the web.

Internet traffic doubles every three months, with web traffic accounting for more than 75% of the total traffic between major ISP providers. Web caching at proxies can reduce this traffic as well as web user latencies. One way to increase the hit rate of caches is to have caches cooperate using the Internet Cache Protocol or using hash functions. This paper provides an experimental and theoretical analysis of two well known hash based caching schemes. Our paper also proposes a new allocation scheme which improves the performance of one of the analyzed techniques. It reduces the standard deviation of the URLs distribution on caches by a factor greater than two.

We present a method for retrieving concepts from web search queries and from candidate documents on the web, to help determine which of these documents are semantically (rather than simply key-word wise) related to the query. Our method combines hypothetical reasoning, which we use both for natural language analysis and for concept extraction, and domain-oriented taxonomies of concepts to guide the system's reasoning. 1 Introduction Realistic natural language analysis, whether for web or traditional applications, cannot make abstraction of semantics and pragmatics, any more than programming languages can fully make abstraction of their run-time environments. Computer-based discourse understanding (as its human counterpart) is basically a form of model-building. It involves constant constraint-solving to keep, at a given time, only a manageable subset of (intended) models. The task is harder, but similar to that of compilers for programming languages. The experience of being drowned wi...