Topical crawlers are increasingly seen as a way to address the scalability limitations of universal search engines, by distributing the crawling process across users, queries, or even client computers. The context available to such crawlers can guide the navigation of links with the goal of efficiently locating highly relevant target pages. We developed a framework to fairly evaluate topical crawling algorithms under a number of performance metrics. Such a framework is employed here to evaluate different algorithms that have proven highly competitive among those proposed in the literature and in our own previous research. In particular we focus on the tradeoff between exploration and exploitation of the cues available to a crawler, and on adaptive crawlers that use machine learning techniques to guide their search. We find that the best performance is achieved by a novel combination of explorative and exploitative bias, and introduce an evolutionary crawler that surpasses the performance of the best non-adaptive crawler after sufficiently long crawls. We also analyze the computational complexity of the various crawlers and discuss how performance and complexity scale with available resources. Evolutionary crawlers achieve high efficiency and scalability by distributing the work across concurrent agents, resulting in the best performance/cost ratio.

Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. As the Web mining field matures, the disparate crawling strategies proposed in the literature will have to be evaluated and compared on common tasks through welldefined performance measures. This paper presents a general framework to evaluate topical crawlers. We identify a class of tasks that model crawling applications of di#erent nature and di#culty. We then introduce a set of performance measures for fair comparative evaluations of crawlers along several dimensions including generalized notions of precision, recall, and e#ciency that are appropriate and practical for the Web. The framework relies on independent relevance judgements compiled by human editors and available from public directories. Two sources of evidence are proposed to assess crawled pages, capturing di#erent relevance criteria. Finally we introduce a set of topic characterizations to analyze the variability in crawling e#ectiveness across topics. The proposed evaluation framework synthesizes a number of methodologies in the topical crawlers literature and many lessons learned from several studies conducted by our group. The general framework is described in detail and then illustrated in practice by a case study that evaluates four public crawling algorithms. We found that the proposed framework is e#ective at evaluating, comparing, di#erentiating and interpreting the performance of the four crawlers. For example, we found the IS crawler to be most sensitive to the popularity of topics.

The amount of information on Internet is increasing very fast and, as a result, search becomes more and more a harder task. A common solution is to use authority-based search engines. However, for a community of people with similar interests, quality of results can be improved exploiting also implicit knowledge. We propose an agentbased recommendation system for supporting communities of people in searching the web by means of a popular search engine. Agents use data mining techniques in order to learn and discover users behaviors, and they interact to share knowledge about the users. We also present a set of experimental results showing in terms of precision and recall how interaction increases the performance of the system. 1.

Topic driven crawlers are increasingly seen as a way to address the scalability limitations of universal search engines, by distributing the crawling process across users, queries, or even client computers.

Abstract. This paper presents a consensus-based approach utilized within a multi-agent system which assists users in retrieving information from the Internet. In this system consensus methods are applied for reconciling inconsistencies among independent answers generated by agents (using different search engines) for a given query. Proposed agent system has been implemented and initial experimental results are presented. 1

podpis promotora podpis autoraPOLITECHNIKA WARSZAWSKA

journal homepage: www.elsevier.com/locate/dss

The paper describes the process involving distributed data access from a mobile device, employing mobile agents. To answer any query in the distributed environment the search is conducted to answer the query only in the databases, which are known to the systems. The transfer of database to the system, where the query is originated will involve high communication cost, response time and increase the network traffic. In order to reduce the values of these parameters and incidentally the network traffic, mobile agents are used to fetch the result from various sites.. A mobile agent is a software program that migrates from one node to another where the data is located instead of transmitting data across the network. This paper presents a study of deployment of Multi-agent system for retrieval of data and also the management of distributed resources. Experiments conducted reveal the performance of Mobile agents, using Parallel Query Retrieval (PQR) approach.