Topical web crawlers: Evaluating adaptive algorithms

Topical web crawlers: Evaluating adaptive algorithms (2004)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Filippo Menczer , Gautam Pant , Padmini Srinivasan

Venue:	ACM Transactions on Internet Technology
Citations:	35 - 11 self

Versions

Version History

Metadata Version 1

Datum	Value	Source
TITLE	Topical web crawlers: Evaluating adaptive algorithms	INFERENCE
AUTHOR NAME	Filippo Menczer	SVM HeaderParse 0.2
AUTHOR AFFIL	The University of Iowa	SVM HeaderParse 0.2
AUTHOR NAME	Gautam Pant	SVM HeaderParse 0.2
AUTHOR AFFIL	The University of Iowa	SVM HeaderParse 0.2
AUTHOR NAME	Padmini Srinivasan	SVM HeaderParse 0.2
AUTHOR AFFIL	The University of Iowa	SVM HeaderParse 0.2
ABSTRACT	Topical crawlers are increasingly seen as a way to address the scalability limitations of universal search engines, by distributing the crawling process across users, queries, or even client computers. The context available to such crawlers can guide the navigation of links with the goal of efficiently locating highly relevant target pages. We developed a framework to fairly evaluate topical crawling algorithms under a number of performance metrics. Such a framework is employed here to evaluate different algorithms that have proven highly competitive among those proposed in the literature and in our own previous research. In particular we focus on the tradeoff between exploration and exploitation of the cues available to a crawler, and on adaptive crawlers that use machine learning techniques to guide their search. We find that the best performance is achieved by a novel combination of explorative and exploitative bias, and introduce an evolutionary crawler that surpasses the performance of the best non-adaptive crawler after sufficiently long crawls. We also analyze the computational complexity of the various crawlers and discuss how performance and complexity scale with available resources. Evolutionary crawlers achieve high efficiency and scalability by distributing the work across concurrent agents, resulting in the best performance/cost ratio.	SVM HeaderParse 0.2
YEAR	2004	INFERENCE
VENUE	ACM Transactions on Internet Technology	INFERENCE
VENUE TYPE	JOURNAL	INFERENCE
PAGES	http://dollar.biz.ui	INFERENCE
VOLUME	4	INFERENCE
CITATIONS	45 found	ParsCit 1.0