DMCA
Workload-Aware Web Crawling and Server Workload Detection (2004)
| Venue: | In Network Research Workshop, 18th Asian Pacific Advanced Network Meeting (APAN 2004 |
| Citations: | 2 - 0 self |
BibTeX
@INPROCEEDINGS{Ye04workload-awareweb,
author = {Shaozhi Ye and Guohan Lu and Xing Li},
title = {Workload-Aware Web Crawling and Server Workload Detection},
booktitle = {In Network Research Workshop, 18th Asian Pacific Advanced Network Meeting (APAN 2004},
year = {2004}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
With the development of search engines, more and more web crawlers are used to gather web pages. The rising crawling tra#c has brought the concern that crawlers may impact web sites. On the other hand, more e#cient crawling strategy is required for the coverage and freshness of search engine index. In this paper, crawlers of several major search engines are analyzed using one six-months access log of a busy web site. Surprisingly, we find that none of these crawlers pays attention to the workload of web site, which may hurt both server performance and crawling e#- ciency. Based on this observation, a server workload-aware crawling strategy is proposed. By measuring the web service time with a hybrid back-to-back packets pair, server workload is detected on the client side, thus crawler can adapt its crawling speed to web server. The experiment results show the power of our workload detection approach. This paper concludes with a discussion of future work on server workload detection and its applications.
Keyphrases
server workload detection workload-aware web crawling server performance future work several major search engine client side workload detection approach six-months access log search engine crawling tra server workload web crawler web site web service time search engine index server workload-aware crawling strategy experiment result hybrid back-to-back packet pair web page busy web site
