DMCA
The Evolution of the Web and Implications for an Incremental Crawler (1999)
Cached
Download Links
- [www-db.stanford.edu]
- [www.vldb.org]
- [www.acm.org]
- [www-db.stanford.edu]
- [rose.cs.ucla.edu]
- [oak.cs.ucla.edu]
- [cui.unige.ch]
- [ilpubs.stanford.edu:8090]
- DBLP
Other Repositories/Bibliography
| Citations: | 281 - 18 self |
BibTeX
@INPROCEEDINGS{Cho99theevolution,
author = {Junghoo Cho and Hector Garcia-molina},
title = {The Evolution of the Web and Implications for an Incremental Crawler},
booktitle = {},
year = {1999},
pages = {200--209}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In this paper we study how to build an effective incremental crawler. The crawler selectively and incrementally updates its index and/or local collection of web pages, instead of periodically refreshing the collection in batch mode. The incremental crawler can improve the "freshness" of the collection significantly and bring in new pages in a more timely manner. We first present results from an experiment conducted on more than half million web pages over 4 months, to estimate how web pages evolve over time. Based on these experimental results, we compare various design choices for an incremental crawler and discuss their trade-offs. We propose an architecture for the incremental crawler, which combines the best design choices.
