Mercator: A scalable, extensible web crawler (1999)
Cached
Download Links
| Venue: | Word Wide Web |
| Citations: | 128 - 5 self |
BibTeX
@ARTICLE{Heydon99mercator:a,
author = {Allan Heydon and Marc Najork},
title = {Mercator: A scalable, extensible web crawler},
journal = {Word Wide Web},
year = {1999}
}
Years of Citing Articles
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
This paper describes Mercator, a scalable, extensible web crawler written entirely in Java. Scalable web crawlers are an important component of many web services, but their design is not well-documented in the literature. We enumerate the major components of any scalable web crawler, comment on alternatives and tradeoffs in their design, and describe the particular components used in Mercator. We also describe Mercator’s support for extensibility and customizability. Finally, we comment on Mercator’s performance, which we have found to be comparable to that of other crawlers for which performance numbers have been published. 1
