Design and Implementation of a High-Performance Distributed Web Crawler (2002)
| Venue: | In Proc. of the Int. Conf. on Data Engineering |
| Citations: | 61 - 10 self |
BibTeX
@INPROCEEDINGS{Shkapenyuk02designand,
author = {Vladislav Shkapenyuk and Torsten Suel},
title = {Design and Implementation of a High-Performance Distributed Web Crawler},
booktitle = {In Proc. of the Int. Conf. on Data Engineering},
year = {2002},
pages = {357--368}
}
Years of Citing Articles
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
Broad web search engines as well as many more specialized search tools rely on web crawlers to acquire large collections of pages for indexing and analysis. Such a web crawler may interact with millions of hosts over a period of weeks or months, and thus issues of robustness, flexibility, and manageability are of major importance. In addition, I/O performance, network resources, and OS limits must be taken into account in order to achieve high performance at a reasonable cost.
Citations
| 2569 | The Anatomy of a Large-Scale Hypertextual Web Search Engine - Brin, Page - 1998 |
| 1926 | Modern Information Retrieval - Baeza-Yates, Ribeiro-Neto - 1999 |
| 139 | Mining the Web’s Link Structure - Chakrabarti, Dom, et al. - 1999 |
| 108 | Searching the Web - Arasu, Cho, et al. - 2000 |
| 103 | The Connectivity Server: Fast access to linkage information on the Web - Bharat, Broder, et al. - 1998 |
| 60 | Approximating aggregate queries about web pages via random walks - Bar-Yossef, Berg, et al. - 2000 |
| 45 | Crawling Towards Eternity: Building an Archive of the World Wide Web - Burner - 1997 |
| 27 | Distributed hypertext resource discovery through examples - Chakrabarti, Berg, et al. - 1999 |
