Efficient Crawling Through URL Ordering (1998) [200 citations — 9 self]
http://www-db.stanford.edu/~cho/papers/cho-order.p
http://www-db.stanford.edu/pub/papers/efficient-cr
DBLP
CACHED:
|
Popular Tags
No tags have been applied to this document.
BibTeX | Add To MetaCart
author = {Junghoo Cho and Hector Garcia-molina and Lawrence Page},
title = {Efficient Crawling Through URL Ordering},
booktitle = {Computer Networks and ISDN Systems},
year = {1998},
pages = {161--172}
}
Years of Citing Articles
Bookmarks
OpenURL
Abstract:
In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without. 1. Introduction A crawler is a program that retrieves Web pages, commonly for use by a search engine [Pinkerton 1994] or a Web cache. Roughly, a crawler starts off with the URL for an initial page P 0 . It retrieves P 0 , extracts any URLs in it, and adds them to a queue of URLs to be scanned. Then the crawler gets URLs from the queue (in some order), and repeats the process. Every page that is scanned is given...
Citations
No citations identified.
