The Link Database: Fast Access to Graphs of the Web

The Link Database: Fast Access to Graphs of the Web

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Keith H. Randall , Raymie Stata Rajiv , Rajiv G. Wickremesinghe , Janet L. Wiener

Citations:

33 - 2 self

Versions

Version History

Metadata Version 3

User correction supplied by mph

Datum	Value	Source
TITLE	The Link Database: Fast Access to Graphs of the Web	user correction
AUTHOR NAME	Keith H. Randall	user correction
AUTHOR AFFIL	* Compaq Systems Research Center	user correction
AUTHOR ADDR	130 Lytton Avenue, Palo Alto, CA 94301, USA	user correction
AUTHOR NAME	Raymie Stata Rajiv	user correction
AUTHOR NAME	Rajiv G. Wickremesinghe	user correction
AUTHOR NAME	Janet L. Wiener	user correction
AUTHOR AFFIL	* * Dept. of Computer Science, Duke University, Durham	user correction
AUTHOR ADDR	NC 27708, USA	user correction
ABSTRACT	... graph where URLs are nodes and hyperlinks are directed edges. The Link Database provides fast access to the hyperlinks. To support a wide range of graph algorithms, we find it important to fit the Link Database into memory. In the first version of the Link Database, we achieved this fit by using machines with lots of memory (8GB), and storing each hyperlink in 32 bits. However, this approach was limited to roughly 100 million Web pages. This paper presents techniques to compress the links to accommodate larger graphs. Our techniques combine well-known compression methods with methods that depend on the properties of the web graph. The first compression technique takes advantage of the fact that most hyperlinks on most Web pages point to other pages on the same host as the page itself. The second technique takes advantage of the fact that many pages on the same host share hyperlinks, that is, they tend to point to a common set of pages. Together, these techniques reduce space requirements to under 6 bits per link. While (de)compression adds latency to the hyperlink access time, we can still compute the strongly connected components of a 6 billion-edge graph in under 20 minutes and run applications such as Kleinberg's HITS in real time. This paper describes our techniques for compressing the Link Database, and provides performance numbers for compression ratios and decompression speed.	user correction
CITATIONS	19 found	ParsCit 1.0