PEGASUS: A Peta-Scale Graph Mining System- Implementation and Observations (2009)
Cached
Download Links
| Venue: | IEEE INTERNATIONAL CONFERENCE ON DATA MINING |
| Citations: | 127 - 24 self |
BibTeX
@MISC{Kang09pegasus:a,
author = {U Kang and Charalampos E. Tsourakakis and Christos Faloutsos},
title = {PEGASUS: A Peta-Scale Graph Mining System- Implementation and Observations},
year = {2009}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract—In this paper, we describe PEGASUS, an open source Peta Graph Mining library which performs typical graph mining tasks such as computing the diameter of the graph, computing the radius of each node and finding the connected components. As the size of graphs reaches several Giga-, Tera- or Peta-bytes, the necessity for such a library grows too. To the best of our knowledge, PEGASUS is the first such library, implemented on the top of the HADOOP platform, the open source version of MAPREDUCE. Many graph mining operations (PageRank, spectral clustering, diameter estimation, connected components etc.) are essentially a repeated matrix-vector multiplication. In this paper we describe a very important primitive for PEGASUS, called GIM-V (Generalized Iterated Matrix-Vector multiplication). GIM-V is highly optimized, achieving (a) good scale-up on the number of available machines (b) linear running time on the number of edges, and (c) more than 5 times faster performance over the non-optimized version of GIM-V. Our experiments ran on M45, one of the top 50 supercomputers in the world. We report our findings on several real graphs, including one of the largest publicly available Web Graphs, thanks to Yahoo!, with ≈ 6,7 billion edges. Keywords-PEGASUS; graph mining; hadoop I.
Keyphrases
peta-scale graph mining system implementation several giga library grows first library connected component graph mining spectral clustering generalized iterated matrix-vector multiplication matrix-vector multiplication many graph mining operation available web graph diameter estimation typical graph mining task open source version non-optimized version good scale-up important primitive several real graph hadoop platform available machine
