Results 1  10
of
30
GPS: A Graph Processing System
"... GPS (for Graph Processing System) is a complete opensource system we developed for scalable, faulttolerant, and easytoprogram execution of algorithms on extremely large graphs. GPS is similar to Google’s proprietary Pregel system [MAB+ 11], with some useful additional functionality described in ..."
Abstract

Cited by 68 (3 self)
 Add to MetaCart
(Show Context)
GPS (for Graph Processing System) is a complete opensource system we developed for scalable, faulttolerant, and easytoprogram execution of algorithms on extremely large graphs. GPS is similar to Google’s proprietary Pregel system [MAB+ 11], with some useful additional functionality described in the paper. In distributed graph processing systems like GPS and Pregel, graph partitioning is the problem of deciding which vertices of the graph are assigned to which compute nodes. In addition to presenting the GPS system itself, we describe how we have used GPS to study the effects of different graph partitioning schemes. We present our experiments on the performance of GPS under different static partitioning schemes—assigning vertices to workers “intelligently ” before the computation starts—and with GPS’s dynamic repartitioning feature, which reassigns vertices to different compute nodes during the computation by observing their message sending patterns.
GraphX: Graph Processing in a Distributed Dataflow Framework
 USENIX ASSOCIATION 11TH USENIX SYMPOSIUM ON OPERATING SYSTEMS DESIGN AND IMPLEMENTATION (OSDI ’14)
, 2014
"... In pursuit of graph processing performance, the systems community has largely abandoned generalpurpose distributed dataflow frameworks in favor of specialized graph processing systems that provide tailored programming abstractions and accelerate the execution of iterative graph algorithms. In thi ..."
Abstract

Cited by 23 (1 self)
 Add to MetaCart
In pursuit of graph processing performance, the systems community has largely abandoned generalpurpose distributed dataflow frameworks in favor of specialized graph processing systems that provide tailored programming abstractions and accelerate the execution of iterative graph algorithms. In this paper we argue that many of the advantages of specialized graph processing systems can be recovered in a modern generalpurpose distributed dataflow system. We introduce GraphX, an embedded graph processing framework built on top of Apache Spark, a widely used distributed dataflow system. GraphX presents a familiar composable graph abstraction that is sufficient to express existing graph APIs, yet can be implemented using only a few basic dataflow operators (e.g., join, map, groupby). To achieve performance parity with specialized graph systems, GraphX recasts graphspecific optimizations as distributed join optimizations and materialized view maintenance. By leveraging advances in distributed dataflow frameworks, GraphX brings lowcost fault tolerance to graph processing. We evaluate GraphX on real workloads and demonstrate that GraphX achieves an order of magnitude performance gain over the base dataflow framework and matches the performance of specialized graph processing systems while enabling a wider range of computation.
QueryPIE: Backward reasoning for OWL Horst over very large knowledge bases
 In Proc. 10th Int. Semantic Web Conf. (ISWC’11
, 2011
"... Abstract. Both materialization and backwardchaining as different modes of performing inference have complementary advantages and disadvantages. Materialization enables very efficient responses at query time, but at the cost of an expensive up front closure computation, which needs to be redone ever ..."
Abstract

Cited by 18 (1 self)
 Add to MetaCart
(Show Context)
Abstract. Both materialization and backwardchaining as different modes of performing inference have complementary advantages and disadvantages. Materialization enables very efficient responses at query time, but at the cost of an expensive up front closure computation, which needs to be redone every time the knowledge base changes. Backwardchaining does not need such an expensive and changesensitive precomputation, and is therefore suitable for more frequently changing knowledge bases, but has to perform more computation at query time. Materialization has been studied extensively in the recent semantic web literature, and is now available in industrialstrength systems. In this work, we focus instead on backwardchaining, and we present an hybrid algorithm to perform efficient backwardchaining reasoning on very large datasets expressed in the OWL Horst (pD∗) fragment. As a proof of concept, we have implemented a prototype called QueryPIE (Query Parallel Inference Engine), and we have tested its performance on different datasets of up to 1 billion triples. Our parallel implementation greatly reduces the reasoning complexity of a naive backwardchaining approach and returns results for single querypatterns in the order of milliseconds when running on a modest 8 machine cluster. To the best of our knowledge, QueryPIE is the first reported backwardchaining reasoner for OWL Horst that efficiently scales to a billion triples. 1
Fast Iterative Graph Computation with Block Updates
"... Scaling iterative graph processing applications to large graphs is an important problem. Performance is critical, as data scientists need to execute graph programs many times with varying parameters. The need for a highlevel, highperformance programming model has inspired much research on graph pr ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
Scaling iterative graph processing applications to large graphs is an important problem. Performance is critical, as data scientists need to execute graph programs many times with varying parameters. The need for a highlevel, highperformance programming model has inspired much research on graph programming frameworks. In this paper, we show that the important class of computationally light graph applications – applications that perform little computation per vertex – has severe scalability problems across multiple cores as these applications hit an early “memory wall ” that limits their speedup. We propose a novel blockoriented computation model, in which computation is iterated locally over blocks of highly connected nodes, significantly improving the amount of computation per cache miss. Following this model, we describe the design and implementation of a blockaware graph processing runtime that keeps the familiar vertexcentric programming paradigm while reaping the benefits of blockoriented execution. Our experiments show that blockoriented execution significantly improves the performance of our framework for several graph applications. 1.
Distributed kCore Decomposition
"... Abstract—Several novel metrics have been proposed in recent literature in order to study the relative importance of nodes in complex networks. Among those, kcoreness has found a number of applications in areas as diverse as sociology, proteinomics, graph visualization, and distributed system analys ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
(Show Context)
Abstract—Several novel metrics have been proposed in recent literature in order to study the relative importance of nodes in complex networks. Among those, kcoreness has found a number of applications in areas as diverse as sociology, proteinomics, graph visualization, and distributed system analysis and design. This paper proposes new distributed algorithms for the computation of the kcoreness of a network, a process also known as k–core decomposition. This technique (i) allows the decomposition, over a set of connected machines, of very large graphs, when size does not allow storing and processing them on a single host, and (ii) enables the runtime computation of kcores in “live ” distributed systems. Lower bounds on the algorithms complexity are given, and an exhaustive experimental analysis on realworld datasets is provided. a sample graph, consider Figure 1. Note that by definition cores are nested, meaning that nodes belonging to the 3core belong to the2core and1core, as well. Larger values of “coreness”, though, clearly correspond to nodes with a more central position in the network structure. Index Terms—kcore decomposition, Graph analysis, Bulk Synchronous Parallel 1
GraphX: Unifying DataParallel and GraphParallel Analytics
, 2014
"... From social networks to language modeling, the growing scale and importance of graph data has driven the development of numerous new graphparallel systems (e.g., Pregel, GraphLab). By restricting the computation that can be expressed and introducing new techniques to partition and distribute the ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
(Show Context)
From social networks to language modeling, the growing scale and importance of graph data has driven the development of numerous new graphparallel systems (e.g., Pregel, GraphLab). By restricting the computation that can be expressed and introducing new techniques to partition and distribute the graph, these systems can efficiently execute iterative graph algorithms orders of magnitude faster than more general dataparallel systems. However, the same restrictions that enable the performance gains also make it difficult to express many of the important stages in a typical graphanalytics pipeline: constructing the graph, modifying its structure, or expressing computation that spans multiple graphs. As a consequence, existing graph analytics pipelines compose graphparallel and dataparallel systems using external storage systems, leading to extensive data movement and complicated programming model. To address these challenges we introduce GraphX, a distributed graph computation framework that unifies graphparallel and dataparallel computation. GraphX provides a small, core set of graphparallel operators expressive enough to implement the Pregel and PowerGraph abstractions, yet simple enough to be cast in relational algebra. GraphX uses a collection of query optimization techniques such as automatic join rewrites to efficiently implement these graphparallel operators. We evaluate GraphX on realworld graphs and workloads and demonstrate that GraphX achieves comparable performance as specialized graph computation systems, while outperforming them in endtoend graph pipelines. Moreover, GraphX achieves a balance between expressiveness, performance, and ease of use.
Parallelizing machine learning  functionally: A framework and abstractions for parallel graph processing
 IN 2ND ANNUAL SCALA WORKSHOP
, 2011
"... Implementing machine learning algorithms for large data, such as the Web graph and social networks, is challenging. Even though much research has focused on making sequential algorithms more scalable, their running times continue to be prohibitively long. Meanwhile, parallelization remains a formida ..."
Abstract

Cited by 3 (1 self)
 Add to MetaCart
Implementing machine learning algorithms for large data, such as the Web graph and social networks, is challenging. Even though much research has focused on making sequential algorithms more scalable, their running times continue to be prohibitively long. Meanwhile, parallelization remains a formidable challenge for this class of problems, despite frameworks like MapReduce which hide much of the associated complexity. We present a framework for implementing parallel and distributed machine learning algorithms on large graphs, flexibly, through the use of functional programming abstractions. Our aim is a system that allows researchers and practitioners to quickly and easily implement (and experiment with) their algorithms in a parallel or distributed setting. We introduce functional combinators for the flexible composition of parallel, aggregation, and sequential steps. To the best of our knowledge, our system is the first to avoid inversion of control in a (bulk) synchronous parallel model.
Triplerush: A fast and scalable triple store
 In 9th International Workshop on Scalable Semantic Web Knowledge Base Systems (SSWS
, 2013
"... ..."
(Show Context)
Adaptive partitioning for largescale dynamic graphs
 In proc. ICDCS (2014
"... Abstract—In the last years, largescale graph processing has gained increasing attention, with most recent systems placing particular emphasis on latency. One possible technique to improve runtime performance in a distributed graph processing system is to reduce network communication. The most notab ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
(Show Context)
Abstract—In the last years, largescale graph processing has gained increasing attention, with most recent systems placing particular emphasis on latency. One possible technique to improve runtime performance in a distributed graph processing system is to reduce network communication. The most notable way to achieve this goal is to partition the graph by minimizing the number of edges that connect vertices assigned to different machines, while keeping the load balanced. However, realworld graphs are highly dynamic, with vertices and edges being constantly added and removed. Carefully updating the partitioning of the graph to reflect these changes is necessary to avoid the introduction of an extensive number of cut edges, which would gradually worsen computation performance. In this paper we show that performance degradation in dynamic graph processing systems can be avoided by adapting continuously the graph partitions as the graph changes. We present a novel highly scalable adaptive partitioning strategy, and show a number of refinements that make it work under the constraints of a largescale distributed system. The partitioning strategy is based on iterative vertex migrations, relying only on local information. We have implemented the technique in a graph processing system, and we show through three realworld scenarios how adapting graph partitioning reduces execution time by over 50 % when compared to commonly used hashpartitioning. I.
xdgp: A dynamic graph processing system with adaptive partitioning. arXiv
, 2013
"... Many realworld systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
Many realworld systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing edges cut between partitions incurs a significant performance penalty due to the cost of communication. Thus, several systems in the literature have attempted to improve computational performance by enhancing graph partitioning, but they do not support another characteristic of realworld graphs: graphs are inherently dynamic, their topology evolves continuously, and subsequently the optimum partitioning also changes over time. In this work, we present the first system that dynamically repartitions massive graphs to adapt to structural changes. The system optimises graph partitioning to prevent performance degradation without using data replication. The system adopts an iterative vertex migration algorithm that relies on local information only, making complex coordination unnecessary. We show how the improvement in graph partitioning reduces execution time by over 50%, while adapting the partitioning to a large number of changes to the graph in three realworld scenarios.