DMCA
Quincy: Fair Scheduling for Distributed Computing Clusters
| Citations: | 72 - 1 self |
BibTeX
@MISC{Isard_quincy:fair,
author = {Michael Isard and Vijayan Prabhakaran and Jon Currey and Udi Wieder and Kunal Talwar and Andrew Goldberg},
title = {Quincy: Fair Scheduling for Distributed Computing Clusters},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This paper addresses the problem of scheduling concurrent jobs on clusters where application data is stored on the computing nodes. This setting, in which scheduling computations close to their data is crucial for performance, is increasingly common and arises in systems such as MapReduce, Hadoop, and Dryad as well as many grid-computing environments. We argue that data intensive computation benefits from a fine-grain resource sharing model that differs from the coarser semi-static resource allocations implemented by most existing cluster computing architectures. The problem of scheduling with locality and fairness constraints has not previously been extensively studied under this model of resourcesharing. We introduce a powerful and flexible new framework for scheduling concurrent distributed jobs with fine-grain resource sharing. The scheduling problem is mapped to a graph datastructure, where edge weights and capacities encode the competing demands of data locality, fairness, and starvation-freedom, and a standard solver computes the optimal online schedule according to a global cost model. We evaluate our implementation of this framework, which we call Quincy, on a cluster of a few hundred computers using a varied workload of data- and CPU-intensive jobs. We evaluate Quincy against an existing queue-based algorithm and implement several policies for each scheduler, with and without fairness constraints. Quincy gets better fairness when fairness is requested, while substantially improving data locality. The volume of data transferred across the cluster is reduced by up to a factor of 3.9 in our experiments, leading to a throughput increase of up to 40%.
Keyphrases
distributed computing cluster fairness constraint data locality global cost model flexible new framework application data edge weight cpu-intensive job standard solver graph datastructure throughput increase many grid-computing environment fine-grain resource optimal online schedule concurrent job fine-grain resource sharing varied workload scheduling problem data intensive computation benefit queue-based algorithm implement several policy coarser semi-static resource allocation
