Mesos: A platform for fine-grained resource sharing in the data center

DMCA

Mesos: A platform for fine-grained resource sharing in the data center (2010)

Cached

Download Links

by Benjamin Hindman , Andy Konwinski , Matei Zaharia , Ali Ghodsi , Anthony D. Joseph , Randy Katz , Scott Shenker , Ion Stoica

Citations:

160 - 23 self

BibTeX

@TECHREPORT{Hindman10mesos:a,
    author = {Benjamin Hindman and Andy Konwinski and Matei Zaharia and Ali Ghodsi and Anthony D. Joseph and Randy Katz and Scott Shenker and Ion Stoica},
    title = {Mesos: A platform for fine-grained resource sharing in the data center },
    institution = {},
    year = {2010}
}

OpenURL

Abstract

We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI 1. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today’s frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our experimental results show that Mesos can achieve near-optimal locality when sharing the cluster among diverse frameworks, can scale up to 50,000 nodes, and is resilient to node failures.

Keyphrases

data center fine-grained resource sharing present mesos node failure distributed two-level scheduling mechanism data locality improves cluster utilization multiple diverse cluster resource offer sophisticated scheduler fine-grained manner diverse framework experimental result near-optimal locality commodity cluster many resource mesos share resource avoids per-framework data replication

DMCA