DMCA
MATRIX: MAny-Task computing execution fabRIc at eXascales. http://datasys.cs.iit.edu/projects/MATRIX/index.html (2013)
| Venue: | SimMatrx(Mem/Task) SimGrid(Mem/Task) GridSim(Mem/Task) Scale (No. of Nodes) Scale (No. of Nodes) Figure 4: Resouce consumption comparions among SimMatrix, SimGrid, and GridSim |
| Citations: | 9 - 9 self |
BibTeX
@INPROCEEDINGS{Wang13matrix:many-task,
author = {Ke Wang and Anupam Rajendran and Ioan Raicu},
title = {MATRIX: MAny-Task computing execution fabRIc at eXascales. http://datasys.cs.iit.edu/projects/MATRIX/index.html},
booktitle = {SimMatrx(Mem/Task) SimGrid(Mem/Task) GridSim(Mem/Task) Scale (No. of Nodes) Scale (No. of Nodes) Figure 4: Resouce consumption comparions among SimMatrix, SimGrid, and GridSim},
year = {2013}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Efficiently scheduling large number of jobs over large-scale distributed systems is critical in achieving high system utilization and throughput. Today’s state-of-the-art job management systems have predominantly Master/Slaves architectures, which have inherent limitations, such as scalability issues at extreme scales (e.g. petascales and beyond) and single point of failure. In designing the nextgeneration distributed job management system, we must address new challenges such as load balancing. This paper presents the design, analysis and implementation of a distributed execution fabric called MATRIX (MAny-Task computing execution fabRIc at eXascale). MATRIX utilizes an adaptive work stealing algorithm for distributed load balancing, and distributed hash tables for managing task metadata. MATRIX supports both high-performance computing (HPC) and many-task computing (MTC) workloads, as well as task dependencies in the execution of complex large-scale workflows. We have evaluated it using synthetic workloads up to 4K-cores on an IBM Blue Gene/P supercomputer, and have shown high efficiency rates (e.g. 85%+) are possible with certain workloads with task granularities as low as 64ms. MATRIX has shown throughput rates as high as 13K tasks/sec at 4K-core scales (one to two orders of magnitude higher than existing centralized systems). We also explore the feasibility of adaptive work stealing up to 1M-node scale through simulations. 1.
Keyphrases
execution fabric edu project matrix index adaptive work job management system 1m-node scale scalability issue throughput rate high-performance computing large number large-scale distributed system task metadata distributed load balancing load balancing certain workload ibm blue gene supercomputer high efficiency rate today state-of-the-art job management system task granularity many-task computing master slave architecture single point complex large-scale workflow 4k-core scale distributed execution fabric new challenge task dependency extreme scale high system utilization synthetic workload inherent limitation hash table centralized system
