DMCA
Tiled-MapReduce: Optimizing Resource Usages of Data-parallel Applications on Multicore with Tiling
Cached
Download Links
| Citations: | 23 - 1 self |
BibTeX
@MISC{Chen_tiled-mapreduce:optimizing,
author = {Rong Chen and Haibo Chen and Binyu Zang},
title = {Tiled-MapReduce: Optimizing Resource Usages of Data-parallel Applications on Multicore with Tiling},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
The prevalence of chip multiprocessor opens opportunities of running data-parallel applications originally in clusters on a single machine with many cores. MapReduce, a simple and elegant programming model to program large scale clusters, has recently been shown to be a promising alternative to harness the multicore platform. The differences such as memory hierarchy and communication patterns between clusters and multicore platforms raise new challenges to design and implement an efficient MapReduce system on multicore. This paper argues that it is more efficient for Map-Reduce to iteratively process small chunks of data in turn than processing a large chunk of data at one time on shared memory multicore platforms. Based on the argument, we extend the general MapReduce programming model with “tiling strategy”, called Tiled-MapReduce (TMR). TMR partitions a large MapReduce job into a number of small sub-jobs and iteratively processes one subjob at a time with efficient use of resources; TMR finally merges the results of all sub-jobs for output. Based on Tiled-MapReduce, we design and implement several optimizing techniques targeting multicore, including the reuse of input and intermediate data structure among sub-jobs, a NUCA/NUMA-aware scheduler, and pipelining a sub-job’s reduce phase with the successive sub-job’s map phase, to optimize the memory, cache and CPU resources accordingly. We have implemented a prototype of Tiled-MapReduce based on Phoenix, an already highly optimized MapReduce runtime for shared memory multiprocessors. The prototype, namely Ostrich, runs on an Intel machine with 16 cores. Experiments on four different types of benchmarks show that Ostrich saves up to 85 % memory, causes less cache misses and makes more efficient uses of CPU cores, resulting in a speedup ranging from 1.2X to 3.3X.
Keyphrases
data-parallel application optimizing resource usage multicore platform general mapreduce nuca numa-aware scheduler cpu resource chip multiprocessor cpu core large chunk program large scale cluster communication pattern intel machine large mapreduce job small chunk memory hierarchy efficient us intermediate data structure small sub-jobs shared memory multicore platform promising alternative elegant programming model shared memory multiprocessor new challenge different type successive sub-job map phase efficient mapreduce system single machine cache miss many core mapreduce runtime efficient use speedup ranging
