Citation Context

...re. Tenants are primarily interested in predictable performance and costs for their applications [2, 28]; for instance, tenants want to satisfy constraints regarding their application completion time =-=[9,14,20,35,39]-=-. With today’s setup, tenants bear the burden of translating these high-level goals into the corresponding resource requirements. Motivated by this disconnect, we argue for refactoring the tenant-prov...

Citation Context

...s over 180 configuration parameters that have varying impact on job performance. Examples include the number of reducers to be launched, the size of memory buffer to use while sorting map output, the number of concurrent connections a reducer should use when fetching its input from mappers etc. Furthermore, the optimal configuration of Hadoop parameters is tightly coupled with hardware configuration of selected resource sets as well as the type of jobs submitted. There are techniques that each addresses resource allocation and parameter configuration in a Hadoop MapReduce framework separately [10, 12, 23]. However, in practice the effectiveness and cost efficiency of those techniques are often inter-dependent. For instance, a Hadoop parameter such as mapred · tasktracker ·map · tasks ·maximum (the maximum number of parallel mappers) is optimal at different values when the number of virtual CPU cores of a Hadoop node is different [12]. It is important to consider a holistic approach that integrates various aspects of resource provisioning and auto-configuration for Hadoop jobs. We propose and develop AROMA, a system that automates the allocation of heterogeneous Cloud resources and auto-configu...

Citation Context

...lysis, etc. These MapReduce applications are deadlinedriven and typically require completion time guarantees and achieving service level objectives (SLOs). While there have been some research efforts =-=[17, 19, 14]-=- towards developing performance models for MapReduce jobs, these techniques do not apply to complex queries consisting of MapReduce DAGs. To address this limitation, our paper studies the popular Pig ...

Citation Context

...s. Related to methodology, cloud research in general relies either on analytical models [8, 16, 64, 73, 74, 76] or on running actual experiments and building empirical traffic profiles and signatures =-=[28, 52, 65, 67, 68]-=-. Our system uses an analytical model to infer performance from measurements taken from the actual system. Zheng et al. [75] argue that running actual experiments is cheaper than building accurate mod...

Citation Context

...ed to derive the estimated JCT. We build the database of job profiles, averaged across 3 runs of each job instance. Note that similar procedure for profiling MapReduce jobs has been explored in [12], =-=[33]-=-. There are also complex techniques [18], [33], where profiling is done for finegrained stages of MapReduce framework. Also, the technique can be extended for online profiling [12], [33]. However, we ...

Citation Context

...lied orthogonally in systems like Amoeba [6]. We are also the first to incorporate such support directly into Hadoop YARN. Finally, Mapreduce deadline scheduling has been studied in infinite clusters =-=[18, 38, 49, 51]-=- but not in resource-constrained clusters. 2 Eviction Policies for a Dual Priority Setting In the first part of the paper, we focus purely on the dual priority setting. This section presents eviction ...

Citation Context

...e business intelligence that require completion time guarantees. While there were a few research efforts to design different models and approaches for predicting performance of MapReduce applications =-=[6, 5, 8, 9, 10]-=-, this question still remains a challenging research problem. Some of the past modeling efforts aim to predict the job completion time by analyzing the execution times’ distribution of map and reduce ...

Citation Context

...nd regressions of an observed trend to generate more accurate predictions. There has been a lot of work on resource provisioning, right-sizing, and performance prediction of individual MapReduce jobs =-=[15]-=-, but techniques for dynamic scheduling are orthogonal to longer-term capacity planning. Menascé and Ngo [10] explore some of the implications of capacity planning from the point of view of the cloud ...

Citation Context

...eans are in several cases already available, since estimating job size is not only interesting for the scheduler; relevant examples are approaches that deal with predicting the size of MapReduce jobs =-=[8]-=-–[10] and of database queries [11]. The estimation error can be always evaluated a posteriori, and this evaluation can be used to decide if the size-based scheduling works better than policies blind t...

Citation Context

...to approximate the overall completion time of a MapReduce job, 1 Similar ideas were explored in the classic papers on scheduling, e.g., to characterize makespan bounds in [3]. For a formal proof, see =-=[11]-=-. we need to estimate the average and maximum task durations for different execution phases of the job, i.e., map, shuffle/sort, and reduce phases. These metrics and timing of different phases can be ...