Citation Context

...optimal specification that can help guide tenant’s decisions about data partitioning on the available storage services for their analytics workload(s). The solver uses a simulated annealing algorithm =-=[29]-=- to systematically search through the solution space and find a desirable tiering plan, given the workload specification, analytics models, and tenants’ goals. 4.2.1 CAST Solver: Modeling The data pla...

Citation Context

...enants’ goals such as achieving high utility or reducing deadline miss rates. 4.1 Estimating Analytics Job Performance The well-defined execution phases of the MapReduce parallel programming paradigm =-=[20, 27]-=- implies that the runtime characteristics of analytics jobs can be predicted with high accuracy. Moreover, extensive recent research has focused on data analytics performance prediction [42, 25, 24, 1...

Citation Context

..., private, and hybrid clouds for not only web applications, such as Netflix, Instagram and Airbnb, but also modern big data analytics using parallel programming paradigms such as Hadoop [2] and Dryad =-=[26]-=-. Cloud providers such as Amazon Web Services, Google Cloud, and Microsoft Azure, have started providing data analytics platform as a service [1, 8, 12], which is being adopted widely. With the improv...

Citation Context

...as number of VMs and the estimated runtime based on Equation 1 as parameters. After carefully considering multiple regression models, we find that a third degree polynomial-based cubic Hermite spline =-=[30]-=- is a good fit for the applications and storage services considered in the paper. While we do not delve into details about the model, we show the accuracy of the splines in Figure 2. We also evaluate ...

Citation Context

... a flash tier for serving reads is beneficial for HDFS-based HBase workloads with random I/Os. As opposed to HBase I/O characteristics, typical MapReduce-like batch jobs issues large, sequential I/Os =-=[40]-=- and run in multiple stages (map, shuffle, reduce). Hence, lessons learned from HBase 46 tiering are not directly applicable to such analytics workloads. hatS [31] and open source Hadoop community [9]...

Citation Context

... analysis goals. Thus, optimizing the system for such applications, as in Cast, can significantly impact the data analytics field. Dynamic vs. Static Storage Tiering Big data frameworks such as Spark =-=[47]-=- and Impala [11] have been used for real-time interactive analytics, where dynamic storage tiering is likely to be more beneficial. In contrast, our work focuses on traditional batch processing analyt...

Citation Context

...t data placement decisions for the same applications. 3.1.1 Experimental Study Setup We select four representative analytics applications that are typical components of real-world analytics workloads =-=[18, 46]-=- and exhibit diversified I/O and computation characteristics, as listed in Table 2. Sort, Join and Grep are I/Ointensive applications. The execution time of Sort is dominated by the shuffle phase I/O,...

Citation Context

...d Granularity We next study the impact of cross-job interactions within an analytics workload. While individual job-level optimization and tiering has been the major focus of a number of recent works =-=[31, 32, 44, 25, 24, 17]-=-, we argue that this is not sufficient for data placement in the cloud for analytics workloads. To this end, we analyze two typical workload characteristics that have been reported in production workl...

Citation Context

...eans spends most of the time performing computation. Furthermore, short-term (within hours) and long-term (daily, weekly or monthly) data reuse across jobs is common in production analytics workloads =-=[18, 15]-=-. As reported in [18], 78% of jobs in Cloudera Hadoop workloads involve data reuse. Another distinguishing feature of analytics workloads is the presence of workflows that represents interdependencies...

Citation Context

...d Granularity We next study the impact of cross-job interactions within an analytics workload. While individual job-level optimization and tiering has been the major focus of a number of recent works =-=[31, 32, 44, 25, 24, 17]-=-, we argue that this is not sufficient for data placement in the cloud for analytics workloads. To this end, we analyze two typical workload characteristics that have been reported in production workl...

Citation Context

...eans spends most of the time performing computation. Furthermore, short-term (within hours) and long-term (daily, weekly or monthly) data reuse across jobs is common in production analytics workloads =-=[18, 15]-=-. As reported in [18], 78% of jobs in Cloudera Hadoop workloads involve data reuse. Another distinguishing feature of analytics workloads is the presence of workflows that represents interdependencies...

Citation Context

...adigm [20, 27] implies that the runtime characteristics of analytics jobs can be predicted with high accuracy. Moreover, extensive recent research has focused on data analytics performance prediction =-=[42, 25, 24, 13, 41, 27]-=-. We leverage and adapt MRCute [27] model in Cast to predict job execution time, due to its ease-of-use, availability, and applicability to our problem domain. Equation 1 defines our performance predi...

Citation Context

...ion platforms. In contrast, we explore the inherent performance and cost trade-off of different storage services in public cloud environments. Analytics Workflow Optimization A large body of research =-=[45, 35, 21, 36, 33]-=- focuses on Hadoop workflow optimizations by integrating workflow-aware scheduler into Hadoop or interfacing Hadoop with a standalone workflow scheduler. Our workflow enhancement is orthogonal and com...

Citation Context

...d Granularity We next study the impact of cross-job interactions within an analytics workload. While individual job-level optimization and tiering has been the major focus of a number of recent works =-=[31, 32, 44, 25, 24, 17]-=-, we argue that this is not sufficient for data placement in the cloud for analytics workloads. To this end, we analyze two typical workload characteristics that have been reported in production workl...

Citation Context

...Cold Data Classification-based Tiering Recent research [28, 34, 43] has focused on improving storage cost and utilization efficiency by placing hot/cold data in different storage tiers. Guerra et. al.=-=[22]-=- builds an SSD-based dynamic tiering system to minimize cost and power consumption, and existing works handle file system and block level I/Os (e.g., 4 – 32 KB) for POSIX-style workloads (e.g., server...

Citation Context

...nstrate very different access characteristics and data dependencies (described in Section 3); requiring a rethink of how storage tiering is done to benefit these workloads. Other works such as Bazaar =-=[27]-=- and Conductor [44], focus on automating cloud resource deployment to meet cloud tenants’ requirements while reducing deployment cost. Our work takes a thematically similar view — exploring the trade-...

Citation Context

...ion platforms. In contrast, we explore the inherent performance and cost trade-off of different storage services in public cloud environments. Analytics Workflow Optimization A large body of research =-=[45, 35, 21, 36, 33]-=- focuses on Hadoop workflow optimizations by integrating workflow-aware scheduler into Hadoop or interfacing Hadoop with a standalone workflow scheduler. Our workflow enhancement is orthogonal and com...

Citation Context

...ion platforms. In contrast, we explore the inherent performance and cost trade-off of different storage services in public cloud environments. Analytics Workflow Optimization A large body of research =-=[45, 35, 21, 36, 33]-=- focuses on Hadoop workflow optimizations by integrating workflow-aware scheduler into Hadoop or interfacing Hadoop with a standalone workflow scheduler. Our workflow enhancement is orthogonal and com...

Citation Context

...f the same size as the input data. Others, such as inverted indexing, would require a large capacity for storing intermediate data as significant larger shuffle data is generated during the map phase =-=[16]-=-. The generic Equation 3 accounts for all such scenarios and guarantees that the workload will not fail. Given a specific tiering solution, the estimated total completion time of the workload is defin...

Citation Context

...ion platforms. In contrast, we explore the inherent performance and cost trade-off of different storage services in public cloud environments. Analytics Workflow Optimization A large body of research =-=[45, 35, 21, 36, 33]-=- focuses on Hadoop workflow optimizations by integrating workflow-aware scheduler into Hadoop or interfacing Hadoop with a standalone workflow scheduler. Our workflow enhancement is orthogonal and com...

Citation Context

...adigm [20, 27] implies that the runtime characteristics of analytics jobs can be predicted with high accuracy. Moreover, extensive recent research has focused on data analytics performance prediction =-=[42, 25, 24, 13, 41, 27]-=-. We leverage and adapt MRCute [27] model in Cast to predict job execution time, due to its ease-of-use, availability, and applicability to our problem domain. Equation 1 defines our performance predi...

Citation Context

...ent access characteristics and data dependencies (described in Section 3); requiring a rethink of how storage tiering is done to benefit these workloads. Other works such as Bazaar [27] and Conductor =-=[44]-=-, focus on automating cloud resource deployment to meet cloud tenants’ requirements while reducing deployment cost. Our work takes a thematically similar view — exploring the trade-offs of cloud servi...

Citation Context

... framework for cloud-based data analytics workloads. Fine-Grained Tiering for Analytics Storage tiering has been studied in the context of data-intensive analytics batch applications. Recent analysis =-=[23]-=- demonstrates that adding a flash tier for serving reads is beneficial for HDFS-based HBase workloads with random I/Os. As opposed to HBase I/O characteristics, typical MapReduce-like batch jobs issue...

Citation Context

... RELATEDWORK In the following, we provide a brief background of storage tiering, and categorize and compare previous work with our research. Hot/Cold Data Classification-based Tiering Recent research =-=[28, 34, 43]-=- has focused on improving storage cost and utilization efficiency by placing hot/cold data in different storage tiers. Guerra et. al.[22] builds an SSD-based dynamic tiering system to minimize cost an...

Citation Context

...ion platforms. In contrast, we explore the inherent performance and cost trade-off of different storage services in public cloud environments. Analytics Workflow Optimization A large body of research =-=[45, 35, 21, 36, 33]-=- focuses on Hadoop workflow optimizations by integrating workflow-aware scheduler into Hadoop or interfacing Hadoop with a standalone workflow scheduler. Our workflow enhancement is orthogonal and com...

Citation Context

...argue that this is not sufficient for data placement in the cloud for analytics workloads. To this end, we analyze two typical workload characteristics that have been reported in production workloads =-=[18, 15, 38, 23, 19]-=-, namely data reuse across jobs, and dependency between jobs, i.e., workflows, within a workload. Data Reuse across Jobs As reported in the analysis of production workloads from Facebook and Microsoft...

Citation Context

... RELATEDWORK In the following, we provide a brief background of storage tiering, and categorize and compare previous work with our research. Hot/Cold Data Classification-based Tiering Recent research =-=[28, 34, 43]-=- has focused on improving storage cost and utilization efficiency by placing hot/cold data in different storage tiers. Guerra et. al.[22] builds an SSD-based dynamic tiering system to minimize cost an...

Citation Context

... — exploring the trade-offs of cloud services — but with a different scope that targets data analytics workloads and leverages their unique characteristics to provide storage tiering. Several systems =-=[14, 37]-=- are specifically designed to tackle flash storage allocation inefficiency in virtualization platforms. In contrast, we explore the inherent performance and cost trade-off of different storage service...

Citation Context

...h jobs issues large, sequential I/Os [40] and run in multiple stages (map, shuffle, reduce). Hence, lessons learned from HBase 46 tiering are not directly applicable to such analytics workloads. hatS =-=[31]-=- and open source Hadoop community [9] have taken the first steps towards integrating heterogeneous storage devices in HDFS for local clusters. However, the absence of task-level tier-aware scheduling ...

Citation Context

...d Granularity We next study the impact of cross-job interactions within an analytics workload. While individual job-level optimization and tiering has been the major focus of a number of recent works =-=[31, 32, 44, 25, 24, 17]-=-, we argue that this is not sufficient for data placement in the cloud for analytics workloads. To this end, we analyze two typical workload characteristics that have been reported in production workl...

Citation Context

... — exploring the trade-offs of cloud services — but with a different scope that targets data analytics workloads and leverages their unique characteristics to provide storage tiering. Several systems =-=[14, 37]-=- are specifically designed to tackle flash storage allocation inefficiency in virtualization platforms. In contrast, we explore the inherent performance and cost trade-off of different storage service...

Citation Context

... RELATEDWORK In the following, we provide a brief background of storage tiering, and categorize and compare previous work with our research. Hot/Cold Data Classification-based Tiering Recent research =-=[28, 34, 43]-=- has focused on improving storage cost and utilization efficiency by placing hot/cold data in different storage tiers. Guerra et. al.[22] builds an SSD-based dynamic tiering system to minimize cost an...

Citation Context

...cloud storage services. Cloud Resource Provisioning Considerable prior work has examined ways to automate resource configuration and provisioning process in the cloud. Frugal Cloud File System (FCFS) =-=[39]-=- is a cost-effective cloud-based file storage that spans multiple cloud storage services. In contrast to POSIX file system workloads, modern analytics jobs (focus of our study) running on parallel pro...