Citation Context

...heduling techniques, and that Corral can work together with such techniques to provide further performance benefits. 2 Motivation Big data clusters use frameworks like Hadoop [3], Hive [44], or Scope =-=[50]-=- to run simple MapReduce [25] jobs, and more complex, DAG-structured jobs. Each job consists of stages, such as map or reduce, linked by data dependencies. Each stage consists of several tasks, which ...

Citation Context

...base system solutions are no longer practical. Several companies have thus developed distributed data storage and processing systems on large clusters of thousands of shared-nothing commodity servers =-=[2, 26, 10, 13]-=-. In the MapReduce model, developers provide map and reduce functions in procedural languages like Java, which Permission to make digital or hard copies of all or part of this work for personal or cla...

Citation Context

...cessed in a shared-nothing distributed system that scales to thousands of machines. MapReduce and Hadoop are two well-known software frameworks. Microsoft’s SCOPE is another instance in this category =-=[4, 22]-=-. It combines MapReduce’s explicit parallel executions and SQL’s declarative query constructs. Today inside Microsoft, SCOPE jobs are running on tens of thousands of machines everyday. While a shared-...

Citation Context

...alized DBMS solutions are no longer practical. Several companies have thus developed distributed data storage and processing systems on large clusters of thousands of shared-nothing commodity servers =-=[2, 4, 11, 24]-=-. In the MapReduce model, developers provide map and reduce functions in procedural languages like Java or C#, which perform data transformation and aggregation. The underlying runtime system achieves...

Citation Context

...ventual manner. Worker nodes in distributed architectures maintain a queue of tasks to minimize time the node spends idle and to throttle polling. Similar to Mercury, Apollo estimates wait times at each node and lazily propagates updates to schedulers. In particular, Apollo uses a principled approach that combines optimizer statistics and observed execution behavior to refine task runtime estimates. These techniques can be incorporated by YARN AMs, which can in turn improve Mercury’s placement and load balancing policies. Note that, unlike Mercury, the scheduler in Apollo is part of the SCOPE [34] application runtime, so operator policies are not enforced, updated, or deployed by the platform. Executor model Single-framework distributed schedulers focus on a different class of workloads. Sparrow [22] and Impala [19] schedule tasks in long-running daemons, targeting sub-second latencies. This pattern is also used in YARN deployments, as applications will retain resources to amortize allocation costs [4, 29, 33] or retain data across queries [20, 32]. In contrast, Mercury not only mixes heterogeneous workloads with fine granularity, but its API also enables jobs to suitably choose a comb...

Citation Context

...ative reducer is a pure function [11] that does not depend on any hidden information or state, nor does have side effects. In practice, a reducer can be impure if it calls a random number generator, reads environment variables on the local machine, or maintains stateful variables in class members whose lifecycle spans multiple invocations on different input groups. Impure reducers might produce different results in different runs. We do not consider impure reducers in this study, using non-commutative reducer to describe a reducer that is pure but not commutative. 2.1 The SCOPE Language SCOPE [15] is a SQL-like declarative language with C# user-defined functions, similar to Pig [10] and Hive [12], that is used by many production teams in Microsoft to write MapReduce programs. SCOPE makes it easy to build a MapReduce pipeline through a single script. We briefly describe the SCOPE language through the example in Figure 4. The EXTRACT statement (lines 1 to 2) reads the input file on the distributed file system into a row set called R, each row of which consists of two columns: Url of default type string and Score of type int. The REDUCE statement (lines 4 to 6) specifies a reduce stage th...

Citation Context

... IO, and traffic volume needed to achieve such scalability. 1 Introduction Data volume has been increasing at an unprecedented pace. Systems for storing and querying big data have developed from a single server to thousands of commodity machines, with no end in sight to this growth. Such drastic changes bring many challenges to system design; deep complexities are baked into the ever-increasing cloud of machines. Yet, performance is often lame, limiting both the system’s and the data’s usefulness. For example, an analysis of hundreds of daily inproduction data mining jobs on Microsoft’s SCOPE [7, 22] platform—a large-scale data processing engine that combines MapReduce’s explicit parallel executions and SQL’s declarative query constructs—reveals that they consume two thousand machine-hours on average, and some jobs take up to 10 hours. Such performance numbers highlight the limits of today’s brute-force, heavy provisioning approach; despite consisting of thousands of machines, applications can issue only a low volume of queries toward these systems. In this paper, we investigate the benefits of a radically different approach. We postulate that, by and large, the complexity of today’s syst...

Citation Context

...y running on expensive high-end servers. However, when the data volumes to be stored and processed reach a point where clusters of hundreds or thousands of servers are required, parallel database solutions become prohibitively expensive. At such scale, many of the underlying assumptions of parallel database systems (e.g., fault tolerance) begin to break down, and the classical solutions are no longer viable without substantial extensions. To address the scalability and reliability challenges, several batch processing systems were proposed, including MapReduce [8], Dryad [12], Hive [19], Scope [21], Pig [15], Stratosphere [3], Impala [13], etc., some of which offer high level SQL-like programming languages and conceptual data models. Such systems are designed to achieve great query throughput over both structured and unstructured datasets with scalability and fault tolerance. The queries usually process tens of terabytes of data or more in batch systems and can take up to hours or days to finish. It is difficult for these systems to achieve interactive query response times for several architectural reasons. For instance, the intermediate results are often materialized to disks, in order...

Citation Context

...ptimizations of procedural code across pipeline stages that can improve I/O performance. This paper presents PeriSCOPE, which automatically optimizes the procedural code of programs that run on SCOPE =-=[8, 42]-=-, a production data-parallel computation system. PeriSCOPE connects the data flow of a SCOPE program’s procedural code together by examining a high-level declarative encoding of the program’s pipeline...