Citation Context

...se algorithms together, then describe novel, practical optimizations for both algorithms. 1. INTRODUCTION Novel large-scale data analytics engines such as SharkSpark [35], Dremel [19], F1 [30], Myria =-=[15]-=- and others [3, 32] use massive parallelism in order to support complex queries on large data sets. These engines are designed to evaluate queries in main memory, because they use sufficiently many se...

Citation Context

...gure 1 shows an example PSLA that our service generates (we describe the experimental setup in Section 4). The example shows a PSLA with four tiers generated for the Myria shared-nothing DBMS service =-=[13, 1]-=-. The database to analyze is a 10GB instance of the Star Schema Benchmark [21]. The PSLA makes cost and performance trade-offs obvious: If a user needs to execute just a few simple selection queries, ...

Citation Context

...has gained importance since the advent of Google’s MapReduce [11] and the emergence of a series of distributed systems with relational operators, such as Pig [20], Hive [22], SparkSQL [21], and Myria =-=[18]-=-. The costs of join algorithms in such systems can be broken down to: (1) local computation of machines; (2) communication between the machines; and (3) the number of global synchronizations that need...

Citation Context

...ary data in parallel directly from another engine. Lastly, we are investigating cross-system monitoring that will migrate data objects between storage engines as query workloads change. We are building a monitoring system that will re-execute portions of a query workload on multiple engines, learning which engines excel at which types of queries. For example, if the majority of the queries accessing MIMIC II’s waveforms use linear algebra, this data would naturally be migrated to an array store. 2.1.1 Multi-system Islands In BigDAWG, we implemented two cross-system islands: D4M [10] and Myria [7]. Each offers a different interface to overlapping set of backend engines. Myria has adopted a programming model of relational algebra extended with iteration. Among other engines, it includes shims to SciDB and Postgres. Myria includes a sophisticated optimizer to efficiently process its query language. On the other hand, D4M uses a new data model, associative arrays, as an access mechanism for existing data stores. This data model unifies multiple storage abstractions, including spreadsheets, matrices, and graphs. D4M has a query language that includes filtering, subsetting, and linear algeb...

Citation Context

...compared inside each worker. We provide in the Appendix of the full paper2 [9] the general description of the distribution strategy. However, whether our distribution strategy can be implemented in a particular shared-nothing system depends on the APIs of the system. For simplicity, we describe and implement our distribution strategy using MapReduce, which uses mappers and partitioners to specify (1) and uses reducers to specify (2). Other example platforms where our distribution strategies could be implemented include Spark [28], Apache Flink (previously known as Stratosphere) [3], and Myria [15]. 3. SINGLE BLOCK DEDUPLICATION In this section, we study the problem of data deduplication for tuples in a single block that is produced by one blocking function; in other words, we need to compare every tuple with every other tuple in the block. The distribution strategy presented in this section will serve as a building block when discussing distribution strategies in Sections 4 and 5. Assume that there are n tuples in the block, and k available reducers to compute the pair-wise similarities. 3.1 Lower Bounds We first analyze the lower bounds Xlow and Ylow for X and Y , respectively. The lo...

Citation Context

...The need to analyze large collections of data continues to grow in industry, government, and sciences. Data analysis has even become recognized as the fourth paradigm of science [38]. Users need tools to analyze this data efficiently and easily. Many systems exist to support big data management and analytics (e.g. [61, 69, 71, 46, 43, 3, 26]), but much room remains for improvement in terms of performance and usability. Beginning in 2012, the database group at the University of Washington, in collaboration with the UW eScience Institute, has built our own big data management stack called Myria [52, 35]. We have been operating Myria as a cloud service initially in our own physical cluster and more recently in the public Amazon EC2 [1] cloud. A demonstration version of the service is available on our project website [52]. The Myria project has two main goals. The first is to build an engine sufficiently mature, fast, and easy-to-use to be adopted by domain scientists. The second goal is to build a platform for testing novel database research ideas motivated by our users’ needs. Rather than extend other big data systems such as Hadoop [69], Spark [71], or SciDB [61], we decided to build our ow...

Citation Context

... Eventually, Jaql and Pig Latin scripts are automatically compiled to MapReduce jobs. A variant of Pig Latin, called MyriaL, extends its programming model with looping constructs and is used in Myria =-=[15]-=-, a big data management service. Iterative processes are also a limitation for Jaql and Sopremo, as the developer is not able to dene in these programming models that a given work ow will be repeated...

Citation Context

... has gained importance since the advent of Google’s MapReduce [9] and the emergence of a series of distributed systems with relational operators, such as Pig [20], Hive [22], SparkSQL [21], and Myria =-=[17]-=-. The costs of join algorithms in such systems can be broken down to: (1) local computation of machines; (2) communication between the machines; and (3) the number of global synchronizations that need...