Pig Latin: A Not-So-Foreign Language for Data Processing

Pig Latin: A Not-So-Foreign Language for Data Processing

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Christopher Olston , Benjamin Reed , Utkarsh Srivastava , Ravi Kumar , Andrew Tomkins

Citations:

196 - 10 self

Versions

Version 0

Version History

Metadata Version 0

Datum	Value	Source
TITLE	Pig Latin: A Not-So-Foreign Language for Data Processing	SVM HeaderParse 0.2
AUTHOR NAME	Christopher Olston	SVM HeaderParse 0.2
AUTHOR AFFIL	Yahoo! Research	SVM HeaderParse 0.2
AUTHOR NAME	Benjamin Reed	SVM HeaderParse 0.2
AUTHOR AFFIL	Yahoo! Research	SVM HeaderParse 0.2
AUTHOR NAME	Utkarsh Srivastava	SVM HeaderParse 0.2
AUTHOR AFFIL	Yahoo! Research	SVM HeaderParse 0.2
AUTHOR NAME	Ravi Kumar	SVM HeaderParse 0.2
AUTHOR AFFIL	Yahoo! Research	SVM HeaderParse 0.2
AUTHOR NAME	Andrew Tomkins	SVM HeaderParse 0.2
AUTHOR AFFIL	Yahoo! Research	SVM HeaderParse 0.2
ABSTRACT	There is a growing need for ad-hoc analysis of extremely large data sets, especially at internet companies where innovation critically depends on being able to analyze terabytes of data collected every day. Parallel database products, e.g., Teradata, offer a solution, but are usually prohibitively expensive at this scale. Besides, many of the people who analyze this data are entrenched procedural programmers, who find the declarative, SQL style to be unnatural. The success of the more procedural map-reduce programming model, and its associated scalable implementations on commodity hardware, is evidence of the above. However, the map-reduce paradigm is too low-level and rigid, and leads to a great deal of custom user code that is hard to maintain, and reuse. We describe a new language called Pig Latin that we have designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce. The accompanying system, Pig, is fully implemented, and compiles Pig Latin into physical plans that are executed over Hadoop, an open-source, map-reduce implementation. We give a few examples of how engineers at Yahoo! are using Pig to dramatically reduce the time required for the development and execution of their data analysis tasks, compared to using Hadoop directly. We also report on a novel debugging environment that comes integrated with Pig, that can lead to even higher productivity gains. Pig is an open-source, Apache-incubator project, and available for general use. 1.	SVM HeaderParse 0.2
CITATIONS	14 found	ParsCit 1.0