Abstract

ABSTRACT The advent of cloud computing is turning datacenter clusters into a commodity. By making large clusters trivial to acquire, manage, and maintain, cloud computing promises to seed a phase of innovative software development, with a wide range of programmers developing new services that scale out quickly and flexibly. However, current cloud platforms focus on relatively traditional, single-server programming models over shared storage, and do little to simplify the task of coordinating large-scale distributed systems. There is as yet no widely-used programming model that lets a developer easily harness the distributed power of a cluster. In this paper, we detail our experience building distributed datacenter software via high-level, data-centric programming. Using the Overlog language and Java, we developed a "Big Data" analytics stack that is API-compatible with Hadoop and HDFS. We describe our experience reimplementing the Hadoop stack and extending it incrementally with new features not yet available in Hadoop, including availability, scalability, and unique monitoring and debugging facilities. Developed in a relatively short nine-month design cycle, our Overlog interpreter and Hadoop implementation perform within a modest factor of the standard Java-only implementation, with a compact and easily-extendible codebase. We reflect on the opportunities and challenges we encountered along the way, which may inform new development environments for distributed programming in datacenters.