Abstract—Developing modern distributed software systems is difficult in part because they have little control over the environments in which they execute. For example, hardware and software resources on which these systems rely may fail or become compromised and malicious. Redundancy can help manage such failures and compromises, but when faced with dynamic, unpredictable resources and attackers, the system reliability can still fluctuate greatly. Empowering the system with self-adaptive and self-managing reliability facilities can significantly improve the quality of the software system and reduce reliance on the developer predicting all possible failure conditions. We present iterative redundancy, a novel approach to improving software system reliability by automatically injecting redundancy into the system’s deployment. Iterative redundancy self-adapts in three ways: (1) by automatically detecting when the resource reliability drops, (2) by identifying unlucky parts of the computation that happen to deploy on disproportionately many compromised resources, and (3) by not relying on a priori estimates of resource reliability. Further, iterative redundancy is theoretically optimal in its resource use: Given a set of resources, iterative redundancy guarantees to use those resources to produce the most reliable version of that software system possible; likewise, given a desired increase in the system’s reliability, iterative redundancy guarantees achieving that reliability using the least resources possible. Iterative redundancy handles even the Byzantine threat model, in which compromised resources collude to attack the system. We evaluate iterative redundancy in three ways. First, we formally prove its self-adaptation, efficiency, and optimality properties. Second, we simulate it at scale using discrete event simulation. Finally, we modify the existing, open-source, volunteer-computing BOINC software system and deploy it on the globally-distributed PlanetLab testbed network to empirically evaluate that iterative redundancy is self-adaptive and more efficient than existing techniques.

The ubiquitous nature of computers is driving a massive in-crease in the amount of data generated by humans and ma-chines. Two natural consequences of this are the increased efforts to a. derive meaningful information from accumu-lated data and b. ensure that data is not used for unintended purposes. In the direction of analyzing massive amounts of data (a.), tools like MapReduce, Spark, Dryad and higher-level scripting languages like Pig Latin and DryadLINQ have significantly improved corresponding tasks for software de-velopers. The second, but equally important aspect of en-suring confidentiality (b.), has seen little support emerge for programmers: while advances in cryptographic techniques allow us to process directly on encrypted data, programmer-friendly and efficient ways of programming such data analy-sis jobs are still missing. This paper presents novel data flow analyses and program transformations for Pig Latin, that automatically enable the execution of corresponding scripts on encrypted data. We avoid fully homomorphic encryp-tion because of its prohibitively high cost; instead, in some cases, we rely on a minimal set of operations performed by the client. We present the algorithms used for this transla-tion, and empirically demonstrate the practical performance of our approach as well as improvements for programmers in terms of the effort required to preserve data confidentiality.

The “pay-as-you-go ” cloud computing model has strong potential for efficiently supporting big data analysis jobs expressed via data-flow languages such as Pig Latin. Due to security concerns — in particular leakage of data — government and enterprise institutions are however re-luctant to moving data and corresponding computations to public clouds. We present Crypsis, a system that al-lows execution of MapReduce-style data analysis jobs directly on encrypted data. Crypsis transforms data anal-ysis scripts written in Pig Latin so that they can be ef-ficiently executed on encrypted data. Crypsis to that end employs existing practical partially homomorphic encryption schemes, and adopts a global perspective in that it can perform partial computations on the client side when PHE alone would fail. We outline the original program transformations underlying Crypsis for reduc-ing the cost of data analysis computations in this larger perspective. We show practicality of our approach by evaluating Crypsis on standard benchmarks. 1