Online services hosted in data centers show significant diurnal variation in load levels. Thus, there is significant potential for saving power by powering down excess servers during the troughs. However, while techniques like VM migration can consolidate computational load, storage state has always been the elephant in the room preventing this powering down. Migrating storage is not a practical way to consolidate I/O load. This paper presents Sierra, a power-proportional distributed storage subsystem for data centers. Sierra allows powering down of a large fraction of servers during troughs without migrating data and without imposing extra capacity requirements. It addresses the challenges of maintaining read and write availability, no performance degradation, consistency, and fault tolerance for general I/O workloads through a set of techniques including power-aware layout, a distributed virtual log, recovery and migration techniques, and predictive gear scheduling. Replaying live traces from a large, real service (Hotmail) on a cluster shows power savings of 23%. Savings of 40–50 % are possible with more complex optimizations.

A significant number of emerging on-line data analysis applications require the processing of data streams, large amounts of data that get updated continuously, to generate outputs of interest or to identify meaningful events. Example domains include network traffic management, stock price monitoring, customized e-commerce websites, and analysis of sensor data. In this paper we look at the problem of high availability in such a distributed stream processing system. By taking into account the particular characteristics of stream processing applications we first identify design principles for a replica placement algorithm for high availability. We incorporate these principles in a decentralized replica placement protocol that aims to maximize availability, while respecting resource constraints, and making performance-aware placement decisions. We have integrated our replica placement protocol in Synergy, our distributed stream processing middleware. Our experimental comparison over PlanetLab with the current state of the art corroborates our claims that our techniques maximize availability while sustaining good performance.

I would like to thank my advisor, Prof. Vana Kalogeraki, for the inspiration and guidance, motivation and support she has offered me. I would also like to thank Prof. Xiaohui Gu for her sharp guidance and inspiring attitude. I am also grateful to my committee members, Prof. Dimitrios Gunopulos and Prof. Michalis Faloutsos, for their time and their insightful feedback. I am especially thankful to all my mentors, Dr. Arun Iyengar and Dr. Isabelle Rouvellou from IBM Research, Dr. Michael Kaminsky and Dr. Haifeng Yu from Intel Research, Dr. Debby Levinson and Chris Stroberger from Hewlett-Packard, and Dr. Eric Burger from BEA, for their time and guidance. I would also like to thank the rest of the members of the Distributed Real-Time Systems lab and the anonymous reviewers of [67,68,70–72] for their comments. Finally, I would like to thank my friends and above all my family for their support throughout these years. iv Wenn die Nacht am tiefsten ist, ist der Tag am nächsten.