Stream processing fits a large class of new applications for which conventional DBMSs fall short. Because many stream-oriented systems are inherently geographically distributed and because distribution offers scalable load management and higher availability, future stream processing systems will operate in a distributed fashion. They will run across the Internet on computers typically owned by multiple cooperating administrative domains. This paper describes the architectural challenges facing the design of large-scale distributed stream processing systems, and discusses novel approaches for addressing load management, high availability, and federated operation issues. We describe two stream processing systems, Aurora * and Medusa, which are being designed to explore complementary solutions to these challenges. 1

In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture. 1

This dissertation makes the case that programs can be updated while they run, with modest programmer effort, while providing certain update safety guarantees, and without imposing a significant performance overhead. Few systems are designed with on-the-fly updating in mind. Those systems that permit it support only a very limited class of updates, and generally provide no guarantees that following the update, the system will behave as intended. We tackle the on-the-fly updating problem using a compiler-based approach called dynamic software updating (DSU), in which a program is patched with new code and data while it runs. The challenge is in making DSU practical: it should support changes to programs as they occur in practice, yet be safe, easy to use, and not impose a large overhead. This dissertation makes both theoretical contributions—formalisms for reasoning about, and ensuring update safety—and practical contributions—Ginseng, a DSU implementation for C. Ginseng supports a broad range of changes to C programs, and performs a suite of safety analyses to ensure certain update safety

into a computing giant able to run resource-intensive applications such as natural language translation, speech recognition, face recognition, and augmented reality. However, easily partitioning these applications for remote execution while retaining application-specific information has proven to be a difficult challenge. In this paper, we show that automated dynamic repartitioning of mobile applications can be reconciled with the need to exploit application-specific knowledge. We show that the useful knowledge about an application relevant to remote execution can be captured in a compact declarative form called tactics. Tactics capture the full range of meaningful partitions of an application and are very small relative to code size. We present the design of a tactics-based remote execution system, Chroma, that performs comparably to a runtime system that makes perfect partitioning decisions. Furthermore, we show that Chroma can automatically use extra resources in an overprovisioned environment to improve application performance.

A distributed Java Virtual Machine (DJVM) spanning multiple cluster nodes can provide a true parallel execution environment for multi-threaded Java applications. Most existing DJVMs suffer from the slow Java execution in interpretive mode and thus may not be efficient enough for solving computation-intensive problems. We present JESSICA2, a new DJVM running in JIT compilation mode that can execute multi-threaded Java applications transparently on clusters. JESSICA2 provides a single system image (SSI) illusion to Java applications via an embedded global object space (GOS) layer. It implements a cluster-aware Java execution engine that supports transparent Java thread migration for achieving dynamic load balancing. We discuss the issues of supporting transparent Java thread migration in a JIT compilation environment and propose several lightweight solutions. An adaptive migrating-home protocol used in the implementation of the GOS is introduced. The system has been implemented on x86-based Linux clusters, and significant performance improvements over the previous JESSICA system have been observed.

Virtual machine distributed computing greatly simplifies the use of widespread computing resources by lowering the level of abstraction, benefiting both resource providers and users. Towards that end our Virtuoso middleware closely emulates the existing process of buying, configuring and using physical machines. Virtuoso's VNET component is a simple and efficient layer two virtual network tool that makes these virtual machines (VMs) appear to be physically connected to the home network of the user while simultaneously supporting arbitrary topologies and routing among them. Virtuoso's VTTIF component continually infers the communication behavior of the application running in a collection of VMs. The combination of overlays like VNET and inference frameworks like VTTIF has great potential to increase the performance, with no user or developer involvement, of existing, unmodified applications by adapting their virtual environments to the underlying computing infrastructure to best suit the applications. We show here how to use the continually inferred application topology and traffic to dynamically control three mechanisms of adaptation, VM migration, overlay topology, and forwarding to significantly increase the performance of two classes of applications, bulk synchronous parallel applications and transactional web ecommerce applications.

In this paper, we present the design and implementation of Smart Messages, a distributed computing platform for networks of embedded systems based on execution migration. A Smart Message (SM) is a user-defined distributed program which executes on nodes of interest, named by their properties, and uses an explicit lightweight migration to reach these nodes. During migrations, an SM carries its code and execution state, and it self-routes at each intermediate node between two nodes of interest. The nodes in the network cooperate to support the SM execution by providing a virtual machine and a shared memory region addressable by names (tag space). To illustrate the flexibility of SMs to program real world applications, we describe EZCab, an application for booking cabs in densely populated urban areas. We also present experimental results to quantify the performance achieved by the SM prototype. 1.

To be able to fully exploit ever larger computing platforms, modern HPC applications and system software must be able to tolerate inevitable faults. Historically, MPI implementations that incorporated fault tolerance capabilities have been limited by lack of modularity, scalability and usability. This paper presents the design and implementation of an infrastructure to support checkpoint/restart fault tolerance in the Open MPI project. We identify the general capabilities required for distributed checkpoint/restart and realize these capabilities as extensible frameworks within Open MPI’s modular component architecture. Our design features an abstract interface for providing and accessing fault tolerance services without sacrificing performance, robustness, or flexibility. Although our implementation includes support for some initial checkpoint/restart mechanisms, the framework is meant to be extensible and to encourage experimentation of alternative techniques within a production quality MPI implementation. 1.

Traditional application programming interfaces for transport protocols make a virtue of hiding most internal per-connection state. We argue that this informationhiding precludes many potentially useful application features and performance optimizations. We advocate a disciplined, portable, and secure interface that gives applications both "get" and "set" access to transport connection state.

Abstract. This paper describes MobJeX, an adaptive Java based application framework that uses a combination of pre-processing and runtime support to provide transparent object mobility (including AWT and Swing user interface components) between workstations, PDAs and smartphones. Emphasis is placed on the mobility subsystem (MS), a mobile object transport mechanism providing a high level of transparency and portability from the perspective of the system and the developer. The MS is compared to its most similar predecessor FarGo, demonstrating the advantages of the MS in terms of transparency and portability. Furthermore, a series of laboratory tests are conducted in order to quantify the runtime performance of the MS and two other systems, FarGo and Voyager.