Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [14] and distribution functionality from Medusa [51]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly required by newly-emerging stream processing applications. In this paper, we outline the basic design and functionality of Borealis. Through sample real-world applications, we motivate the need for dynamically revising query results and modifying query specifications. We then describe how Borealis addresses these challenges through an innovative set of features, including revision records, time travel, and control lines. Finally, we present a highly flexible and scalable QoS-based optimization model that operates across server and sensor networks and a new fault-tolerance model with flexible consistency-availability trade-offs.

Regis is a programming environment aimed at supporting the development and execution of distributed programs. It embodies a constructive approach to the development of programs based on separating program structure from communication and computation. The emphasis is on constructing programs from multiple parallel computational components which cooperate to achieve the overall goal. The environment is designed to easily accommodate multiple communication mechanisms and primitives. Both the computational and communication elements of Regis programs are programmed in the Object Oriented programming language C++. The elements are combined into distributed programs using the configuration language Darwin. The paper describes programming in Regis through a set of small example programs drawn from the implementation of an Active Badge system. Keywords configuration programming, dynamic reconfiguration, inter-process communication, distributed programming language, software development enviro...

Networking computers has provided a great potential for performance improvement of job execution through dynamic load balancing. In effect, the literature on this subject concludes that dynamic load balancing algorithms can significantly reduce jobs waiting time. The application of these encouraging results onto real systems has proven to be difficult due to the complexity, lack of support, and lack of adequate software tools for constructing distributed load balancing systems This paper presents the design, implementation and a performance analysis of a dynamic load balancing system. The performance evaluation is undertaken using an Office Information System (OIS) as a case study. The dynamic load balancing prototype has been developed using a distributed programming model which supports dynamic configuration. It is argued that this latter exhibits capabilities that are essential for a system operating in a changing and evolving environment. The overall system has been implemented an...

Abstract: This paper presents an algorithm for scheduling parallel applications in large-scale, multiuser, heterogeneous distributed systems. The approach is primarily targeted at systems that harvest idle cycles in general-purpose workstation networks, but is also applicable to clustered computer systems and massively parallel processors. The algorithm handles unequal processor capacities, multiple architecture types and dynamic variations in the number of processes and available processors. Scheduling decisions are driven by the desire to minimize turnaround time while maintaining fairness among competing applications. For efficiency, the virtual processors (VPs) of each application are gang scheduled on some subset of the available physical processors.

Distributed-controlled dynamic load balancing algorithms are known to have several advantages over centralised algorithms such as scalability, and fault tolerance. Distributed implies that the control is decentralised and that a copy of the algorithm (called a scheduler) is replicated on each host of the network. However, distributed control also contributes to the lack of global goals and lack of coherence. This paper presents a new algorithm called DGP (Decentralised Global Plans) that addresses the problem of coherence and coordination in distributed dynamic load balancing algorithms. The DGP algorithm is based on a strategy called Global Plans (GP), and aims at maintaining all computational loads of a distributed system within a band called D. The rationale for the design of DGP is to allow each scheduler to consider the actions of its peer schedulers. With this level of coordination, the schedulers can act more as a coherent team. This new approach first explicitly specifies a glo...

This paper presents the performance evaluation of a coherent distributed dynamic load balancing algorithm called the DGP (Decentralised Global Plans) algorithm. The performance evaluation is underpinned by a research methodology integrating a simulation-based performance evaluation with a prototype-based performance evaluation. The DGP algorithm addresses the issue of coherence and coordination in distributed dynamic load balancing algorithms. The DGP algorithm is based on a strategy called Global Plans (GP). This new approach first explicitly specifies a global goal and then design a strategy around this global goal such that each scheduler (1) takes into account local decisions made by other schedulers; (2) takes into account the effect of its local decisions on the overall system and (3) ensures load balancing. A simulation-based performance evaluation of DGP with two other well-known dynamic load balancing algorithms published in the literature shows that DGP performs consistently ...

Formal static analysis of the correctness and complexity of scalable and adaptive algorithms for distributed systems is difficult and often not appropriate. Rather, tool support is required to facilitate the 'trial and error' approach which is often adopted. Simulation supports this experimental approach well. In this paper we discuss the need for a rigorous approach to simulation results analysis and model validation. These aspects are often neglected in simulation studies, particularly in distributed simulation. Our aim is to provide the practitioner with a set of guidelines which can be used as a `recipe' in different simulation environments, making sound techniques (simulation and statistics) accessible to users. We demonstrate use of the suggested analysis method with two different distributed simulators (CNCSIM [8]) and (NEST[3]) thus illustrating its generality. The same guidelines may be used with other simulation tools to ensure meaningful results while obviating the need to a...

Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [13] and distribution functionality from Medusa [49]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly required by newly-emerging stream processing applications. In this paper, we outline the basic design and functionality of Borealis. Through sample real-world applications, we motivate the need for dynamically revising query results and modifying query specifications. We then describe how Borealis addresses these challenges through an innovative set of features, including revision records, time travel, and control lines. Finally, we present a highly flexible and scalable QoS-based optimization model that operates across server and sensor networks and a new fault-tolerance model with flexible consistency-availability trade-offs.

In distributed systems decisions should be based on currently available local information whenever possible. When weak-consistency suffices, partial system state information is used to arrive at scalable and adaptive decisions. Adaptive partitioning which is part of a load sharing algorithm is proposed in [6] resulting in improved system performance. Adaptive partitioning is applicable to applications requiring scalability and possessing some criteria (such as load state) for node mutual interest. Also, distributed systems such as the WWW cover vast geographical areas. In such systems communication may suffer lengthy network delays. This paper proposes to further improve performance by handling proximity in addition to system size. The benefit of the extended adaptive partitioning algorithm is demonstrated with simulation results produced by the NEST [4] simulator for the extended flexible load sharing algorithm (FLS). Introduction Scale is recognized as a primary factor influencing t...

The 1996 Atlanta Olympic Games were the first to have an official Web site. This site had more than 60 Web server hosts, spread across 5 locations in 4 countries, and registered over 192 million hits during the 17 days of the games. This paper describes some of the challenges of building, operating and managing such a large distributed system. In particular, it outlines the use of novel TCP/IP technologies for sharing the load of HTTP requests across local and global servers. Using these technologies enabled us to build a scalable distributed Web server site that was able to comfortably support a huge demand. We also describe some of the management tools used for monitoring the system, logging HTTP requests, and providing secure access to system resources. 1 Introduction The 1996 Atlanta Olympic Games Web site, www.atlanta.olympic.org, was created and operated by IBM. The name WOMplex 1 was used as the umbrella code-name to describe a combination of technologies that included conte...