We propose a small number of basic concepts that can be used to explain the architecture of fault-tolerant distributed systems and we discuss a list of architectural issues that we find useful to consider when designing or examining such systems. For each issue we present known solutions and design alternatives, we discuss their relative merits and we give examples of systems which adopt one approach or the other. The aim is to introduce some order in the complex discipline of designing and understanding fault-tolerant distributed systems. 1 Introduction Computing systems consist of a multitude of hardware and software components that are bound to fail eventually. In many systems, such component failures can lead to unanticipated, potentially disruptive failure behavior and to service unavailability. Some systems are designed to be fault-tolerant: they either exhibit a well-defined failure behavior when components fail or mask component failures to users, that is, continue to provid...

We present a new protocol for maintaining replicated data that can provide both high data availability and low response time. In the protocol, the nodes are organized in a logical grid. Existing protocols are designed primarily to achieve high availability by updating a large fraction of the copies which provides some (although not significant) load sharing. In the new protocol, transaction processing is shared effectively among nodes storing copies of the data and both the response time experienced by transactions and the system throughput are improved significantly. We present an analysis of the availability of the new protocol and use simulation to study the effect of load sharing on the response time of transactions. We also compare the new protocol with a voting based scheme. This work was supported in part by NSF grants NCR-8604850 and CCR-8806358, and by the University Research Committee of Emory University. 1 Introduction A distributed system consists of cooperating process...

Ad-hoc wireless networks have no wired component, and may have unpredictable mobility pattern. Such networks can get partitioned and reconnected several times. One possible approach for information dissemination in such networks is to replicate information at multiple nodes acting as repositories, and employ quorum based strategies to update and query information. We propose three such strategies that also use local knowledge about the reachability of repositories to judiciously select quorums. The primary goal is high availability of information in the face of network partitioning. We also consider four policies to determine the appropriate time to perform updates. Experimental results indicate that a hybrid information management strategy and an absolute connectivity-based update trigger policy are most suited for partitionable ad-hoc networks.

... this article, we analyze several quorum types in order to better understand their behavior in practice. The results obtained challenge many of the assumptions behind quorum based replication. Our evaluation indicates that the conventional read-one/write-all-available approach is the best choice for a large range of applications requiring data replication. We believe this is an important result for anybody developing code for computing clusters as the read-one/write-all-available strategy is much simpler to implement and more flexible than quorum-based approaches. In this article, we show that, in addition, it is also the best choice using a number of other selection criteria

In the weighted voting protocol which is used to maintain the consistency of replicated data, the availability of the data to read and write operations not only depends on the availability of the nodes storing the data but also on the vote and quorum assignments used. We consider the problem of determining the vote and quorum assignments that yield the best permormance in a distributed system where node availabilities can be different and the mix of the read and write operations is arbitrary. The optimal vote and quorum assignments depend not only on the system parameters such as node availability and operation mix, but also on the performance measure. We present an enumeration algorithm that can be used to find the vote and quorum assignments that need to be considered for achieving optimal performance. When the performance measure is data availability, an analytical method is derived to evaluate it for any vote and quorum assignment. This method and the enumeration algorithm is used ...

Voting protocols ensure the consistency of replicated objects by requiring all read and write requests to collect an appropriate quorum of replicas. We propose to replace some of these replicas by volatile witnesses that have no data and require no stable storage, and to regenerate them instead of waiting for recovery. The small size of volatile witnesses allows them to be regenerated much easier than full replicas. Regeneration attempts are also much more likely to succeed since volatile witnesses can be stored on diskless sites. We show that under standard Markovian assumptions two full replicas and one regenerable volatile witness managed by a two-tier dynamic voting protocol provide a higher data availability than three full replicas managed by majority consensus voting or optimistic dynamic voting provided site failures can be detected significantly faster than they can be repaired. Keywords: distributed file systems, replicated data, voting, witnesses. 1. INTRODUCTION Fault-tol...

This paper discusses the currency management mechanisms used in Deno, a replicated object storage system designed for use in mobile and weakly-connected environments. Deno primarily differs from previous work in implementing an asynchronous weighted-voting scheme via epidemic information flow, and in committing updates in an entirely decentralized fashion, without requiring any server to have complete knowledge of system membership.

Voting protocols are widely used to provide mutual exclusion in distributed systems and to guarantee the consistency of replicated data in the presence of network partitions. Unfortunately, the most efficient voting protocols require fairly complex metadata to assert which replicas are up-to-date and to denote the replicas that belong to that set. We present a much simpler technique that does not require version numbers and maintains only n+log(n) bits of state per replica. We show, under standard Markovian assumptions, that a static voting protocol using our method provides nearly the same data availability as a static voting protocol using version numbers. We also describe a dynamic voting protocol using our method that provides the same data availability as a dynamic voting protocol using much more complex metadata. I. INTRODUCTION Voting protocols have been extensively used to provide mutual exclusion in distributed systems and to guarantee the consistency of replicated data in ...

. The paper describes a new dynamic protocol for managing replicated data. Like the existing dynamic schemes, our protocol involves reconfiguring the system dynamically to reflect failures and repairs as they occur, so that the data may be kept available for user operations even if only one replica of the data remains accessible. However, in the existing schemes, it is required for the correctness of the protocol that the system reconfiguration either run as a special transaction that competes for locks with user operations and thus can interfere with and delay the latter, or be done within the protocol for the write operation, which increases the cost of writes and makes fault tolerance of the system dependent on the rate of write operations. In contrast, our protocol allows system reconfiguration to be done asynchronously with and separately from read and write operations. Therefore, in our protocol, user operations can execute while system reconfiguration is in progress, with no int...

This article proposes a scalable protocol for replication management in large-scale replicated systems. The protocol organizes sites and data replicas into a tree-structured, hierarchical cluster architecture. The basic idea of the protocol is to accomplish the complex task of updating replicated data with a very large number of replicas by a set of related but independently committed transactions. Each transaction is responsible for updating replicas in exactly one cluster and invoking additional transactions for member clusters. Primary copies (one from each cluster) are updated by a cross-cluster transaction. Then each cluster is independently updated by a separate transaction. This decoupled update propagation process results in possible multiple views of replicated data in a cluster. Compared to other replicated data management protocols, the proposed protocol has several unique advantages. First, thanks to a smaller number of replicas each transaction needs to atomically update in a cluster, the protocol significantly reduces the transaction abort rate, which tends to soar in large transactional systems. Second, the protocol improves user-level transaction response time as top-level update transactions are allowed to commit before all replicas have been updated. Third, read-only queries have the flexibility to see database views of different degrees of consistency and data currency. This ranges from global, most up to date, and consistent views, to local, consistent, but potentially old views, to local, nearest to users but potentially inconsistent views. Fourth, the protocol maintains its scalability by allowing dynamic system reconfiguration as it grows by splitting a cluster into two or more smaller ones. Fifth, autonomy of the clusters is preserved as no speci...