Our growing reliance on online services accessible on the Internet demands highly available systems that provide correct service without interruptions. Software bugs, operator mistakes, and malicious attacks are a major cause of service interruptions and they can cause arbitrary behavior, that is, Byzantine faults. This article describes a new replication algorithm, BFT, that can be used to build highly available systems that tolerate Byzantine faults. BFT can be used in practice to implement real services: it performs well, it is safe in asynchronous environments such as the Internet, it incorporates mechanisms to defend against Byzantine-faulty clients, and it recovers replicas proactively. The recovery mechanism allows the algorithm to tolerate any number of faults over the lifetime of the system provided fewer than 1/3 of the replicas become faulty within a small window of vulnerability. BFT has been implemented as a generic program library with a simple interface. We used the library to implement the first Byzantine-fault-tolerant NFS file system, BFS. The BFT library and BFS perform well because the library incorporates several important optimizations, the most important of which is the use of symmetric cryptography to authenticate messages. The performance results show that BFS performs 2 % faster to 24 % slower than production implementations of the NFS protocol that are not replicated. This supports our claim that the

We consider the consensus problem in asynchronous models enriched with unreliable failure detectors or partial synchrony, where processes can crash or links may fail by losing messages.

. This paper develops a new I/O automaton model called the Clock General Timed Automaton (Clock GTA) model. The Clock GTA is based on the General Timed Automaton (GTA) of Lynch and Vaandrager. The Clock GTA provides a systematic way of describing timingbased systems in which there is a notion of "normal" timing behavior, but that do not necessarily always exhibit this "normal" behavior. It can be used for practical time performance analysis based on the stabilization of the physical system. We use the Clock GTA automaton to model, verify and analyze the paxos algorithm. The paxos algorithm is an efficient and highly faulttolerant algorithm, devised by Lamport, for reaching consensus in a distributed system. Although it appears to be practical, it is not widely known or understood. This paper contains a new presentation of the paxos algorithm, based on a formal decomposition into several interacting components. It also contains a correctness proof and a time performance and fault-tole...

Distributed applications often use quorums in order to guarantee consistency. With emerging world-wide communication technology, many new applications (e.g. conferencing applications and interactive games) wish to allow users to freely join and leave, without restarting the entire system. The dynamic voting paradigm allows such systems to define quorums adaptively, accounting for the changes in the set of participants. Furthermore, dynamic voting was proven to be the most available paradigm for maintaining quorums in unreliable networks. However, the subtleties of implementing dynamic voting were not well understood, in fact many of the suggested protocols may lead to inconsistencies in case of failures. Other protocols severely limit the availability in case failures occur during the protocol. In this paper we present a robust and efficient dynamic voting protocol for unreliable asynchronous networks. The protocol consistently maintains the primary component in a distributed system. O...

We present a simple and efficient protocol for mutual exclusion in synchronous, message-passing distributed systems subject to failures. Our protocol borrows design principles from prior work in backoff protocols for multiple access channels such as Ethernet. Our protocol is adaptive in that the expected amortized system response time— informally, the average time a process waits before entering the critical section—is a function only of the number of clients currently contending and is independent of the maximum number of processes who might contend. In particular, in the contention-free case, a process can enter the critical section after only one round-trip message delay. We use this protocol to derive a protocol for ordering operations on a replicated object in an asynchronous distributed system subject to failures. This protocol is always safe, is probabilistically live during periods of stability, and is suitable for deployment in practical systems. 1

We present an algorithm for Totally Ordered Broadcast in the face of network partitions and process failures, using an underlying group communication service as a building block. The algorithm always allows a majority (or quorum) of connected processes in the network to make progress (i.e., to order messages), if they remain connected for sufficiently long, regardless of past failures. Furthermore, the algorithm always allows processes to initiate messages, even when they are not members of a majority component in the network. These messages are disseminated to other processes using a gossip mechanism. Thus, messages can eventually become totally ordered even if their initiator is never a member of a majority component. The algorithm guarantees that when a majority is connected, each message is ordered within at most two communication rounds, if no failures occur during these rounds.

In this paper, we study the safety guarantees of group communication-based database replication techniques. We show that there is a model mismatch between group communication and database, and because of this, classical group communication systems cannot be used to build 2-safe database replication. We propose a new group communication primitive called \emph{end-to-end atomic broadcast} that solves the problem, i.e., can be used to implement 2-safe database replication. We also introduce a new safety criterion, called \emph{group-safety}, that has advantages both over 1-safety and 2-safety. Experimental results show the gain of efficiency of group-safety over lazy replication, which ensures only 1-safety.

The PAXOS algorithm is an e#cient and highly fault-tolerant algorithm, devised by Lamport, for reaching consensus in a distributed system. Although it appears to be practical, it seems to be not widely known or understood. This paper contains a new presentation of the PAXOS algorithm, based on a formal decomposition into several interacting components. It also contains a correctness proof and a time performance and fault-tolerance analysis. The formal framework used for the presentation of the algorithm is provided by the Clock General Timed Automaton (Clock GTA) model. The Clock GTA provides a systematic way of describing timing-based systems in which there is a notion of "normal" timing behavior, but that do not necessarily always exhibit this "normal" timing behavior. c 2000 Elsevier Science B.V. All rights reserved. Keywords: I=O automata models; Formal veri#cation; Distributed consensus; Partially synchronous systems; Fault-tolerance Contents 1.

Fault tolerant distributed systems often select a primary component to allow a subset of the processes to function when failures occur. Several studies have examined algorithms for selecting primary components. However, these studies have assumed that every attempt made by the algorithm to form a new primary component terminates successfully. Unfortunately, in real systems, this is not always the case: if a change in connectivity occurs while the algorithm is still running, algorithms typically block until processes can resolve the outcome of the interrupted attempt.

Failure detectors of the class denoted eventually suspect all crashed processes in a permanent way (completeness) and ensure that, at any time, no more than n- t -1 alive processes are falsely suspected (accuracy), n being the total number of processes. This paper first shows that a simple combination of such a failure detector with a two-step communication pattern can provide the processes with an interesting intersection property on sets of values. As an example illustrating the benefit and the property that such a combination can provide when designing protocols, a leader-based consensus protocol whose design relies on its systematic use is presented. Then the paper presents a -based protocol that builds quorums in systems where up to t processes can crash with t < n.