In this paper we present a new storage architecture for clusters that creates a shared memory of disk storage that is uniformly accessible to all cluster clients, scales to large capacity, and provides very high performance and connectivity. The cluster structure resembles a symmetric multiprocessor (SMP) in that clients (processors) can access disk data (memory) across a local area network like Fibre Channel (a bus or other interconnection network). All clients can see and access the same disk data with perfect consistency. Our approach avoids buffer copy overheads and server bottlenecks found in traditional file systems while scaling to potentially large numbers of clients and large capacity disk systems.

: This paper presents the design and implementation of the recovery scheme in Calypso. Calypso is a cluster-optimized, distributed file system for UNIX clusters. As in Sprite and AFS, Calypso servers are stateful and scale well to a large number of clients. The recovery scheme in Calypso is non-disruptive, meaning that open files remain open, client modified data is saved, and in-flight operations are properly handled across server recovery. The scheme uses distributed state among the clients to reconstruct the server state on a backup node if disks are multi-ported or on the rebooted server node. It guarantees data consistency during recovery and provides congestion control. Measurements show that the state reconstruction can be quite fast: for example, in a 32-node cluster, when an average node contains state for about 420 files, the reconstruction time is about 3.3 seconds. However, the time to update a file system after a failure can be a major factor in the overall recovery time, ...

Shared file systems like Cray's SFS, DEC's VAXcluster file system, and Oracle's Parallel Server exploit network-attached storage by creating serverless distributed file systems that allow efficient, simultaneous access to shared, network-attached storage devices. In the past, these shared file system designs relied on proprietary networkattached storage like DEC's CI network or higher-cost interfaces like HiPPI, or used software to emulate shared storage networks. With the advent of Fibre Channel, a highvolume, open standard in network-attached storage interfaces is now available. In this paper we will review past work in traditional distributed file systems like NFS and AFS and work in shared file systems by Cray, DEC, CMU and others. New file system architectures for shared network storage will also be described. We will

Reliability, availability and scalability are major concerns in the design of distributed file systems. We have built an island-based file system (IFS) called Archipelago to solve these problems by failure isolation and low-cost consistency maintenance. The building blocks of IFS are smaller self-contained file servers called islands. The main idea underlying island-based design is the one-island principle: as many operations as possible should involve exactly one island. The oneisland principle improves partial reliability and availability because each island can function independently of other islands' failures. It allows IFS to scale efficiently with the system and workload sizes because consistency across islands can be maintained at a low cost. The data distribution strategies in existing file systems cannot satisfy the one-island principle without sacrificing load balance and scalability. We designed a new strategy in which data is distributed to islands at directory granularity ...

Network attached storage systems must provide highly available access to data while maintaining high performance, easy management, and maximum scalability. In this paper, we describe a clustered storage system that was designed with these goals in mind. The system provides a unified file system image across multiple nodes, which allows for simplified management of the system. Availability is preserved with multiple nodes and parity-striped data across these nodes. This architecture provides two key contributions: the ability to use low-cost components to deliver scalable performance and the flexibility to specify redundancy and performance policy management on a file-by-file basis. The file system is also tightly integrated with standard distributed file system protocols thereby allowing it to be used in existing networks without modifying clients. 1. Background The traditional storage solution has typically been direct attached storage (DAS) where the actual disk hardware is directly connected to the application server through highspeed channels such as SCSI or IDE. With the proliferation of local area networks, the use of network file servers has increased, leading to the development of several distributed file systems that make the local server DAS file system visible to other machines on the network. These include AFS/Coda [17, 25], NFS [23], Sprite [20], CIFS [14], amongst others. The desire to increase the performance and simplify the administration of these file servers has led to the development of dedicated machines known as networkattached storage (NAS) appliances by companies such as Network Appliance, Auspex, and EMC. In addition to specialized file systems [12], these NAS appliances are also characterized by specialized hardware components to address scalability and reliability [3]. In an effort to remove the bottleneck of the single server model of NAS servers, there has lately been significant work in the area of distributed or clustered storage systems.

With the emergence of Storage Networking, distributed file systems that allow data sharing through shared disks will become vital. We refer to Cluster File Systems as a distributed file systems optimized for environments of clustered servers. The requirements such file systems is that they guarantee file systems consistency while allowing shared access from multiple nodes in a shared-disk environment. In this paper we evaluate three approaches for designing a cluster file system - conventional client/server distributed file systems, symmetric shared file systems and asymmetric shared file systems. These alternatives are considered by using our prototype cluster file system, HAMFS (Highly Available Multi-server File System). HAMFS is classified as an asymmetric shared file system. Its technologies are incorporated into our commercial cluster file system product named SafeFILE. SafeFILE offers a disk pooling facility that supports off-the-shelf disks, and balances file load across these disks automatically and dynamically. From our measurements, we identify the required disk capabilities, such as multi-node tag queuing. We also identify the advantages of an asymmetric shared file system over other alternatives.

Unix’s lack of a robust and expandable file system has become a significant problem with the growth of UNIX in large commercial environments. The HAMFS (Highly Available Multi-server File System) is a cluster file system designed to address this need. HAMFS offers disk-pooling, supports off-the-shelf disks, and automatically balances file load across disks dynamically. Data residing in a disk pool is directly accessible from every node in a HAMFS cluster. As user’s capacity requirements grow, HAMFS provides easy disk pool expansion. Finally, HAMFS provides uniform scaling of file system performance from a single node configuration to large multi-node clusters, offering significant performance advantage over traditional file systems. For example, in short file access situations, HAMFS provide a factor of five performance improvement over NFS, and a factor of two improvement over conventional local file systems. Technologies developed for HAMFS are applied to Fujitsu’s file system product SafeFILE. 1.

and concrete classes of the Recoverable Distributor pattern Structure The structure and relationships between the abstract and concrete classes of recoverable distributor pattern are shown in Figure 1. Participants ffl GlobalState -- Knows LocalStates, and provides an interface to add or delete LocalStates. ffl LocalState -- Defines an interface to the (local) clients to obtain the state. -- Defines an updating operation so that the GlobalState can change the local state to conform to global constraints. ffl GlobalFailureHandler -- Provides an interface to check liveness of the objects maintaining the distributed state; they include the GlobalState and LocalStates. -- Provides an interface to reconstruct the common state up on detecting a partial failure. ffl LocalFailureHandler -- Defines an interface to respond to liveness queries. -- Defines an interface to provide entire local state for the reconstruction of the common state. ffl ConcreteGlobalState -- Stores the comm...

The ideal distributed file system wouldprovide all its users with coherent, shared access to the same set of files,yet would be arbitrarily scalable to provide more storage space and higher performance to a growing user community. It would be highly available in spite of component failures. It would require minimal human administration, and administration would not become more complex as more components were added. Frangipani is a new file system that approximates this ideal, yet was relatively easy to build because of its two-layer structure. The lower layer is Petal (described in an earlier paper), a distributed storage service that provides incrementally scalable, highly available, automatically managed virtual disks. In the upper layer, multiple machines run the same Frangipani file system code on top of a shared Petal virtual disk, using a distributed lock service to ensure coherence. Frangipani is meant to run in a cluster of machines that are under a common administration and ...

In this paper we present a new storage architecture for clusters that creates a shared memory of disk storage that is uniformly accessible to all cluster clients, scales to large capacity, and provides very high performance and connectivity. The cluster structure resembles a symmetric multiprocessor (SMP) in that clients (processors) can access disk data (memory) across a local area network like Fibre Channel (a bus or other interconnection network). All clients can see and access the same disk data with perfect consistency. Our approach avoids buffer copy overheads and server bottlenecks found in traditional file systems while scaling to potentially large numbers of clients and large capacity disk systems. We provide