We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs). We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system. A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supporting more than 250,000 metadata operations per second. 1

The Panasas file system uses parallel and redundant access to object storage devices (OSDs), per-file RAID, distributed metadata management, consistent client caching, file locking services, and internal cluster management to provide a scalable, fault tolerant, high performance distributed file system. The clustered design of the storage system and the use of clientdriven RAID provide scalable performance to many concurrent file system clients through parallel access to file data that is striped across OSD storage nodes. RAID recovery is performed in parallel by the cluster of metadata managers, and declustered data placement yields scalable RAID rebuild rates as the storage system grows larger. This paper presents performance measures of I/O, metadata, and recovery operations for storage clusters that range in size from 10 to 120 storage nodes, 1 to 12 metadata nodes, and with file system client counts ranging from 1 to 100 compute nodes. Production installations are as large as 500 storage nodes, 50 metadata managers, and 5000 clients. 1

In petabyte-scale distributed file systems that decouple read and write from metadata operations, behavior of the metadata server cluster will be critical to overall system performance and scalability. We present a dynamic subtree partitioning and adaptive metadata management system designed to efficiently manage hierarchical metadata workloads that evolve over time. We examine the relative merits of our approach in the context of traditional workload partitioning strategies, and demonstrate the performance, scalability and adaptability advantages in a simulation environment.

OS virtualization is drastically changing the face of system administration for large computer installations such as commercial datacenters and scientific clusters. A recent report by Gartner predicts that commercial use of

As mobile clients travel, their costs to reach home filing services change, with serious performance implications. Current file systems mask these performance problems by reducing the safety of updates, their visibility, or both. This is the result of combining the propagation and notification of updates from clients to servers. Fluid

This paper elaborates on mechanisms by which users, data, and applications can be decoupled from individual computers and administrative domains. The mechanisms, which consist of logical user accounts and a virtual file system, introduce a layer of abstraction between the physical computing infrastructure and the virtual computational grid perceived by users. This abstraction converts compute servers into interchangeable parts, allowing a computational grid to assemble computing systems at run time without being limited by the traditional constraints associated with user accounts, file systems, and administrative domains. The described approach has already been deployed in the context of PUNCH (the Purdue University Network Computing Hubs), and is unique in its ability to integrate unmodified applications (even commercial ones) and existing computing infrastructure into a heterogeneous, wide-area network computing environment. 1. Introduction Network-centric computing promises to re...

This paper describes a virtual le system that allows data to be transferred on demand between storage and compute servers for the duration of a computing session. The solution works with unmodi ed applications (even commercial ones) running on standard operating systems and hardware. The virtual le system employs software proxies to broker transactions between standard NFS clients and servers; the proxies are dynamically con gured and controlled bycomputational grid middleware. The approach has been implemented and extensively exercised in the context of the Purdue University Network Computing Hubs, an operational computing portal that has more than 1,500 users across 24 countries. Results show that the virtual le system performs well in comparison to native NFS: performance analyses show that the proxy incurs mean overheads of 1 % and 18 % with respect to native NFS for a singleclient execution of the Andrew benchmark in two representative computing environments, and that the average overhead for eight clients can be reduced to within 1 % of native NFS with concurrent proxies. 1.

Efficient metadata management is a critical aspect of overall system performance in large distributed storage systems. Directory subtree partitioning and pure hashing are two common techniques used for managing metadata in such systems, but both suffer from bottlenecks at very high concurrent access rates. We present a new approach called Lazy Hybrid (LH) metadata management that combines the best aspects of these two approaches while avoiding their shortcomings.

: This paper presents the design and implementation of the recovery scheme in Calypso. Calypso is a cluster-optimized, distributed file system for UNIX clusters. As in Sprite and AFS, Calypso servers are stateful and scale well to a large number of clients. The recovery scheme in Calypso is non-disruptive, meaning that open files remain open, client modified data is saved, and in-flight operations are properly handled across server recovery. The scheme uses distributed state among the clients to reconstruct the server state on a backup node if disks are multi-ported or on the rebooted server node. It guarantees data consistency during recovery and provides congestion control. Measurements show that the state reconstruction can be quite fast: for example, in a 32-node cluster, when an average node contains state for about 420 files, the reconstruction time is about 3.3 seconds. However, the time to update a file system after a failure can be a major factor in the overall recovery time, ...

Over the last few years, there have been several efforts to use logging to improve performance, reliability, and recovery times of file systems. The two major techniques are metadata logging, where the log records metadata changes and is a supplement to the on-disk file system, and logstructured file systems, whose log is their only ondisk representation. When the file system is mainly or wholly accessed through the Network File System (NFS) protocol, it adds new considerations to the suitability of the logging technique. NFS requires that all operations be updated to stable storage before returning. As a result, file system implementations that were effective for local access may perform poorly on an NFS server. This paper analyzes the issues regarding the use of logging on an NFS server, and describes an implementation of a BSD Fast File System (FFS) with metadata logging that performs effectively for a dedicated NFS server. 1.