Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. In a previous paper we showed that prefetching and caching have the potential to deliver the performance bene ts of parallel le systems to parallel applications. In this paper we describe experiments with practical prefetching policies, and show that prefetching can be implemented e ciently even for the more complex parallel le access patterns. We also test the ability of these policies across a range of architectural parameters. 1

Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. Such parallel disk systems require parallel le system software to avoid performance-limiting bottlenecks. We discuss cache management techniques that can be used inaparallel le system implementation. We examine several writeback policies, and give results of experiments that test their performance. 1

As parallel computers are increasingly used to run scienti c applications with large data sets, and as processor speeds continue to increase, it becomes more important to provide fast, e ective parallel le systems for data storage and for temporary les. In an earlier work we demonstrated that a technique we call disk-directed I/O has the potential to provide consistent high performance for large, collective, structured I/O requests. In this paper we expand on this potential by demonstrating the ability of a disk-directed I/O system to read irregular subsets of data from a le, and to lter and distribute incoming data according to data-dependent functions. 1

New le systems are critical to obtain good I/O performance on large multiprocessors. Several researchers have suggested the use of collective le-system operations, in which all processes in an application cooperate in each I/O request. Others have suggested that the traditional lowlevel interface (read, write, seek) be augmented with various higher-level requests (e.g., read matrix), allowing the programmer to express a complex transfer in a single (perhaps collective) request. Collective, high-level requests permit techniques like two-phase I/O and disk-directed I/O to signi cantly improve performance over traditional le systems and interfaces. Neither of these techniques have been tested on anything other than simple benchmarks that read or write matrices. Many applications, however, intersperse computation and I/O to work with data sets that cannot t in main memory. In this paper, we present the results of experiments with an \out-of-core " LU-decomposition program, comparing a traditional interface and le system with a system that has a high-level, collective interface and disk-directed I/O. We found that a collective interface was awkward in some places, and forced additional synchronization. Nonetheless, disk-directed I/O was able to obtain much better performance than the traditional system.

As parallel systems move into the production scienti c computing world, the emphasis will be on cost-e ective solutions that provide high throughput for a mix of applications. Coste ective solutions demand that a system make e ective use of all of its resources. Many MIMD multiprocessors today, however, distinguish between \compute " and \I/O " nodes, the latter having attached disks and being dedicated to running the le-system server. This static division of responsibilities simpli es system management but does not necessarily lead to the best performance in workloads that need a di erent balance of computation and I/O. Of course, computational processes sharing a node with a le-system service may receive less CPU time, network bandwidth, and memory bandwidth than they would on a computationonly node. In this paper we examine this issue experimentally. We found that high-performance I/O does not necessarily require substantial CPU time, leaving plenty of time for application computation. There were some complex le-system requests, however, which left little CPU time available to the application. (The impact on network and memory bandwidth still needs to be determined.) For applications (or users) that cannot tolerate an occasional interruption, we recommend that they continue to use only compute nodes. For tolerant applications needing more cycles than those provided by the compute nodes, we recommend that they take full advantage of both compute and I/O nodes for computation, and that operating systems should make this possible. 1

Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. Such parallel disk systems require parallel file system software to avoid performance-limiting bottlenecks. We discuss cache management techniques that can be used in a parallel file system implementation for multiprocessors with scientific workloads. We examine several writeback policies, and give results of experiments that test their performance.