As massively parallel, distributed memory systems replace traditional vector supercomputers, effective application program optimization and system resource management become more than research curiosities --- they are crucial to achieving substantial fractions of peak performance for scientific application codes. By recording dynamic activity, either at the application or system software level, one can identify and remove performance bottlenecks. Pablo is a performance analysis environment designed to provide performance data capture, analysis, and presentation across a wide variety of scalable parallel systems. The Pablo environment includes software performance instrumentation, graphical performance data reduction and analysis, and support for mapping performance data to both graphics and sound. Current research directions include complete performance data immersion via head-mounted displays and the integration of Pablo with data parallel Fortran compilers based on the emerging High ...

Input/output performance on current parallel file systems is sensitive to a good match of application access pattern to file system capabilities. Automatic input/output access classification can determine application access patterns at execution time, guiding adaptive file system policies. In this paper we examine a new method for access pattern classification that uses hidden Markov models, trained on access patterns from previous executions, to create a probabilistic model of input/output accesses. We compare this approach to a neural network classification framework, presenting performance results from parallel and sequential benchmarks and applications. 1 Introduction Input/output is a critical bottleneck for many important scientific applications. One reason is that performance of extant parallel file systems is particularly sensitive to file access patterns. Often the application programmer must match application input/output requirements to the capabilities of the file system....

Contents Preface ii 1 Introduction 1 1.1 Motivation for the Development of SDDF : : : : : : : : : : : : : : : : : : : : : : : : 1 1.2 The Pablo Self-Defining Data Format : : : : : : : : : : : : : : : : : : : : : : : : : : 2 1.3 The SDDF Interface Library : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 3 1.4 Using the Self-Defining Data Format : : : : : : : : : : : : : : : : : : : : : : : : : : : 3 2 SDDF Basics---Explanation by Example 4 2.1 A Sample SDDF File : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 4 2.2 A Sample Program using the SDDF Interface Library : : : : : : : : : : : : : : : : : 4 3 The Components of an SDDF File 9 3.1 SDDF File Headers : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 3.2 SDDF Packet Types : : : : : : : : : : : : : : : : : : : : : : :

Traditionally, maximizing input/output performance has required tailoring application input /output patterns to the idiosyncrasies of specific input/output systems. In this paper, we show that one can achieve high application input/output performance via a low overhead input /output system that automatically recognizes file access patterns and adaptively modifies system policies to match application input/output needs. This approach reduces the application developer's input/output optimization effort by isolating input/output optimization decisions within a retargetable file system infrastructure. To validate these claims, we have built a lightweight file system policy testbed that uses a trained learning mechanism to recognize access patterns. The file system then uses these access pattern classifications to select appropriate caching strategies, dynamically adapting file system policies to changing input/output demands throughout application execution. Our experimental data show dram...

Parallel input/output systems attempt to alleviate the performance bottleneck that affects many input/output intensive applications. In such systems, an understanding of the application access pattern, especially how requests from multiple processors for different file regions are logically related, is important for optimizing file system performance. We propose a method for automatically classifying these global access patterns and using these global classifications to select and tune file system policies to improve input/output performance. We demonstrate this approach on benchmarks and scientific applications using global classification to automatically select appropriate underlying Intel PFS input/output modes and server buffering strategies. 1 Introduction Despite continued innovations in disk design, input/output performance has not kept pace with concurrent increases in processor speeds. File systems that utilize parallel disks to improve system throughput offer some hope of al...

Despite continued innovations in disk design, input/output performance has not kept pace with concurrent increases in processor speeds. Much research has focused on developing algorithms to avoid input/output or hide input/output latency in an attempt to redress this widening gap. Many studies have shown that with advance knowledge of access patterns, file systems can improve input/output performance by selecting policies appropriate for the resource demands. Unfortunately, access patterns may be complex or data dependent, and therefore unknown a priori. Our thesis is that the file system can automatically detect qualitative file access patterns both locally (per parallel program thread) and globally (per parallel program) and use this information to dynamically choose appropriate file system policies. We propose two complementary methods for automatic classification, based on neural networks and hidden Markov models, respectively. Global classifications are created from a combination...

Parallel input/output characterization studies and experiments with flexible resource management algorithms indicate that adaptivity is crucial to file system performance. In this paper we propose an automatic technique for selecting and refining file system policies based on application access patterns and execution environment. An automatic classification framework allows the file system to select appropriate caching and prefetching policies, while performance sensors provide feedback used to tune policy parameters for the specific system environment. To illustrate the potential performance improvements possible using adaptive file system policies, we present results from experiments involving classification-based and performance-based steering.