Efficient Structured Data Access in Parallel File Systems

DMCA

Efficient Structured Data Access in Parallel File Systems (2003)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Avery Ching , Alok Choudhary , Wei-keng Liao

Venue:	In Proceedings of the IEEE International Conference on Cluster Computing
Citations:	29 - 13 self

BibTeX

@INPROCEEDINGS{Ching03efficientstructured,
    author = {Avery Ching and Alok Choudhary and Wei-keng Liao},
    title = {Efficient Structured Data Access in Parallel File Systems},
    booktitle = {In Proceedings of the IEEE International Conference on Cluster Computing},
    year = {2003}
}

OpenURL

Abstract

Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel file systems do not traditionally support such access. In this work we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support “datatype I/O” because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation. 1.

Keyphrases

parallel file system efficient structured data access structured datasets efficient access mpi datatypes convenient apis test application high-level interface mpi-io operation parallel virtual file system data representation reusable datatype-processing component mpi-io interface important step structured data access support mpich2 mpi implementation processing path support datatype parallel scientific application store efficient operation

DMCA