DMCA
Providing efficient I/O redundancy in MPI environments
| Venue: | Lecture Notes in Computer Science, 3241:77–86, September 2004. 11th European PVM/MPI Users’ Group Meeting |
| Citations: | 1 - 1 self |
BibTeX
@INPROCEEDINGS{Ross_providingefficient,
author = {Robert Ross and Neill Miller},
title = {Providing efficient I/O redundancy in MPI environments},
booktitle = {Lecture Notes in Computer Science, 3241:77–86, September 2004. 11th European PVM/MPI Users’ Group Meeting},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract. Highly parallel applications often make use of either highly parallel file systems or large numbers of independent disks. Either of these approaches can provide high data rates necessary for parallel applications. However, the failure of a single disk or server can render the data useless. Conventional techniques, such as those based on applying erasure correcting codes to each file write, are prohibitively expensive for massively parallel scientific applications because of the granularity of access at which codes are applied. In this paper we demonstrate a scalable method for recovering from single disk failures that is optimized for typical scientific data sets. This approach exploits coarser-grain (but precise) semantics to reduce the overhead of constructing recovery data and makes use of parallel computation (proportional to the data size and independent of number of processors) to construct data. Experiments showing the efficiency of this approach on a cluster with independent disks are presented, and a technique for hiding the creation of redundant data within the MPI-IO implementation is described. 1
Keyphrases
mpi environment independent disk redundant data parallel file system scalable method single disk parallel computation data size high data rate conventional technique highly parallel application recovery data large number file write data useless mpi-io implementation typical scientific data set parallel application single disk failure parallel scientific application
