DMCA
External Memory Algorithms and Data Structures (1998)
Cached
Download Links
| Citations: | 349 - 23 self |
BibTeX
@MISC{Vitter98externalmemory,
author = {Jeffrey Scott Vitter},
title = { External Memory Algorithms and Data Structures},
year = {1998}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Data sets in large applications are often too massive to fit completely inside the computer's internal memory. The resulting input/output communication (or I/O) between fast internal memory and slower external memory (such as disks) can be a major performance bottleneck. In this paper, we survey the state of the art in the design and analysis of external memory algorithms and data structures (which are sometimes referred to as "EM" or "I/O" or "out-of-core" algorithms and data structures). EM algorithms and data structures are often designed and analyzed using the parallel disk model (PDM). The three machine-independent measures of performance in PDM are the number of I/O operations, the CPU time, and the amount of disk space. PDM allows for multiple disks (or disk arrays) and parallel CPUs, and it can be generalized to handle tertiary storage and hierarchical memory. We discuss several important paradigms for how to solve batched and online problems efficiently in external memory. Programming tools and environments are available for simplifying the programming task. The TPIE system (Transparent Parallel I/O programming Environment) is both easy to use and efficient in terms of execution speed. We report on some experiments using TPIE in the domain of spatial databases. The newly developed EM algorithms and data structures that incorporate the paradigms we discuss are significantly faster than methods currently used in practice.
Keyphrases
data structure external memory algorithm external memory em algorithm machine-independent measure out-of-core algorithm hierarchical memory several important paradigm large application cpu time programming task programming tool disk space fast internal memory transparent parallel parallel disk model major performance bottleneck spatial database online problem tertiary storage tpie system multiple disk parallel cpu execution speed data set input output communication internal memory
