## Design and Implementation of a Practical I/O-efficient Shortest Paths Algorithm

Citations: | 2 - 0 self |

### BibTeX

@MISC{Meyer_designand,

author = {Ulrich Meyer and Vitaly Osipov},

title = {Design and Implementation of a Practical I/O-efficient Shortest Paths Algorithm},

year = {}

}

### OpenURL

### Abstract

We report on initial experimental results for a practical I/O-efficient Single-Source Shortest-Paths (SSSP) algorithm on general undirected sparse graphs where the ratio between the largest and the smallest edge weight is reasonably bounded (for example integer weights in {1,...,2 32}) and the realistic assumption holds that main memory is big enough to keep one bit per vertex. While our implementation only guarantees average-case efficiency, i.e., assuming randomly chosen edge-weights, it turns out that its performance on real-world instances with non-random edge weights is actually even better than on the respective inputs with random weights. Furthermore, compared to the currently best implementation for external-memory BFS [6], which in a sense constitutes a lower bound for SSSP, the running time of our approach always stayed within a factor of five, for the most difficult graph classes the difference was even less than a factor of two. We are not aware of any previous I/O-efficient implementation for the classic general SSSP in a (semi) external setting: in two recent projects [10, 23], Kumar/Schwabe-like SSSP approaches on graphs of at most 6 million vertices have been tested, forcing the authors to artificially restrict the main memory size, M, to rather unrealistic 4 to 16 MBytes in order not to leave the semi-external setting or produce huge running times for larger graphs: for random graphs of 2 20 vertices, the best previous approach needed over six hours. In contrast, for a similar ratio of input size vs. M, but on a 128 times larger and even sparser random graph, our approach was less than seven times slower, a relative gain of nearly 20. On a real-world 24 million node street graph, our implementation was over 40 times faster. Even larger gains of over 500 can be estimated for ran-

### Citations

1436 | A note on two problems in connexion with graphs
- Dijkstra
- 1959
(Show Context)
Citation Context ... find, for every vertex v ∈ V , the distance, dist(s, v), from s to v, that is, the length of a shortest path from s to v in G. The classical SSSP-algorithm for general graphs is Dijkstra’s algorithm =-=[14]-=-. Unfortunately, it performs poorly on massive graphs that do not fit into the main memory and are stored on disk. The reason is that Dijkstra’s algorithm accesses the data in an unstructured fashion.... |

537 |
The input/output complexity of sorting and related problems
- Aggarwal, Vitter
- 1988
(Show Context)
Citation Context ...kstra’s algorithm accesses the data in an unstructured fashion. Much recent work has focused on algorithms for massive graphs, see [19, 27] for surveys. These algorithms are analyzed in the I/O-model =-=[3]-=-, which assumes that the computer has a main memory that can hold M vertices or edges and that the graph is stored on disk. In order to process the graph, pieces of it have to be loaded into memory, w... |

320 | External Memory Algorithms and Data Structures: Dealing with Massive Data
- Vitter
- 1981
(Show Context)
Citation Context ...t into the main memory and are stored on disk. The reason is that Dijkstra’s algorithm accesses the data in an unstructured fashion. Much recent work has focused on algorithms for massive graphs, see =-=[19, 27]-=- for surveys. These algorithms are analyzed in the I/O-model [3], which assumes that the computer has a main memory that can hold M vertices or edges and that the graph is stored on disk. In order to ... |

172 | External-memory graph algorithms
- Chiang, Goodrich, et al.
- 1995
(Show Context)
Citation Context ...d by simplifying MZ SSSP in two ways: (1) using the realistic assumption that the main 1 The current bounds for MST(n, m) are O(sort(m) log log (nB/m)) [7] deterministically and O(sort(m)) randomized =-=[11]-=-. memory is big enough to keep one bit per vertex (i.e., the weakest form of the semi-external memory setting), thus facilitating to apply a standard external-memory priority queue without support for... |

124 | Algorithms and data structures for flash memories
- Gal, Toledo
(Show Context)
Citation Context ...e operations sustained by the blocks. Previous and related work on flash. Most previous algorithmic work on flash memories deals with wear leveling, block-mapping and flash-targeted file systems (see =-=[15]-=- for a comprehensive survey). There exists very little work on algorithms designed to exploit the characteristics of flash memories. Wu et al. [29, 30] proposed flash-aware implementations of B-trees ... |

75 | Improved algorithms and data structures for solving graph problems in external memory
- Kumar, Schwabe
- 1996
(Show Context)
Citation Context .../Os to sort N numbers [3]. Previous results and related work. Little is known about solving SSSP on directed graphs I/Oefficiently. For undirected graphs, the algorithm of Kumar and Schwabe (KS SSSP) =-=[17]-=- performs O(n + (m/B)log(n/B)) I/Os. For dense graphs, the second term dominates; but for sparse graphs, the I/O-bound becomes O(n). The SSSP-algorithm of Meyer and Zeh (MZ SSSP) [20] extends the idea... |

69 | I/O complexity of graph algorithms
- Munagala
- 1999
(Show Context)
Citation Context ...ode street graph, our implementation was over 40 times faster. Even larger gains of over 500 can be estimated for random line graphs based on previous experimental results [6] for Munagala/Ranade-BFS =-=[22]-=-. 2 Design and Implementation Overview. Our SSSP approach is an I/O-efficient version of Dijkstra’s algorithm [14]. Dijkstra uses a priority queue Q to store all vertices of G that have not been settl... |

47 | External-memory breadth-first search with sublinear I/O
- Mehlhorn, Meyer
- 2002
(Show Context)
Citation Context ...orms O(n + (m/B)log(n/B)) I/Os. For dense graphs, the second term dominates; but for sparse graphs, the I/O-bound becomes O(n). The SSSP-algorithm of Meyer and Zeh (MZ SSSP) [20] extends the ideas of =-=[18]-=- for breadthfirst search (BFS) to graphs with edge lengths between 1 and K, leading to an O( √ nm log K/B + MST(n, m)) bound, where MST(n, m) is the cost of computing a 85 Copyright © by SIAM. Unautho... |

45 | Fast priority queues for cached memory - Sanders - 1999 |

38 | Stxxl: Standard Template Library for XXL Data Sets
- Dementiev, Kettner, et al.
- 2005
(Show Context)
Citation Context ...ing. Even then, computing SSSP for a random graph with n ≃ 106 vertices in the best case takes over 6 hours [23], which is more time than needed to do n I/Os. Furthermore, using the I/O-library STXXL =-=[12]-=-, Ajwani et al. [4, 6] studied implementations of externalmemory BFS, i.e., the unweighted version of SSSP. They managed to compute BFS on different kinds of undirected graphs featuring over 250 milli... |

36 |
Computing point-to-point shortest paths from external memory
- Goldberg, Werneck
- 2005
(Show Context)
Citation Context ... related research is algorithms for point-to-point shortest-path queries in (semi)external memory using compression and extensive precomputation in internal memory. Typical representatives are, e.g., =-=[9, 16, 25]-=-. The success of these approaches crucially depends on the special characteristics of the input graphs (in particular road networks). In contrast we are interested in I/O-efficient general purpose SSS... |

26 | An efficient B-tree layer implementation for flash-memory storage systems
- Wu, Chang, et al.
- 2007
(Show Context)
Citation Context ..., block-mapping and flash-targeted file systems (see [15] for a comprehensive survey). There exists very little work on algorithms designed to exploit the characteristics of flash memories. Wu et al. =-=[29, 30]-=- proposed flash-aware implementations of B-trees and Rtrees without file system support by explicitly handling block-mapping within the application data structures. Other works include the use of flas... |

24 | On external-memory MST, SSSP and multi-way planar graph separation - Arge, Brodal, et al. - 2000 |

23 |
An efficient R-tree implementation over flash-memory storage systems
- Wu, Chang, et al.
- 2003
(Show Context)
Citation Context ..., block-mapping and flash-targeted file systems (see [15] for a comprehensive survey). There exists very little work on algorithms designed to exploit the characteristics of flash memories. Wu et al. =-=[29, 30]-=- proposed flash-aware implementations of B-trees and Rtrees without file system support by explicitly handling block-mapping within the application data structures. Other works include the use of flas... |

18 | A computational study of external-memory BFS algorithms
- Ajwani, Dementiev, et al.
- 2006
(Show Context)
Citation Context ...uting SSSP for a random graph with n ≃ 106 vertices in the best case takes over 6 hours [23], which is more time than needed to do n I/Os. Furthermore, using the I/O-library STXXL [12], Ajwani et al. =-=[4, 6]-=- studied implementations of externalmemory BFS, i.e., the unweighted version of SSSP. They managed to compute BFS on different kinds of undirected graphs featuring over 250 million nodes and more than... |

16 |
Characterizing the performance of flash memory storage devices and its impact on algorithm design
- Ajwani, Malinger, et al.
- 2008
(Show Context)
Citation Context ...r model checking [8] or for route planning (point-to-point shortest paths) on mobile devices [16, 25]. An adaptation of our previously mentioned EMBFS implementation for flash memory was discussed in =-=[5]-=-. In there, a 32 GB Hama SSD (2.5” IDE) was used. Due to limited bandwidth of this device (less than 30 MB/s) only a combination of flash plus a traditional hard disk (in that case a 500 GB SEAGATE Ba... |

14 |
transit to constant time shortest-path queries in road networks
- In
- 2007
(Show Context)
Citation Context ... related research is algorithms for point-to-point shortest-path queries in (semi)external memory using compression and extensive precomputation in internal memory. Typical representatives are, e.g., =-=[9, 16, 25]-=-. The success of these approaches crucially depends on the special characteristics of the input graphs (in particular road networks). In contrast we are interested in I/O-efficient general purpose SSS... |

14 | Engineering an external memory minimum spanning tree algorithm
- Dementiev, Sanders, et al.
- 2004
(Show Context)
Citation Context ...orting). We apply the external-memory deterministic preprocessing implementation by Ajwani et al. [6], which in turn uses a spanning forest and connected components implementation by Dementiev et al. =-=[13]-=- with expected sort(m)⌈log n/M⌉ I/O runtime [13]. Furthermore, they use an adaptation of Sibeyn’s list ranking algorithm [26]. Both implementations are based on STXXL data structures and its sorting p... |

11 | I/O-efficient undirected shortest paths
- Meyer, Zeh
(Show Context)
Citation Context ...chwabe (KS SSSP) [17] performs O(n + (m/B)log(n/B)) I/Os. For dense graphs, the second term dominates; but for sparse graphs, the I/O-bound becomes O(n). The SSSP-algorithm of Meyer and Zeh (MZ SSSP) =-=[20]-=- extends the ideas of [18] for breadthfirst search (BFS) to graphs with edge lengths between 1 and K, leading to an O( √ nm log K/B + MST(n, m)) bound, where MST(n, m) is the cost of computing a 85 Co... |

11 |
From parallel to external list ranking
- Sibeyn
- 1997
(Show Context)
Citation Context ...panning forest and connected components implementation by Dementiev et al. [13] with expected sort(m)⌈log n/M⌉ I/O runtime [13]. Furthermore, they use an adaptation of Sibeyn’s list ranking algorithm =-=[26]-=-. Both implementations are based on STXXL data structures and its sorting primitive. For more details on the deterministic preprocessing, refer to [6]. SSSP phase. Figure 1 shows the flow-chart of the... |

10 | Improved external memory BFS implementation
- Ajwani, Meyer, et al.
- 2007
(Show Context)
Citation Context ...ld instances with non-random edge weights is actually even better than on the respective inputs with random weights. Furthermore, compared to the currently best implementation for external-memory BFS =-=[6]-=-, which in a sense constitutes a lower bound for SSSP, the running time of our approach always stayed within a factor of five, for the most difficult graph classes the difference was even less than a ... |

8 |
Eds.). Algorithms for Memory Hierarchies, volume 2625 of LNCS
- Meyer, Sanders, et al.
- 2003
(Show Context)
Citation Context ...t into the main memory and are stored on disk. The reason is that Dijkstra’s algorithm accesses the data in an unstructured fashion. Much recent work has focused on algorithms for massive graphs, see =-=[19, 27]-=- for surveys. These algorithms are analyzed in the I/O-model [3], which assumes that the computer has a main memory that can hold M vertices or edges and that the graph is stored on disk. In order to ... |

8 |
Mobile route planning
- Sanders, Schultes, et al.
- 2008
(Show Context)
Citation Context ... related research is algorithms for point-to-point shortest-path queries in (semi)external memory using compression and extensive precomputation in internal memory. Typical representatives are, e.g., =-=[9, 16, 25]-=-. The success of these approaches crucially depends on the special characteristics of the input graphs (in particular road networks). In contrast we are interested in I/O-efficient general purpose SSS... |

5 |
I/O-efficient undirected shortest paths with unbounded edge lengths
- Meyer, Zeh
- 2006
(Show Context)
Citation Context ...to an O( √ nm log K/B + MST(n, m)) bound, where MST(n, m) is the cost of computing a 85 Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.minimum spanning tree. 1 Recently =-=[21]-=-, the result was further improved to O( √ nm/B log n+MST(n, m)) I/Os, thus removing MZ SSSP’s dependence on the edge lengths in the graph. However, the latter approach is extremely involved and would ... |

3 | Can flash memory help in model checking
- Barnat, Brim, et al.
- 2008
(Show Context)
Citation Context ...tations of B-trees and Rtrees without file system support by explicitly handling block-mapping within the application data structures. Other works include the use of flash memories for model checking =-=[8]-=- or for route planning (point-to-point shortest paths) on mobile devices [16, 25]. An adaptation of our previously mentioned EMBFS implementation for flash memory was discussed in [5]. In there, a 32 ... |

2 | An empirical study of cache-oblivious priority queues and their application to the shortest path problem
- Sach, Clifford
- 2008
(Show Context)
Citation Context ...sses the difference was even less than a factor of two. We are not aware of any previous I/O-efficient implementation for the classic general SSSP in a (semi) external setting: in two recent projects =-=[10, 23]-=-, Kumar/Schwabe-like SSSP approaches on graphs of at most 6 million vertices have been tested, forcing the authors to artificially restrict the main memory size, M, to rather unrealistic 4 to 16 MByte... |

1 | Priority queues and Dijkstra’s algorithm
- Chen, Chowdhury, et al.
- 2007
(Show Context)
Citation Context ...sses the difference was even less than a factor of two. We are not aware of any previous I/O-efficient implementation for the classic general SSSP in a (semi) external setting: in two recent projects =-=[10, 23]-=-, Kumar/Schwabe-like SSSP approaches on graphs of at most 6 million vertices have been tested, forcing the authors to artificially restrict the main memory size, M, to rather unrealistic 4 to 16 MByte... |