## Algorithms and Experiments: The New (and Old) Methodology (2001)

Venue: | J. Univ. Comput. Sci |

Citations: | 9 - 4 self |

### BibTeX

@ARTICLE{Moret01algorithmsand,

author = {Bernard M. E. Moret and Henry D. Shapiro},

title = {Algorithms and Experiments: The New (and Old) Methodology},

journal = {J. Univ. Comput. Sci},

year = {2001},

volume = {7},

pages = {2001}

}

### OpenURL

### Abstract

The last twenty years have seen enormous progress in the design of algorithms, but little of it has been put into practice. Because many recently developed algorithms are hard to characterize theoretically and have large running-time coefficients, the gap between theory and practice has widened over these years. Experimentation is indispensable in the assessment of heuristics for hard problems, in the characterization of asymptotic behavior of complex algorithms, and in the comparison of competing designs for tractable problems. Implementation, although perhaps not rigorous experimentation, was characteristic of early work in algorithms and data structures. Donald Knuth has throughout insisted on testing every algorithm and conducting analyses that can predict behavior on actual data; more recently, Jon Bentley has vividly illustrated the difficulty of implementation and the value of testing. Numerical analysts have long understood the need for standardized test suites to ensure robustness, precision and efficiency of numerical libraries. It is only recently, however, that the algorithms community has shown signs of returning to implementation and testing as an integral part of algorithm development. The emerging disciplines of experimental algorithmics and algorithm engineering have revived and are extending many of the approaches used by computing pioneers such as Floyd and Knuth and are placing on a formal basis many of Bentley's observations. We reflect on these issues, looking back at the last thirty years of algorithm development and forward to new challenges: designing cache-aware algorithms, algorithms for mixed models of computation, algorithms for external memory, and algorithms for scientific research.

### Citations

635 | LEDA: a platform for combinatorial and geometric computing - Mehlhorn, aher - 1995 |

532 |
Network Flows
- Ahuja, Magnanti, et al.
- 1993
(Show Context)
Citation Context ...ogramming; for over thirty years, it has been known that the worst-case behavior of this method is exponential, but also that its practical running time is typically bounded by a lowdegree polynomial =-=[AMO93]-=-. (Indeed, in some of its newer versions, its running time is competitive with that of the modern interior-point methods [BFG + 00].) – Even in the absence of any of these problems, deriving tight asy... |

512 |
Optimization by Simulated Annealing: An Experimental Evaluation
- Johnson, Aragon, et al.
- 1991
(Show Context)
Citation Context ...c number determination or in its much easier (yet still challenging) version of planar graph coloring, has seen much work as well; the second study of simulated annealing conducted by Johnson’s group =-=[JAMS89]-=- discussed many facets of the problem, while Morgenstern and Shapiro [MS91b] provided a detailed study of algorithms to color planar graphs. Understanding how a heuristic works to cut down on computat... |

320 | External Memory Algorithms and Data Structures: Dealing with Massive Data
- Vitter
- 1981
(Show Context)
Citation Context ... addressing have been reported to gain speedups of as much as a factor of 10 [MWB + 01]. The related, and much better studied, model of out-of-core computing, as pioneered by Vitter and his coworkers =-=[Vit01]-=-, has inspired new work in cache-aware and cache-independent algorithm design. Once again, though, this trend was pioneered over 40 years ago, when programmers had to work with very limited memory and... |

157 |
The traveling salesman: computational solutions for TSP applications
- Reinelt
- 1994
(Show Context)
Citation Context ...ng Salesperson problem has seen large numbers of experimental studies (including the well publicized study of Jon Bentley [Ben90]), made possible in part by the development of a library of test cases =-=[Rei94]-=-. Graph coloring, whether in its NP-hard version of chromatic number determination or in its much easier (yet still challenging) version of planar graph coloring, has seen much work as well; the secon... |

151 | On implementing the pushrelabel method for the maximum flow problem - Cherkassky, Goldberg - 1997 |

150 |
The stanford graphbase: A platform for combinatorial computing
- Knuth
- 1993
(Show Context)
Citation Context ...so need structural measures of various types (number of iterations; number of calls to a crucial subroutine; etc.), if only to serve as a scale for determining such things as convergence rates. Knuth =-=[Knu93]-=- has advocated the use of mems, or memory references, as a structural substitute for running time. Other authors have used the number of comparisons, the number of data moves (both classical measures ... |

142 | Shortest path algorithms: Theory and experimental evaluation
- Cherkassky, Goldberg, et al.
- 1996
(Show Context)
Citation Context ...tational Challenges, the first of which [JM93] focused on network flow and shortest path algorithms, indirectly giving rise to several modern, thorough studies, by Cherkassky et al. on shortest paths =-=[CGR96]-=-, by Cherkassky et al. on the implementation of the push-relabel method for matching and network flows [CGM + 98, CG97], and by Goldberg and Tsioutsiouliklis on cut trees [GT01]. The DIMACS Computatio... |

76 | A theoretician’s guide to the experimental analysis of algorithms
- Johnson
- 2002
(Show Context)
Citation Context ... caching effects. Recent studies by LaMarca and Ladner [LL96, LL97] have quantified many aspects of caching and offer suggestions on how to work around (or take advantage of) caching effects. Johnson =-=[Joh01]-=- offers a detailed list of the various problems he has observed in experimental studies, particularly those dealing with heuristics for hard optimization problems. Most of these pitfalls can be avoide... |

68 | The influence of caches on the performance of heaps
- LaMarca, Ladner
- 1996
(Show Context)
Citation Context ...ence to linear probing, whereas experimental data clearly indicates that linear probing is the faster method, thanks to its good locality (see [BMQ98]). Pioneering studies by Ladner and his coworkers =-=[LL96, LL97]-=- established that cache optimization was feasible, algorithmically interesting, and worthwhile, even for such old friends as sorting algorithms [ACVW01, LL97, RR99, XZK00] and priority queues [LL96, S... |

55 |
Writing Efficient Programs
- Bentley
- 1984
(Show Context)
Citation Context ... Algorithms and Experiments ... Throughout the last 25 years, Jon Bentley has demonstrated the value of implementation and testing of algorithms, beginning with his text on writing efficient programs =-=[Ben82]-=- and continuing with his invaluable Programming Pearls columns in Communications of the ACM, now collected in a new volume [Ben99], and his Software Explorations columns in the UNIX Review. David John... |

55 | A new implementation and detailed study of breakpoint analysis - Moret, Wyman, et al. - 2001 |

50 |
Set Covering Algorithms Using Cutting Planes, Heuristics and Subgradient Optimization: A
- Balas, Ho
- 1980
(Show Context)
Citation Context ...oblems (see [AMO93], Chapter 18). Inspired in part by experimental studies of integer-programming algorithms for combinatorial optimization, such as algorithms for the set-covering problem—see, e.g., =-=[BH80]-=-, we conducted a large-scale combinatorial study on the minimum test set problem [MS85], one of the first such studies in Computer Science to include both real-world and generated instances. Other lar... |

45 | Fast priority queues for cached memory
- Sanders
- 1999
(Show Context)
Citation Context ...6, LL97] established that cache optimization was feasible, algorithmically interesting, and worthwhile, even for such old friends as sorting algorithms [ACVW01, LL97, RR99, XZK00] and priority queues =-=[LL96, San00]-=-; indeed, even matrix multiplication, which has been optimized in numerical libraries for over 40 years (including optimizations for paging behavior), is amenable to such techniques [ERS90]. Ad hoc re... |

40 | C.: ‘Experimental Study of Minimum Cut Algorithms
- CHEKURI, GOLDBERG, et al.
- 1997
(Show Context)
Citation Context ...f results in this area. The ACM/SIAM Symposium on Discrete Algorithms (SODA) has included a few such studies in each of its dozen events to date, such as the study of cut algorithms by Chekuri et al. =-=[CaDRKLS97]-=-. The Traveling Salesperson problem has seen large numbers of experimental studies (including the well publicized study of Jon Bentley [Ben90]), made possible in part by the development of a library o... |

40 | An empirical assessment of algorithms for constructing a minimum spanning tree
- Moret, Shapiro
- 1994
(Show Context)
Citation Context ...hat, in particular, β(n, n) is just log ∗ n. This bound is much better for dense graphs than that of Prim’s algorithm, which is O(|E| log |V |) when implemented with binary heaps, but experimentation =-=[MS94]-=- verifies that the crossover point occurs for dense graphs with well over a million vertices and thus hundreds of millions of edges—beyond the size of any reasonable data set. – In another facet of th... |

31 |
Pairing heaps: Experiments and analysis
- Stasko, Vitter
- 1987
(Show Context)
Citation Context ...ting in 1983 [Ben83]), then with Jones’ comparison of data structures for priority queues [Jon86] and Stasko and Vitter’s combination of analytical and experimental work in the study of pairing heaps =-=[SV87]-=-. An early experimental study on a large scale was that of Moret and Shapiro on sorting algorithms [MS91a] (Chapter 8), itself inspired by the work ofsMoret B.M.E., Shapiro H.D.: Algorithms and Experi... |

29 | Augment or push: A computational study of bipartite matching and unitcapacity flow algorithms - Cherkassky, Goldberg, et al. - 1998 |

28 | The LEDA platform for combinatorial and geometric computing - Melhorn, Näher - 1999 |

26 |
On minimizing a set of tests
- Moret, Shapiro
- 1985
(Show Context)
Citation Context ...rogramming algorithms for combinatorial optimization, such as algorithms for the set-covering problem—see, e.g., [BH80], we conducted a large-scale combinatorial study on the minimum test set problem =-=[MS85]-=-, one of the first such studies in Computer Science to include both real-world and generated instances. Other large-scale studies were published in the same time frame, most notably the classic and ex... |

25 | Improving memory performance of sorting algorithms - Xiao, Zhang, et al. - 1981 |

24 |
SEYMOUR,Graph minors—a survey
- ROBERTSON, P
- 1985
(Show Context)
Citation Context ...ctical implementation from running to completion, even if the growth rate is quite reasonable. An extreme example of this problem is provided by the theory of graph minors: Robertson and Seymour (see =-=[RS85]-=-) gave a cubic-time algorithm to determine whether a given graph is a minor of another, but the proportionality constants are gigantic—a recent estimate was on the order of 10 150 [Fel99]—and have not... |

19 | Efficient sorting using registers and caches - Arge, Chase, et al. - 2000 |

19 | Analyzing Cache Effects in Distribution Sorting - Rahman, Raman |

16 |
Cliques, coloring and satisfiability second DIMACS implementation challenge
- Johnson, Trick
- 1996
(Show Context)
Citation Context ...of simulated annealing by David Johnson’s group [JAMS89, JAMS91], which, among other things, demonstrated the value of a varied collection of test instances. The Second DIMACS Computational Challenge =-=[JT96]-=- was devoted to satisfiability, graph coloring, and clique problems and thus saw a large collection of results in this area. The ACM/SIAM Symposium on Discrete Algorithms (SODA) has included a few suc... |

14 |
An empirical comparison of priority queue and event set implementations
- Jones
- 1986
(Show Context)
Citation Context ...rs. More recent and comprehensive work began with Bentley’s many contributions in his Programming Pearls (starting in 1983 [Ben83]), then with Jones’ comparison of data structures for priority queues =-=[Jon86]-=- and Stasko and Vitter’s combination of analytical and experimental work in the study of pairing heaps [SV87]. An early experimental study on a large scale was that of Moret and Shapiro on sorting alg... |

14 |
D.: Heuristics for rapidly four-coloring large planar graphs. Algorithmica 6
- MORGENSTERN, SHAPIRO
- 1991
(Show Context)
Citation Context ...of planar graph coloring, has seen much work as well; the second study of simulated annealing conducted by Johnson’s group [JAMS89] discussed many facets of the problem, while Morgenstern and Shapiro =-=[MS91b]-=- provided a detailed study of algorithms to color planar graphs. Understanding how a heuristic works to cut down on computational time is generally too difficult to achieve through formal derivations;... |

12 | Graph and hashing algorithms for modern architectures: Design and performance
- Black, Martel, et al.
- 1998
(Show Context)
Citation Context ... structures still advocate using double hashing in preference to linear probing, whereas experimental data clearly indicates that linear probing is the faster method, thanks to its good locality (see =-=[BMQ98]-=-). Pioneering studies by Ladner and his coworkers [LL96, LL97] established that cache optimization was feasible, algorithmically interesting, and worthwhile, even for such old friends as sorting algor... |

9 |
Private communication
- Fellows
- 2011
(Show Context)
Citation Context ... Seymour (see [RS85]) gave a cubic-time algorithm to determine whether a given graph is a minor of another, but the proportionality constants are gigantic—a recent estimate was on the order of 10 150 =-=[Fel99]-=-—and have not been substantially lowered yet, making the algorithm entirely impractical.sMoret B.M.E., Shapiro H.D.: Algorithms and Experiments ... 437 – The worst-case behavior may be restricted to a... |

8 | MIP: Theory and practice—Closing the gap - Bixby, Fenelon, et al. - 2000 |

8 |
An empirical performance comparison of some variations of the kd tree and bd tree
- Dandamudi, Sorenson
- 1985
(Show Context)
Citation Context ... data structures, but common in Operations Research; early (1960s) work in data structures typically included code and examples, but no systematic study. Scattered articles during the 70s (see, e.g., =-=[DS85]-=-) kept a low level of experimentation active, but did not attempt to provide methodological pointers. More recent and comprehensive work began with Bentley’s many contributions in his Programming Pear... |

7 | Matrix multiplication: a case study of enhanced data cache utilization
- Eiron, Rodeh
- 1999
(Show Context)
Citation Context ...eues [LL96, San00]; indeed, even matrix multiplication, which has been optimized in numerical libraries for over 40 years (including optimizations for paging behavior), is amenable to such techniques =-=[ERS90]-=-. Ad hoc reduction in memory usage and improvement in patterns of memory addressing have been reported to gain speedups of as much as a factor of 10 [MWB + 01]. The related, and much better studied, m... |

5 |
Runtime prediction of real programs on real machines
- Finkler, Mehlhorn
- 1997
(Show Context)
Citation Context ...nt or simplification of the algorithm. Experimentation can also enable us to determine the actual constants in the running time analysis; determining such constants beforehand is quite difficult (see =-=[FM97]-=-s440 for a possible methodology), but a simple regression analysis from the data can gives us quite accurate values. Experimental studies naturally include caching effects, whereas adding those into t... |

2 |
Cut tree algorithms: An experimental study
- Goldberg, Tsioutsiouliklis
(Show Context)
Citation Context ...al. on shortest paths [CGR96], by Cherkassky et al. on the implementation of the push-relabel method for matching and network flows [CGM + 98, CG97], and by Goldberg and Tsioutsiouliklis on cut trees =-=[GT01]-=-. The DIMACS Computational Challenges (the fifth, in 1996, focused on another tractable problem, priority queues and point location data structures) have served to highlight work in the area, to estab... |

1 | H.D.: Algorithms and Experiments - Moret, Shapiro |

1 |
McGeoch (eds.), Network flows and matching
- Johnson, C
- 1993
(Show Context)
Citation Context ...t of the same authors on algorithms for constructing minimum spanning trees [MS94]. In 1991, David Johnson and others initiated the very successful DIMACS Computational Challenges, the first of which =-=[JM93]-=- focused on network flow and shortest path algorithms, indirectly giving rise to several modern, thorough studies, by Cherkassky et al. on shortest paths [CGR96], by Cherkassky et al. on the implement... |

1 |
McGeoch, Analysis of algorithms by simulation: variance reduction techniques and simulation speedups
- C
- 1992
(Show Context)
Citation Context ...tical significance may be quite difficult in the large state spaces we commonly use, various techniques to make the best use of available experiments should be applied (see McGeoch’s excellent survey =-=[McG92]-=- for a discussion of several such methods). Cross-checking the measurements with any available theoretical results, especially those that attempt to predict the actual running time (such as the “equiv... |