## Towards A Discipline Of Experimental Algorithmics

Citations: | 36 - 8 self |

### BibTeX

@MISC{Moret_towardsa,

author = {Bernard M. E. Moret},

title = {Towards A Discipline Of Experimental Algorithmics},

year = {}

}

### Years of Citing Articles

### OpenURL

### Abstract

The last 20 years have seen enormous progress in the design of algorithms, but very little of it has been put into practice, even within academia; indeed, the gap between theory and practice has continuously widened over these years. Moreover, many of the recently developed algorithms are very hard to characterize theoretically and, as initially described, suffer from large running-time coefficients. Thus the algorithms and data structures community needs to return to implementation as the standard of value; we call such an approach Experimental Algorithmics. Experimental Algorithmics studies algorithms and data structures by joining experimental studies with the more traditional theoretical analyses. Experimentation with algorithms and data structures is proving indispensable in the assessment of heuristics for hard problems, in the design of test cases, in the characterization of asymptotic behavior of complex algorithms, in the comparison of competing designs for tractabl...

### Citations

637 | LEDA: A platform for combinatorial and geometric computing
- Mehlhorn, Naher
- 1995
(Show Context)
Citation Context ...ginering and Experiments 1 ) have emphasized the need to develop libraries of robust, well-tested implementations of the basic discrete and combinatorial algorithms, a task that only the LEDA project =-=[17]-=- has successfully undertaken to date. 4. Worthwhile Problems In view of the preceding, what should researchers in the area be working on? We propose below a partial list and brie y discuss the reasons... |

533 |
Network Flows
- Ahuja, Magnanti, et al.
- 1993
(Show Context)
Citation Context ...ar programming; for over thirty years, it has been known that the worst-case behavior of this method is exponential and also that its practical running time appears bounded by a low-degree polynomial =-=[1-=-]. The constants hidden in the asymptotic analysis may prevent any practical implementation from running to completion, even if the growth rate is quite reasonable. An extreme example of this problem... |

510 |
Optimization by simulated annealing: an experimental evaluation
- Johnson, Aragon, et al.
- 1989
(Show Context)
Citation Context ...on the minimum test set problem [19], but other large-scale studies were published in the same time frame, most notably the classic and exemplary study of simulated annealing by David Johnson's group =-=[8, 9-=-]. The Second DIMACS Computational Challenge [11] was devoted to satisability, graph coloring, and clique problems and thus saw a large collection of results in this area. Proceedings of the ACM/SIAM ... |

157 |
The traveling salesman: computational solutions for TSP applications
- Reinelt
- 1994
(Show Context)
Citation Context ...veling Salesperson problem has seen large numbers of experimental studies (including the well publicized study of Jon Bentley [2]), made possible in part by the development of a library of test cases =-=[23]-=-. Graph coloring, whether in its NP-hard version of chromatic number determination or in its much easier (yet still challenging) version of planar graph coloring, has seen much work as well; the secon... |

150 | On implementing push-relabel method for the maximum flow problem
- Cherkassy, Goldberg
- 1995
(Show Context)
Citation Context ...lgorithms, indirectly giving rise to two modern, thorough studies, by Cherkassky et al. on shortest paths [4] and by Cherkassky et al. on the implementation of the push-relabel method for networksows =-=[5]-=-. The DIMACS Computational Challenges (thesfth, in 1996, focused on another tractable problem, priority queues and point location data structures) have served to highlight work in the area, to establi... |

149 |
The Stanford GraphBase: A Platform for Combinatorial Computing
- Knuth
- 1994
(Show Context)
Citation Context ...so need structural measures of various types (number of iterations; number of calls to a crucial subroutine; etc.), if only to serve as a scale for determining such things as convergence rates. Knuth =-=[13]-=- has advocated the use of mems, or memory references, as a structural substitute for running time. Other authors have used the number of comparisons, the number of data moves (both classical measures ... |

142 | Shortest path algorithms: Theory and experimental evaluation
- Cherkassky, Goldberg, et al.
- 1996
(Show Context)
Citation Context ...MACS Computational Challenges, thesrst of which [10] focused on networksow and shortest path algorithms, indirectly giving rise to two modern, thorough studies, by Cherkassky et al. on shortest paths =-=[4]-=- and by Cherkassky et al. on the implementation of the push-relabel method for networksows [5]. The DIMACS Computational Challenges (thesfth, in 1996, focused on another tractable problem, priority qu... |

112 | The influence of caches on the performance of sorting
- LaMarca, Ladner
- 1997
(Show Context)
Citation Context ... algorithms. For instance, in our study of MST algorithms, we observed 3:1 ratios of running time depending on the order in which the adjacency lists were stored. Recent studies by LaMarca and Ladner =-=[14,-=- 15] have quantied many aspects of caching and oers suggestions on how to work around caching eects. Other typical pitfalls that arise in experimental work with algorithms include Uninteresting work:... |

74 |
Relaxed Heaps: An Alternative to Fibonacci Heaps with Applications to Parallel Computation
- Driscoll, Gabow, et al.
- 1988
(Show Context)
Citation Context ... of each algorithm. Early in the implementation phase, we realized that Fibonacci heaps and relaxed heaps were not competitive. We then took a suggestion made in the original paper of Driscoll et al. =-=[6]-=- for implementing relaxed heaps: to group nodes into larger units so that changes in key would most often be resolved within a unit and not TOWARDS A DISCIPLINE OF EXPERIMENTAL ALGORITHMICS 11 require... |

68 | The Influence of Caches on the Performance of Heaps
- LaMarca, Ladner
- 1996
(Show Context)
Citation Context ... algorithms. For instance, in our study of MST algorithms, we observed 3:1 ratios of running time depending on the order in which the adjacency lists were stored. Recent studies by LaMarca and Ladner =-=[14,-=- 15] have quantied many aspects of caching and oers suggestions on how to work around caching eects. Other typical pitfalls that arise in experimental work with algorithms include Uninteresting work:... |

40 | C.: â€˜Experimental Study of Minimum Cut Algorithms
- CHEKURI, GOLDBERG, et al.
- 1997
(Show Context)
Citation Context ...of the ACM/SIAM Symposium on Discrete Algorithms (SODA) have included a few such studies for each of the last few years; an outstanding recent example is the study of cut algorithms by Chekuri et al. =-=[3]-=-. The Traveling Salesperson problem has seen large numbers of experimental studies (including the well publicized study of Jon Bentley [2]), made possible in part by the development of a library of te... |

40 | An empirical assessment of algorithms for constructing a minimum spanning tree
- Moret, Shapiro
- 1994
(Show Context)
Citation Context ...en by minfi j log (i) n m=ng, so that, in particular,s(n; n) is just log n. This bound is much better for dense graphs than that of Prim's algorithm, which is O(jEj log jV j), but experimentation [2=-=-=-0] veries that the crossover point occurs for dense graphs with well over a million vertices|beyond the size of any reasonable data set. The worst-case behavior may be restricted to a very small subs... |

38 |
Analyzing algorithms by simulation: variance reduction techniques and simulation speedups
- McGeoch
- 1992
(Show Context)
Citation Context ...atistical signicance may be quite dicult in the large state spaces we commonly use, various techniques to make the best use of available experiments should be applied (see McGeoch's excellent survey [=-=16]-=- for a discussion of several such methods). Cross-checking the measurements with any available theoretical results, especially those that attempt to predict the actual running time (such as the \equiv... |

30 |
I.: An experimental study of the basis for graph drawing algorithms
- PURCHASE, COHEN, et al.
- 1997
(Show Context)
Citation Context ...drawing, there would be little reason to spend years developing algorithms that draw graphs with a minimum number of crossings, for instance, if we did not have empirical evidence (see, for instance, =-=[22]-=-) that such drawings are more easily interpreted by humans than drawings with large numbers of crossings. 3. Modes of Empirical Assessment We can classify modes of empirical assessment into a number o... |

26 |
On minimizing a set of tests
- Moret, Shapiro
- 1985
(Show Context)
Citation Context ...th integer programming problems (see [1], Chapter 18). Thesrst large-scale combinatorial study to include both real-world and generated instances was probably our work on the minimum test set problem =-=[19]-=-, but other large-scale studies were published in the same time frame, most notably the classic and exemplary study of simulated annealing by David Johnson's group [8, 9]. The Second DIMACS Computatio... |

14 |
An empirical comparison of priority queue and event set implementations
- Jones
- 1986
(Show Context)
Citation Context ...y (1960s) work in data structures typically included code and examples, but no systematic study. More recent and comprehensive work began with Jones' comparison of data structures for priority queues =-=[12]-=- and Stasko and Vitter's combination of analytical and experimental work in the study of pairing heaps [25]. Thesrst experimental study on a large scale was that of Moret and Shapiro on sorting algori... |

14 |
D.: Heuristics for rapidly four-coloring large planar graphs. Algorithmica 6
- MORGENSTERN, SHAPIRO
- 1991
(Show Context)
Citation Context ...sion of planar graph coloring, has seen much work as well; the second study of simulated annealing conducted by Johnson's group [8] discussed many facets of the problem, while Morgenstern and Shapiro =-=[21]-=- provided a detailed study of algorithms to color planar graphs. 3.2. Assessment of Competing Algorithms and Data Structures for Tractable Problems. The goal here is to measure the actual performance ... |

11 |
Experiments on Geometric Travelling Salesman Heuristics
- Bentley
- 1990
(Show Context)
Citation Context ...cent example is the study of cut algorithms by Chekuri et al. [3]. The Traveling Salesperson problem has seen large numbers of experimental studies (including the well publicized study of Jon Bentley =-=[2]-=-), made possible in part by the development of a library of test cases [23]. Graph coloring, whether in its NP-hard version of chromatic number determination or in its much easier (yet still challengi... |

5 |
Runtime prediction of real programs on real machines
- Finkler, Mehlhorn
- 1997
(Show Context)
Citation Context ...nement or simplication of the algorithm. Experimentation can also enable us to determine the actual constants in the running time analysis; determining such constants beforehand is quite dicult (see [=-=7-=-] for a possible methodology), but a simple regression analysis from the data can gives us quite accurate values. 4.3. Supporting and rening conjectures. Any theoretician knows the pangs of committing... |

2 |
Algorithms from P to NP, Volume I
- Moret, Shapiro
- 1991
(Show Context)
Citation Context ...Stasko and Vitter's combination of analytical and experimental work in the study of pairing heaps [25]. Thesrst experimental study on a large scale was that of Moret and Shapiro on sorting algorithms =-=[18]-=- (Chapter 8), followed by that of the same authors on algorithms for constructing 6 BERNARD M.E. MORET minimum spanning trees [18, 20]. In 1991, Johnson and others initiated the very successful DIMACS... |