## A Theoretician's Guide to the Experimental Analysis of Algorithms (1996)

### Cached

### Download Links

Citations: | 76 - 0 self |

### BibTeX

@MISC{Johnson96atheoretician's,

author = {David Johnson},

title = {A Theoretician's Guide to the Experimental Analysis of Algorithms},

year = {1996}

}

### Years of Citing Articles

### OpenURL

### Abstract

This paper presents an informal discussion of issues that arise when one attempts to analyze algorithms experimentally. It is based on lessons learned by the author over the course of more than a decade of experimentation, survey paper writing, refereeing, and lively discussions with other experimentalists. Although written from the perspective of a theoretical computer scientist, it is intended to be of use to researchers from all fields who want to study algorithms experimentally. It has two goals: first, to provide a useful guide to new experimentalists about how such work can best be performed and written up, and second, to challenge current researchers to think about whether their own work might be improved from a scientific point of view. With the latter purpose in mind, the author hopes that at least a few of his recommendations will be considered controversial.

### Citations

1105 | The Visual Display of Quantitative Information - Tufte - 1983 |

349 | Visualizing Data - Cleveland - 1993 |

328 | Performance of various computers using standard linear equations software,”Report CS-89-85
- Dongarra
(Show Context)
Citation Context ...ntific computing, where floating point computation and Fortran are the dominant computational paradigms, this approach is widely practiced. The benchmark codes are from the LINPACK suite described in =-=[Don00]-=-, a report that lists, for various combinations of machine, operating system, and compiler, the number of megaflops (millions of floating point operations per second) performed on each of three benchm... |

288 | Elements of Graphing Data - Cleveland - 1994 |

271 |
The travelling salesman problem: A case study in local optimization
- Johnson, McGeoch
- 2003
(Show Context)
Citation Context ...riter). In these latter roles I have had the opportunity to read widely in the experimental literature, with special emphasis on papers about the Traveling Salesman Problem (TSP), which I surveyed in =-=[JM97]-=-. I have unfortunately seen many more papers that are seriously defective than ones that set good examples. Even fairly good papers typically receive referee’s reports suggesting significant revisions... |

259 |
TSPLIB – A Traveling Salesman Problem Library
- Reinelt
- 1991
(Show Context)
Citation Context ... to experimenters: particular instances from real-world applications (or pseudo-applications) and randomly generated instances. The former are typically found in standard repositories, such as TSPLIB =-=[Rei81]-=-, but might come from private sources proprietary to the author. The latter are provided by instance generators that, given a seed and a list of instance parameters, such as size and perhaps other det... |

163 | The Travelling Salesman Problem
- Applegate, Bixby, et al.
- 2006
(Show Context)
Citation Context ...value when we were dealing with randomly generated instances. Now however, the bound can be calculated exactly for instances as large as 1,000,000 cities using the publicly available Concorde code of =-=[ABCC98]-=-. Unfortunately, not all problems have such good and computable bounds. • Report the percentage excess/improvement over some other heuristic. This appears to be the ultimate fallback option. It is rep... |

147 |
Local optimization and the traveling salesman problem
- Johnson
- 1990
(Show Context)
Citation Context ...h. It is only reproducible, however, if the lower bound can be feasibly computed or reliably approximated. When I first started using the Held-Karp lower bound as a standard of comparison for the TSP =-=[Joh90]-=-, it probably did not completely meet these criteria, as we could only compute estimates whose quality we could not guarantee. Indeed, subsequent validation experiments showed that our estimates were ... |

142 | Shortest path algorithms: Theory and experimental evaluation - Cherkassky, Goldberg, et al. - 1996 |

129 | Fast algorithms for geometric traveling salesman problems - Bentley - 1992 |

119 | The traveling-salesman problem and minimum spanning trees - Held, Karp - 1970 |

104 | Designing and reporting on computational experiments with heuristic methods - Barr, Golden, et al. - 1995 |

78 |
Near-Optimal Bin Packing Algorithms
- Johnson
- 1973
(Show Context)
Citation Context ...ery large instances. Thus studies that try to extrapolate asymptotic running time by studying times for instances of size 1000 or less can often be led astray. My own early bin packing experiments in =-=[Joh73]-=-, based on experiment with 200-item lists were also woefully misleading. I concluded that for item sizes uniformly distributed between 0 and 1, Best Fit was on average 5.6% worse than optimal. Subsequ... |

73 | Needed: An empirical science of algorithms - Hooker - 1994 |

51 | Asymptotic experimental analysis for the Held-Karp traveling salesman bound
- Johnson, McGeoch, et al.
- 1996
(Show Context)
Citation Context ...es correspond to points uniformly distributed in the unit square and distances are Euclidean, one can obtain very good estimates of what the expected value of that bound is for any number of cities N =-=[JMR96]-=-, and so could compare 12salgorithmic results to these estimates. However, one gets a more precise performance metric (with fewer samples) if one focuses on the difference between the algorithm’s tour... |

38 |
Analyzing algorithms by simulation: variance reduction techniques and simulation speedups
- McGeoch
- 1992
(Show Context)
Citation Context ...bers of instances to obtain reliable estimates of average performance. Variance reduction techniques help reduce these numbers. For an introduction to and broad-ranging survey of such techniques, see =-=[McG92]-=-. Here I will mention one that I have found useful for reducing the effects of instance variability. Suppose you are attempting to compare average performance for several approximation algorithms over... |

36 | Finite size and dimensional dependence of the Euclidean traveling salesman problem - Percus, Martin - 1996 |

35 |
Toward an experimental method for algorithm simulation
- McGeoch
- 1996
(Show Context)
Citation Context ...e the relative goodness of a solution. Here is a list of the answers to that question that I have encountered, several of which have associated reproducibility drawbacks, some of them fatal (see also =-=[McG96]-=-.) • Simply report the solution value. This certainly is reproducible in the narrow sense, but not in the broader sense that one can perform experiments on similar instances and determine whether one ... |

29 | Data Structures for Traveling Salesmen - Fredman, Johnson, et al. - 1995 |

28 | Experimental evaluation of heuristic optimization algorithms: A tutorial - Rardin, Uzsoy |

22 | Experimental Analysis of Algorithms - McGeoch - 2001 |

15 |
Nearoptimal solutions to very large traveling salesman problems (for experimental results from this work see also Ref
- Johnson, Bentley, et al.
(Show Context)
Citation Context ...ny TSP heuristics, the relative quality of the tours found by two heuristics on 100-city instances can be far different from the relative quality found for 10,000-city instances (in either direction) =-=[JBMR03]-=-. If one is interested in asymptotics, one should study as large instances as possible. In addition, if one is interested in running time, it pays to study it in as much detail as possible. Suggestion... |

14 |
An Experimental Study of Bin Packing
- BENTLEY, JOHNSON, et al.
- 1983
(Show Context)
Citation Context ...woefully misleading. I concluded that for item sizes uniformly distributed between 0 and 1, Best Fit was on average 5.6% worse than optimal. Subsequent experiments with lists of 100,000 items or more =-=[BJLM83]-=- implied that Best Fit was in fact asymptotically optimal, as has now been proved [Sho86]. Similarly, for many TSP heuristics, the relative quality of the tours found by two heuristics on 100-city ins... |

12 |
The average case analysis of some on-line algorithms for bin packing
- Shor
- 1986
(Show Context)
Citation Context ..., Best Fit was on average 5.6% worse than optimal. Subsequent experiments with lists of 100,000 items or more [BJLM83] implied that Best Fit was in fact asymptotically optimal, as has now been proved =-=[Sho86]-=-. Similarly, for many TSP heuristics, the relative quality of the tours found by two heuristics on 100-city instances can be far different from the relative quality found for 10,000-city instances (in... |

9 | Statistical analysis of computational tests of algorithms and heuristics - Coffin, Saltzman - 2000 |

8 | How to present a paper on experimental work with algorithms - McGeoch, Moret - 1999 |

6 | Tools for experiments on algorithms - Bentley - 1990 |