## Testing Heuristics: We Have It All Wrong (1995)

Venue: | Journal of Heuristics |

Citations: | 119 - 2 self |

### BibTeX

@ARTICLE{Hooker95testingheuristics:,

author = {J.N. Hooker},

title = {Testing Heuristics: We Have It All Wrong},

journal = {Journal of Heuristics},

year = {1995},

volume = {1},

pages = {33--42}

}

### Years of Citing Articles

### OpenURL

### Abstract

The competitive nature of most algorithmic experimentation is a source of problems that are all too familiar to the research community. It is hard to make fair comparisons between algorithms and to assemble realistic test problems. Competitive testing tells us which algorithm is faster but not why. Because it requires polished code, it consumes time and energy that could be spent doing more experiments. This paper argues that a more scientific approach of controlled experimentation, similar to that used in other empirical sciences, avoids or alleviates these problems. We have confused research and development; competitive testing is suited only for the latter. Most experimental studies of heuristic algorithms resemble track meets more than scientific endeavors. Typically an investigator has a bright idea for a new algorithm and wants to show that it works better, in some sense, than known algorithms. This requires computational tests, perhaps on a standard set of benchmark p...

### Citations

578 | Where the really hard problems are
- Cheeseman, Kanefsky, et al.
- 1991
(Show Context)
Citation Context ... on the satisfiability problem. Several investigators have noted that random problems tend to be hard when the ratio of the number of formulas to the number of variables is close to a critical value (=-=[5, 6, 7, 10, 12, 15]-=-, etc.). But this observation scarcely implies that one can predict the difficulty of a given problem by computing the ratio of formulas to variables. Random problems with a given ratio may differ alo... |

217 | and easy distributions of SAT problems
- Mitchell, Selman, et al.
- 1992
(Show Context)
Citation Context ... on the satisfiability problem. Several investigators have noted that random problems tend to be hard when the ratio of the number of formulas to the number of variables is close to a critical value (=-=[5, 6, 7, 10, 12, 15]-=-, etc.). But this observation scarcely implies that one can predict the difficulty of a given problem by computing the ratio of formulas to variables. Random problems with a given ratio may differ alo... |

199 | Experimental results on the cross-over point in satisfiability problems
- Crawford, Auton
- 1993
(Show Context)
Citation Context ... on the satisfiability problem. Several investigators have noted that random problems tend to be hard when the ratio of the number of formulas to the number of variables is close to a critical value (=-=[5, 6, 7, 10, 12, 15]-=-, etc.). But this observation scarcely implies that one can predict the difficulty of a given problem by computing the ratio of formulas to variables. Random problems with a given ratio may differ alo... |

121 |
Interior point methods for linear programming problems: Computational state of the art
- Lustig, Marsten, et al.
- 1994
(Show Context)
Citation Context ...cation and funding decisions are rather sensitive to initial computational results, and the technology of commercial codes can discourage the development of new approaches. Lustig, Marsten and Shanno =-=[13]-=- suggest, for example, that if interior point methods had come along a couple of years later than they did---after the recent upswing in simplex technology now embodied in such codes as CPLEX---they m... |

78 | Branching rules for satisfiability
- Hooker, Vinay
- 1995
(Show Context)
Citation Context ...hat are the same except for the branching rule, as was done to a limited extent in [8]. This raises the further question, however, as to why some branching rules are better than others. A later study =-=[11]-=- considered two hypotheses: a) that better branching rules try to maximize the probability that subproblems are satisfiable, and b) that better branching rules simplify the subproblems as much as poss... |

73 | Needed: An empirical science of algorithms
- Hooker
- 1994
(Show Context)
Citation Context ...consequences that can be put to the test. I will give this sort of experimentation the deliberately honorific name `scientific testing' to distinguish it from competitive testing. I discuss elsewhere =-=[9]-=- how empirical models might be constructed and defend them as a viable and necessary alternative to a purely deductive science of algorithms. My main object in this paper is to show that scientific te... |

64 | The TSP Phase Transition
- Gent, Walsh
- 1996
(Show Context)
Citation Context |

46 |
Branch-and-cut solution of inference problems in propositional logic
- Hooker, Fedjki
- 1990
(Show Context)
Citation Context |

46 | Evidence for a Satisfiability Threshold for Random 3CNF Formulas
- Larrabee, Tsuji
- 1992
(Show Context)
Citation Context |

35 |
ªToward an Experimental Method for Algorithm Simulation,º
- McGeoch
- 1996
(Show Context)
Citation Context ...e, has a significant effect on performance when the remaining factors are held constant at any given set of levels (k 2 ; : : : ; k n ). It is also possible to measure interactions among factors. See =-=[11, 14]-=- for details. This scheme requires random generation of problems, but it bears scant resemblance to traditional random generation. The goal is not to generate realistic problems, which random generati... |

27 |
Recognizing disguised nr(1) instances of the satisfiability problem
- Aspvall
- 1980
(Show Context)
Citation Context ...ype of experimental design (a "factorial design") begins with a list of n factors that could affect performance---perhaps problem size, density, existence of a solution, closeness to `renama=-=ble Horn' [1, 3, 4]-=-, etc. Each factor i has several levels k i = 1; : : : ; m i , corresponding to different problem sizes, densities, etc. The levels need not correspond to values on a scale, as for instance if the fac... |

26 |
A Computational Study of Satisfiability Algorithms for Propositional Logic
- HARCHE, HOOKER, et al.
- 1993
(Show Context)
Citation Context ... have the form shown; i.e., they consist of variables or their negations joined by or's.) At the moment some of the most effective algorithms for checking satisfiability use a simple branching scheme =-=[2, 8, 16]-=-. A variable x j is set to true and then to false to create subproblems at two successor nodes of the root node of a search tree. When the truth value of x j is fixed, the problem can normally be simp... |

13 |
On renamable Horn and generalized Horn functions
- Chandru, Coulard, et al.
- 1990
(Show Context)
Citation Context ...ype of experimental design (a "factorial design") begins with a list of n factors that could affect performance---perhaps problem size, density, existence of a solution, closeness to `renama=-=ble Horn' [1, 3, 4]-=-, etc. Each factor i has several levels k i = 1; : : : ; m i , corresponding to different problem sizes, densities, etc. The levels need not correspond to values on a scale, as for instance if the fac... |

4 |
Report on a SAT competition
- Bohm
- 1992
(Show Context)
Citation Context ... have the form shown; i.e., they consist of variables or their negations joined by or's.) At the moment some of the most effective algorithms for checking satisfiability use a simple branching scheme =-=[2, 8, 16]-=-. A variable x j is set to true and then to false to create subproblems at two successor nodes of the root node of a search tree. When the truth value of x j is fixed, the problem can normally be simp... |

2 |
Detecting extended Horn structure
- Chandru, Hooker
- 1992
(Show Context)
Citation Context ...ype of experimental design (a "factorial design") begins with a list of n factors that could affect performance---perhaps problem size, density, existence of a solution, closeness to `renama=-=ble Horn' [1, 3, 4]-=-, etc. Each factor i has several levels k i = 1; : : : ; m i , corresponding to different problem sizes, densities, etc. The levels need not correspond to values on a scale, as for instance if the fac... |