The competitive nature of most algorithmic experimentation is a source of problems that are all too familiar to the research community. It is hard to make fair comparisons between algorithms and to assemble realistic test problems. Competitive testing tells us which algorithm is faster but not why. Because it requires polished code, it consumes time and energy that could be spent doing more experiments. This paper argues that a more scientific approach of controlled experimentation, similar to that used in other empirical sciences, avoids or alleviates these problems. We have confused research and development; competitive testing is suited only for the latter. Most experimental studies of heuristic algorithms resemble track meets more than scientific endeavors. Typically an investigator has a bright idea for a new algorithm and wants to show that it works better, in some sense, than known algorithms. This requires computational tests, perhaps on a standard set of benchmark p...

Computer scientists and practitioners defend the lack of experimentation with a wide range of arguments. Some arguments suggest that experimentation may be inappropriate, too difficult, useless, and even harmful. This article discusses several such arguments to illustrate the importance of experimentation for computer science. This is a preprint of an article with the same title that appeared in IEEE Computer, 31(5), May 1998, 32--40. Keywords: Empiricism, experiments, laboratory, scientific method. 1 Is computer science an experimental science? Do computer scientists need to experiment at all? Only if we answer "yes" does it make sense to ask whether there is enough of it. In his Allen Newell Award Lecture, Fred Brooks suggests that computer science is "not a science, but a synthetic, an engineering discipline"[2]. In an engineering field, testing theories by experiments would be misplaced. Brooks and others seem troubled by the fact that the phenomena studied by computer scientists ...

A survey of 400 recent research articles suggests that computer scientists publish relatively few papers with experimentally validated results. The survey includes complete volumes of several refereed computer science journals, a conference, and 50 titles drawn at random from all articles published by ACM in 1993. The journals of Optical Engineering (OE) and Neural Computation (NC) were used for comparison. Of the papers in the random sample that would require experimental validation, 40 % have none at all. In journals related to software engineering, this fraction is 50%. In comparison, the fraction of papers lacking quantitative evaluation in OE and NC is only 15 % and 12%, respectively. Conversely, the fraction of papers that devote one fifth or more of their space to experi-mental validation is almost 70 % for OE and NC, while it is a mere 30 % for the computer science (CS) random sample and 20 % for software engineering. The low ratio of validated results appears to be a serious weak-ness in computer science research. This weakness should be rectified for the long-term health of the field.

Recent experience suggests that branching algorithms are among the most attractive for solving propositional satisfiability problems. A key factor in their success is the rule they use to decide on which variable to branch next. We attempt to explain and improve the performance of branching rules with an empirical model-building approach. One model is based on the rationale given for the Jeroslow-Wang rule, variations of which have performed well in recent work. The model is refuted by carefully designed computational experiments. A second model explains the success of the Jeroslow-Wang rule, makes other predictions confirmed by experiment, and leads to the design of branching rules that are clearly superior to Jeroslow-Wang. Recent computational studies [2, 7, 13, 21] suggest that branching algorithms are among the most attractive for solving the propositional satisfiability problem. An important factor in their success---perhaps the dominant factor---is the branching rule they use [...

We wish to bring to the attention of the OR community the phenomenon of phase transitions in randomly generated problems. These are of considerable practical use for benchmarking algorithms. They also offer insight into problem hardness and algorithm performance. Whilst phase transition experiments are frequently performed by AI researchers, such experiments do not appear to be in common use in the OR community. To illustrate the value of such experiments, we examine a typical OR problem, the traveling salesman problem. We report in detail many features of the phase transition in this problem, and show how some of these features are also seen in real problems. Acknowledgements The second author is supported by a HCM Postdoctoral Fellowship. We thank Iain Buchanan for comments on a draft of this paper, and Alan Bundy, and the members of the Mathematical Reasoning Group in Edinburgh for their constructive comments and many CPU cycles donated to these and other experiments from SERC grant GR/H/23610. We also thank the MRG group at Trento and the Department of Computer Science at the University of Strathclyde for additional CPU cycles. Finally, we thank Robert Craig for providing us with his code. 1

This paper presents an informal discussion of issues that arise when one attempts to analyze algorithms experimentally. It is based on lessons learned by the author over the course of more than a decade of experimentation, survey paper writing, refereeing, and lively discussions with other experimentalists. Although written from the perspective of a theoretical computer scientist, it is intended to be of use to researchers from all fields who want to study algorithms experimentally. It has two goals: first, to provide a useful guide to new experimentalists about how such work can best be performed and written up, and second, to challenge current researchers to think about whether their own work might be improved from a scientific point of view. With the latter purpose in mind, the author hopes that at least a few of his recommendations will be considered controversial.

SATLIB is an online resource for SAT-related research established in June 1998. Its core components, a benchmark suite of SAT instances and a collection of SAT solvers, aim to facilitate empirical research on SAT by providing a uniform test-bed for SAT solvers along with freely available implementations of high-performing SAT algorithms. In this article, we give an overview of SATLIB; in particular, we describe its current set of benchmark problems. Currently, the main SATLIB web site

Local search algorithms are among the standard methods for solving hard combinatorial problems from various areas of Artificial Intelligence and Operations Research. For SAT, some of the most successful and powerful algorithms are based on stochastic local search and in the past 10 years a large number of such algorithms have been proposed and investigated. In this article, we focus on two particularly well-known families of local search algorithms for SAT, the GSAT and WalkSAT architectures. We present a detailed comparative analysis of these algorithms' performance using a benchmark set which contains instances from randomised distributions as well as SAT-encoded problems from various domains. We also investigate the robustness of the observed performance characteristics as algorithm-dependent and problem-dependent parameters are changed. Our empirical analysis gives a very detailed picture of the algorithms' performance for various domains of SAT problems; it also reveals a fundamental weakness in some of the best-performing algorithms and shows how this can be overcome.

Stochastic search algorithms are among the most sucessful approaches for solving hard combinatorial problems. A large class of stochastic search approaches can be cast into the framework of Las Vegas Algorithms (LVAs). As the run-time behavior of LVAs is characterized by random variables, the detailed knowledge of run-time distributions provides important information for the analysis of these algorithms. In this paper we propose a novel methodology for evaluating the performance of LVAs, based on the identification of empirical run-time distributions. We exemplify our approach by applying it to Stochastic Local Search (SLS) algorithms for the satisfiability problem (SAT) in propositional logic. We point out pitfalls arising from the use of improper empirical methods and discuss the benefits of the proposed methodology for evaluating and comparing LVAs.

We describe an extensive study of search in GSAT, an approximation procedure for propositional satisfiability. GSAT performs greedy hill-climbing on the number of satisfied clauses in a truth assignment. Our experiments provide a more complete picture of GSAT's search than previous accounts. We describe in detail the two phases of search: rapid hillclimbing followed by a long plateau search. We demonstrate that when applied to randomly generated 3-SAT problems, there is a very simple scaling with problem size for both the mean number of satisfied clauses and the mean branching rate. Our results allow us to make detailed numerical conjectures about the length of the hill-climbing phase, the average gradient of this phase, and to conjecture that both the average score and average branching rate decay exponentially during plateau search. We end by showing how these results can be used to direct future theoretical analysis. This work provides a case study of how computer experiments can be used to improve understanding of the theoretical properties of algorithms. 1.