## Some Pitfalls for Experimenters with Random SAT (1996)

Venue: | Artificial Intelligence |

Citations: | 20 - 3 self |

### BibTeX

@ARTICLE{Mitchell96somepitfalls,

author = {David G. Mitchell and Hector J. Levesque},

title = {Some Pitfalls for Experimenters with Random SAT},

journal = {Artificial Intelligence},

year = {1996},

volume = {81},

pages = {81--1}

}

### Years of Citing Articles

### OpenURL

### Abstract

We consider the use of random CNF formulas in evaluating the performance of SAT testing algorithms, and in particular the role that the phase transition phenomenon plays in this use. Examples from the literature illustrate the importance of understanding the properties of formula distributions prior to designing an experiment. We expect this to be of increasing importance in the field. 1 Introduction Satisfiability testing lies at the core of many computational problems and because of its close relationship to various reasoning tasks, this is especially so in Artificial Intelligence. Randomly generated CNF formulas are a popular class of test problems for evaluating the performance of SAT testing programs. Not surprisingly, the choice of formula distribution is crucial to the validity of any investigation using random formulas. In [23], we argued that some families of distributions were more useful sources of test material than others, and suggested choosing formulas from the "hard reg...

### Citations

11502 |
Computers and Intractability, A Guide to the Theory of NPCompleteness
- Garey, Johnson
- 1979
(Show Context)
Citation Context ...hat we now know about the formulas used. Additional data is presented as needed. In Section 8 we summarize. 2 Materials and Methods We assume the reader is familiar with the problem SAT as defined in =-=[12]. The data-=- for our analysis was produced by testing randomly generated SAT instances with a simple version of the Davis-Putnam Procedure [8]. Our procedure, which we refer to as "DP", is shown in Figu... |

1124 | A computing procedure for quantification theory - Davis, Putnam - 1960 |

707 | A new method for solving hard satisfiability problems
- Selman, Levesque, et al.
- 1992
(Show Context)
Citation Context ...may not be the case. Likely candidates here are some of the localsearch based methods. These have been used with considerable success for finding satisfying truth assignments for satisfiable formulas =-=[26,16,25]-=-. Random 3-SAT formulas have been among the primary test sets used in this work, and success has been reported with formulas near the hard region and having thousands of variables [25]. Although perfo... |

613 | Where the really hard problems are
- Cheeseman, Kanefsky, et al.
- 1991
(Show Context)
Citation Context ...ower (that is, occurs over a smaller range of c) as n is increased. The peak hardness for DP occurs at approximately the point where 40% of the formulas are satisfiable, and shifts 5 Cheeseman et al. =-=[3]-=- hypothesized that phase transitions might not occur for problems in P. Random 2-SAT is one of several counterexamples, but the present observation suggests a revised version of the hypothesis. 0.0 0.... |

573 |
A machine program for theoremproving
- Davis, Logemann, et al.
- 1962
(Show Context)
Citation Context ...minfvars(!)g if DP(!np)=satisfiable then return satisfiable else return DP(!n:p) end DP Fig. 1. The procedure DP. DP is essentially the splitting variant of the Davis-Putnam Procedure as described in =-=[7]-=-, but without the pure literal rule. It uses no heuristics other than unit propagation. This is perhaps the simplest complete SAT testing procedure that performs well enough to be useful, and so provi... |

378 | Noise Strategies for Improving Local Search
- Selman, Kautz, et al.
- 1994
(Show Context)
Citation Context ...may not be the case. Likely candidates here are some of the localsearch based methods. These have been used with considerable success for finding satisfying truth assignments for satisfiable formulas =-=[26,16,25]-=-. Random 3-SAT formulas have been among the primary test sets used in this work, and success has been reported with formulas near the hard region and having thousands of variables [25]. Although perfo... |

202 | Experimental results on the crossover point in satisfiability problems
- Crawford, Auton
- 1993
(Show Context)
Citation Context ...portion of satisfiable formulas changes from almost 1 to almost 0. The location of this "transition region" varies with k, but for each k it occurs at approximately the same ratio for all va=-=lues of n [5,14,20]-=-. The asymptotic location of the transition region is analytically bounded above and below: For k = 3, the currently known bounds are c = 4:758 [18] and c = 3:003 [10] respectively. The lower graph of... |

154 | Critical behavior in the satisfiability of random Boolean expressions
- Kirkpatrick, Selman
- 1994
(Show Context)
Citation Context ...portion of satisfiable formulas changes from almost 1 to almost 0. The location of this "transition region" varies with k, but for each k it occurs at approximately the same ratio for all va=-=lues of n [5,14,20]-=-. The asymptotic location of the transition region is analytically bounded above and below: For k = 3, the currently known bounds are c = 4:758 [18] and c = 3:003 [10] respectively. The lower graph of... |

119 | Experimental results on the application of satisfiability algorithms to scheduling problems
- Crawford, Baker
- 1994
(Show Context)
Citation Context ...opular "unstructured" random formulas. The value of these as test material -- even when sampled from the "hard region" -- can be questioned on the grounds that they may not be much=-= like real problems [4,17,21]-=-, but some of these distributions appear challenging for a variety of methods, and we expect their use to continue. Moreover, most available alternatives are either puzzle-type problems (such as cross... |

105 |
Efficient local search for very large-scale satisfiability problems. Sigart Bulletin 3(1):8–12
- Gu
- 1992
(Show Context)
Citation Context ...ndom k-SAT formulas. In this section we will see that experimenters have often used formulas from this distribution, but from below the hard region, which are trivial in spite of their large size. Gu =-=[16]-=- and Kamath et al. [19] evaluated their procedures (based on local search and interior point programming, respectively) by testing large random E3-SAT formulas, Gu with n 2 f50; 500; 1000g and Kamath ... |

102 | Tail bounds for occupancy and the satisfiability threshold conjecture, Random Structures Algorithms 7
- Kamath, Motwani, et al.
- 1995
(Show Context)
Citation Context ...proximately the same ratio for all values of n [5,14,20]. The asymptotic location of the transition region is analytically bounded above and below: For k = 3, the currently known bounds are c = 4:758 =-=[18]-=- and c = 3:003 [10] respectively. The lower graph of Figure 2 shows the median number of DP steps required to test the same formulas. Note the logarithmic y-axis, necessitated by the dramatic increase... |

92 | Probabilistic analysis of a generalization of the unit-clause literal selection heuristics for the k satisfiable problem
- Chao, T, et al.
- 1990
(Show Context)
Citation Context ...work by Franco [9] and others presented an algorithm which solves instances of this family with probability not less than 1 \Gamma ffl, for any constant ffl ? 0, in polynomial expected time. See also =-=[24,2]-=-. 0.0 0.5 1.0 Pr[SAT] 10 100 1000 10000 0 5 10 15 20 25 30 DP Steps Ratio of Clauses to Variables 2-SAT 3-SAT 4-SAT 5-SAT Fig. 2. Median DP steps for random k-SAT: varying c for selected values of k. ... |

65 |
Algorithms for testing the satisfiability of propositional formulae
- Gallo, Urbani
- 1989
(Show Context)
Citation Context ...essed as a resolution strategy [15] -- required almost no backtracking (less than 3 backtracks on average) to test them, and consistently solves them in a small fraction of a second. Gallo and Urbani =-=[11]-=- compared the performance of several enhancements to the Davis-Putnam Procedure on [1,7]-SAT. We again generated a range of these formulas and tested them with DP. Figure 6 shows the median DP steps a... |

62 | The SAT Phase Transition
- Gent, Walsh
- 1994
(Show Context)
Citation Context ...portion of satisfiable formulas changes from almost 1 to almost 0. The location of this "transition region" varies with k, but for each k it occurs at approximately the same ratio for all va=-=lues of n [5,14,20]-=-. The asymptotic location of the transition region is analytically bounded above and below: For k = 3, the currently known bounds are c = 4:758 [18] and c = 3:003 [10] respectively. The lower graph of... |

54 |
Resolution vs cutting plane solution of inference problems: some computational experience
- Hooker
- 1988
(Show Context)
Citation Context ...opular "unstructured" random formulas. The value of these as test material -- even when sampled from the "hard region" -- can be questioned on the grounds that they may not be much=-= like real problems [4,17,21]-=-, but some of these distributions appear challenging for a variety of methods, and we expect their use to continue. Moreover, most available alternatives are either puzzle-type problems (such as cross... |

46 | Evidence for satisfiability threshold for random 3CNF formulas
- Larrabee, T, et al.
- 1993
(Show Context)
Citation Context ...hanges with c. In [23] we suggested that peak hardness of random 3-SAT for the Davis-Putnam Procedure was near the ratio where about half the instances were satisfiable, but the algorithms in each of =-=[23,22,5]-=- appear to peak at slightly different ratios. 3.2 Random Ek-SAT Figure 4 shows the proportion of satisfiable formulas and the median DP steps for testing random Ek-SAT with n = 25 variables, for k 2 f... |

45 |
Endre Tarjan. A linear-time algorithm for testing the truth of certain quantified boolean formulas
- Aspvall, Plass, et al.
- 1982
(Show Context)
Citation Context ... and then begins to decrease again. Random 2-SAT does not exhibit a hard region in the sense that is seen when k is larger 5 , which is perhaps not surprising since 2-SAT is solvable in time O(n + m) =-=[1]-=-. Table 1 shows median DP steps, and the ratio of median DP steps to n, to test random 2-SAT with c = 1 and n up to 250. At least over this range the median number of steps is smaller than n, and grow... |

44 | Computational experience with an interior point algorithm on the satisfiability problem
- Kamath, Karmarkar, et al.
- 1990
(Show Context)
Citation Context ... this section we will see that experimenters have often used formulas from this distribution, but from below the hard region, which are trivial in spite of their large size. Gu [16] and Kamath et al. =-=[19]-=- evaluated their procedures (based on local search and interior point programming, respectively) by testing large random E3-SAT formulas, Gu with n 2 f50; 500; 1000g and Kamath with n = 1000, in all c... |

29 |
On the complexity of the satisfiability problem
- Goldberg
- 1979
(Show Context)
Citation Context ...o solve these problems, while his resolution procedure performed so poorly that he was unable to report completion times for most conditions. Yet DP -- which can be expressed as a resolution strategy =-=[15]-=- -- required almost no backtracking (less than 3 backtracks on average) to test them, and consistently solves them in a small fraction of a second. Gallo and Urbani [11] compared the performance of se... |

24 |
On the Probabilistic Performance of Algorithms for the Satisfiability Problem
- FRANCO
- 1986
(Show Context)
Citation Context ... over some fixed range [k,l]. We will call such distributions "[k,l]-SAT". 3 A clause is trivial if it contains both a variable and its negation. 4 Moreover, Wu and Tang [27] extending work =-=by Franco [9]-=- and others presented an algorithm which solves instances of this family with probability not less than 1 \Gamma ffl, for any constant ffl ? 0, in polynomial expected time. See also [24,2]. 0.0 0.5 1.... |

23 | The hardest random SAT problems, in
- Gent, Walsh
- 1994
(Show Context)
Citation Context ...imilarly sized hard random 3-SAT formulas). The harder formulas occur so rarely we would not expect to see them in studies with small sample sizes such as the examples in this section. Gent and Walsh =-=[13]-=- have studied apparently harder, but rarer still, formulas in this region. These are interesting in themselves, but we believe they occur even more rarely as n gets large, and do not make the distribu... |

18 |
Easy to be hard: difficult problems for greedy algorithms
- Konolige
- 1994
(Show Context)
Citation Context ...opular "unstructured" random formulas. The value of these as test material -- even when sampled from the "hard region" -- can be questioned on the grounds that they may not be much=-= like real problems [4,17,21]-=-, but some of these distributions appear challenging for a variety of methods, and we expect their use to continue. Moreover, most available alternatives are either puzzle-type problems (such as cross... |

10 |
A Survey of Average Time Analyses of Satisfiability Algorithms
- PURDOM
- 1990
(Show Context)
Citation Context ...work by Franco [9] and others presented an algorithm which solves instances of this family with probability not less than 1 \Gamma ffl, for any constant ffl ? 0, in polynomial expected time. See also =-=[24,2]-=-. 0.0 0.5 1.0 Pr[SAT] 10 100 1000 10000 0 5 10 15 20 25 30 DP Steps Ratio of Clauses to Variables 2-SAT 3-SAT 4-SAT 5-SAT Fig. 2. Median DP steps for random k-SAT: varying c for selected values of k. ... |

9 |
Solving the satisfiability problem by using randomized approach
- Wu, Tang
- 1992
(Show Context)
Citation Context ... lengths distributed uniformly over some fixed range [k,l]. We will call such distributions "[k,l]-SAT". 3 A clause is trivial if it contains both a variable and its negation. 4 Moreover, Wu=-= and Tang [27]-=- extending work by Franco [9] and others presented an algorithm which solves instances of this family with probability not less than 1 \Gamma ffl, for any constant ffl ? 0, in polynomial expected time... |

4 |
Solving satisfiability via Boltzmann Machines
- DAnjou, Gra��a, et al.
- 1993
(Show Context)
Citation Context ...rmulas we tested required as many as three. At lower ratios even fewer backtracks were needed. These formulas, despite being very large, are trivial to test on average. 6 Interaction of Parameters In =-=[6]-=- d'Anjou and his colleagues report tests using a variety of random formulas to demonstrate the performance of a Boltzmann Machine method for SAT testing (which we will refer to as BM). Their data is p... |

3 |
Analysis of three simple heuristics on a random instance of k-SAT
- Frieze, Suen
- 1992
(Show Context)
Citation Context ...e ratio for all values of n [5,14,20]. The asymptotic location of the transition region is analytically bounded above and below: For k = 3, the currently known bounds are c = 4:758 [18] and c = 3:003 =-=[10]-=- respectively. The lower graph of Figure 2 shows the median number of DP steps required to test the same formulas. Note the logarithmic y-axis, necessitated by the dramatic increase of peak difficulty... |

1 |
Generating hard satisfiability problems. This volume
- Mitchell, Selman, et al.
(Show Context)
Citation Context ...of test problems for evaluating the performance of SAT testing programs. Not surprisingly, the choice of formula distribution is crucial to the validity of any investigation using random formulas. In =-=[23], we argue-=-d that some families of distributions were more useful sources of test material than others, and suggested choosing formulas from the "hard region" associated with the satisfiable--to--unsat... |