## SATzilla: Portfolio-based Algorithm Selection for SAT

### Cached

### Download Links

Citations: | 91 - 16 self |

### BibTeX

@MISC{Xu_satzilla:portfolio-based,

author = {Lin Xu and Frank Hutter and Holger H. Hoos and Kevin Leyton-brown},

title = {SATzilla: Portfolio-based Algorithm Selection for SAT},

year = {}

}

### OpenURL

### Abstract

It has been widely observed that there is no single “dominant ” SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of our SATzilla portfolios has been independently verified in the 2007 SAT Competition, where our SATzilla-07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla-07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition. 1.

### Citations

1075 |
A computing procedure for quantification theory
- Davis, Putnam
- 1960
(Show Context)
Citation Context ...& Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., =-=Davis & Putnam, 1960-=-; Dechter & Rish, 1994; Bacchus, 2002b, 2002a; Bacchus & Winter, 2003; Subbarayan & Pradhan, 2005). Unfortunately, most of these SAT algorithms are highly complex, and thus have largely resisted forma... |

740 |
A machine program for theorem proving
- Davis, Logemann, et al.
- 1962
(Show Context)
Citation Context ...velopment and study of algorithms. Today’s high-performance SAT solvers include treec○2007 AI Access Foundation. All rights reserved.sXu, Hutter, Hoos & Leyton-Brown search algorithms (e.g., those by =-=Davis, Logemann, & Loveland, 1962-=-; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selm... |

680 | A New Method for Solving Hard Satisfiability Problems
- Selman, Levesque, et al.
- 1992
(Show Context)
Citation Context ...1962; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., =-=Selman, Levesque, & Mitchell, 1992-=-; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preproces... |

646 | No free lunch theorems for optimization
- Wolpert, Macready
- 1997
(Show Context)
Citation Context ...blem instances from a given class, but dramatically worse on other instances. In fact, it has been shown that no solver can be guaranteed to dominate all others on unrestricted sets of SAT instances (=-=Wolpert & Macready, 1997-=-). Thus, practitioners with hard SAT problems to solve face a potentially difficult “algorithm selection problem” (Rice, 1976): which algorithm(s) should be run in order to minimize some performance o... |

521 | Pushing the envelope: planning, propositional logic and stochastic search
- Kautz, Selman
(Show Context)
Citation Context ...proach has proven effective for tackling many real-world applications, including planning, scheduling, graph coloring, bounded model checking, and formal verification (examples have been described by =-=Kautz & Selman, 1996-=-, 1999; Crawford & Baker, 1994; van Gelder, 2002; Biere, Cimatti, Clarke, Fujita, & Zhu, 1999; Stephan, Brayton, & Sangiovanni-Vencentelli, 1996). The conceptual simplicity of SAT facilitates algorith... |

462 |
An extensible SAT-solver
- Eén, Sörensson
(Show Context)
Citation Context ...rch algorithms (e.g., those by Davis, Logemann, & Loveland, 1962; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; =-=Eén & Sörensson, 2003-=-), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sa... |

361 | B.: Noise strategies for improving local search
- Selman, Kautz, et al.
- 1994
(Show Context)
Citation Context ...alik, 2001; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; =-=Selman, Kautz, & Cohen, 1994-=-; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putna... |

288 | Efficient Conflict Driven Learning in a Boolean Satisfiability Solver
- Zhang, Moskewicz, et al.
(Show Context)
Citation Context ...Today’s high-performance SAT solvers include treec○2007 AI Access Foundation. All rights reserved.sXu, Hutter, Hoos & Leyton-Brown search algorithms (e.g., those by Davis, Logemann, & Loveland, 1962; =-=Zhang, Madigan, Moskewicz, & Malik, 2001-=-; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Ka... |

264 | Symbolic Model Checking using SAT procedures instead of BDDs
- Biere, Cimatti, et al.
- 1999
(Show Context)
Citation Context ... including planning, scheduling, graph coloring, bounded model checking, and formal verification (examples have been described by Kautz & Selman, 1996, 1999; Crawford & Baker, 1994; van Gelder, 2002; =-=Biere, Cimatti, Clarke, Fujita, & Zhu, 1999-=-; Stephan, Brayton, & Sangiovanni-Vencentelli, 1996). The conceptual simplicity of SAT facilitates algorithm development, and considerable research and engineering efforts over the past decades have l... |

258 | Unifying SAT-based and graph-based planning, in - Kautz, Selman - 1999 |

238 | Stochastic Local Search: Foundations and Applications - Hoos, Stützle - 2004 |

176 | The Bayes Net Toolbox for MATLAB
- Murphy
(Show Context)
Citation Context ...on convenience, we used an existing mixture of experts implementation, which is built around an expectation maximization (EM) algorithm that performs iterative reweighted least squares in the M step (=-=Murphy, 2001-=-). We modified this code slightly to clamp the experts. We also initialized the choice of expert to the classifier’s output by setting the initial values of P (z|x, s) to s. To evaluate the model and ... |

126 | Algorithm portfolios - Gomes, Selman - 2001 |

114 | Experimental results on the application of satisfiability algorithms to scheduling problems
- Crawford, Baker
- 1990
(Show Context)
Citation Context ...for tackling many real-world applications, including planning, scheduling, graph coloring, bounded model checking, and formal verification (examples have been described by Kautz & Selman, 1996, 1999; =-=Crawford & Baker, 1994-=-; van Gelder, 2002; Biere, Cimatti, Clarke, Fujita, & Zhu, 1999; Stephan, Brayton, & Sangiovanni-Vencentelli, 1996). The conceptual simplicity of SAT facilitates algorithm development, and considerabl... |

114 |
Combinatorial test generation using satisfiability
- Stephan, Brayton, et al.
- 1996
(Show Context)
Citation Context ...ng, bounded model checking, and formal verification (examples have been described by Kautz & Selman, 1996, 1999; Crawford & Baker, 1994; van Gelder, 2002; Biere, Cimatti, Clarke, Fujita, & Zhu, 1999; =-=Stephan, Brayton, & Sangiovanni-Vencentelli, 1996-=-). The conceptual simplicity of SAT facilitates algorithm development, and considerable research and engineering efforts over the past decades have led to sophisticated algorithms with highly-optimize... |

113 | Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Transaction on pattern analysis and machine learning
- Krishnapuram, Carin, et al.
- 2005
(Show Context)
Citation Context ... an instance’s satisfiability using a classification algorithm that depends on the same instance features used by the empirical hardness models. We chose Sparse Multinomial Logistic Regression, SMLR (=-=Krishnapuram, Carin, Figueiredo, & Hartemink, 2005-=-), but any other classification algorithm that returns the probability that an instance belongs to each class could be used instead. Then, we train conditional empirical hardness models (Msat, Munsat)... |

111 | An economics approach to hard computational problems - Huberman, Lukose, et al. - 1997 |

102 | Estimating the Efficiency of Backtrack Programs - Knuth - 1975 |

100 | Directional resolution: The davis-putnam procedure, revisited
- Dechter, Rish
- 1994
(Show Context)
Citation Context ... Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putnam, 1960; =-=Dechter & Rish, 1994-=-; Bacchus, 2002b, 2002a; Bacchus & Winter, 2003; Subbarayan & Pradhan, 2005). Unfortunately, most of these SAT algorithms are highly complex, and thus have largely resisted formal, average-case analys... |

87 | Scaling and probabilistic smoothing: Efficient dynamic local search for SAT
- Hutter, Tompkins, et al.
- 2002
(Show Context)
Citation Context ...ann, 2002; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; =-=Hutter, Tompkins, & Hoos, 2002-=-; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putnam, 1960; Dechter & Rish, 1994; B... |

86 |
The algorithm selection problem
- Rice
- 1976
(Show Context)
Citation Context ...o dominate all others on unrestricted sets of SAT instances (Wolpert & Macready, 1997). Thus, practitioners with hard SAT problems to solve face a potentially difficult “algorithm selection problem” (=-=Rice, 1976-=-): which algorithm(s) should be run in order to minimize some performance objective, such as expected runtime? The most widely-adopted solution to such algorithm selection problems is to measure every... |

67 | An adaptive noise mechanism for walksat
- Hoos
(Show Context)
Citation Context ...Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & =-=Hoos, 2002-=-; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putnam, 1960; Dechter & Rish, 1994; B... |

64 | Effective preprocessing with hyper-resolution and equality reduction
- Bacchus, Winter
- 2003
(Show Context)
Citation Context ...ng, 2005; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putnam, 1960; Dechter & Rish, 1994; Bacchus, 2002b, 2002a; =-=Bacchus & Winter, 2003-=-; Subbarayan & Pradhan, 2005). Unfortunately, most of these SAT algorithms are highly complex, and thus have largely resisted formal, average-case analysis. Instead, empirical studies are often the on... |

64 | G.: A backbone-search heuristic for efficient solving of hard 3-SAT formulæ
- Dubois, Dequen
- 2001
(Show Context)
Citation Context ...ation. All rights reserved.sXu, Hutter, Hoos & Leyton-Brown search algorithms (e.g., those by Davis, Logemann, & Loveland, 1962; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; Kullmann, 2002; =-=Dubois & Dequen, 2001-=-; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; ... |

63 | A Bayesian approach to tackling hard computational problems
- Horvitz, Ruan, et al.
- 2001
(Show Context)
Citation Context ...r (2005) employed case-based reasoning to select a solution strategy for instances of a CP problem. Various authors have proposed classification-based methods for building algorithm portfolios (e.g., =-=Horvitz, Ruan, Gomes, Kautz, Selman, & Chickering, 2001-=-; Bayesian approach to tackling hard computational problems, ; Gebruers, Guerri, Hnich, & Milano, 2004; Guo & Hsu, 2004). One problem with such approaches is that they use an error metric that penaliz... |

59 | Learning the empirical hardness of optimization problems: The case of combinatorial auctions
- Leyton-Brown, Nudelman, et al.
- 2002
(Show Context)
Citation Context ...xpensive predictor of an algorithm’s runtime on a given problem instance based on features of the instance and the algorithm’s past performance (Nudelman, Leyton-Brown, Hoos, Devkar, & Shoham, 2004a; =-=Leyton-Brown, Nudelman, & Shoham, 2002-=-). By modeling several algorithms and, at runtime, choosing the algorithm predicted to have the best performance, empirical hardness models can serve as the basis for an algorithm portfolio that solve... |

54 | Performance prediction and automated tuning of randomized and parametric algorithms
- Hutter, Hamadi, et al.
- 2006
(Show Context)
Citation Context ...previously proven to be very successful in predicting runtime on uniform random k-SAT, on structured SAT instances, and on combinatorial auction winner determination problems (Nudelman et al., 2004a; =-=Hutter et al., 2006-=-; Leyton-Brown et al., 2002). It should be noted that our portfolio methodology can make use of a variety of different methods for estimating an algorithm’s runtime. For example, we have explored othe... |

54 |
Feature Extraction. Foundations and Applications
- Guyon, Gunn, et al.
- 2006
(Show Context)
Citation Context ... it is difficult to construct features that do not suffer from these problems. Therefore, we first reduce the set of features by performing feature selection, in our case forward selection (see e.g., =-=Guyon, Gunn, Nikravesh, & Zadeh, 2006-=-), a simple iterative method that starts with an empty feature set and greedily adds one feature at a time, aiming to reduce cross-validation error as much as possible with every added feature. Next, ... |

53 | UBCSAT: An implementation and experimentation environment for SLS algorithms for SAT & MAX-SAT
- Tompkins, Hoos
- 2004
(Show Context)
Citation Context ...ause graph, variable graph, clause graph, balance, proximity to Horn formulae, LP-based, DPLL probing and local search probing features — the code for this last group of features was based on UBCSAT (=-=Tompkins & Hoos, 2004-=-). In order to limit the time spent computing features, we slightly modified the feature computation code of Nudelman et al. (2004a). For SATzilla07, we excluded a number of computationally expensive ... |

47 | Enhancing Davis Putnam with extended binary clause reasoning - Bacchus - 2002 |

44 | Understanding random sat: Beyond the clauses-to-variables ratio
- Nudelman, Leyton-Brown, et al.
- 2004
(Show Context)
Citation Context ...uild an empirical hardness model, a computationally inexpensive predictor of an algorithm’s runtime on a given problem instance based on features of the instance and the algorithm’s past performance (=-=Nudelman, Leyton-Brown, Hoos, Devkar, & Shoham, 2004-=-a; Leyton-Brown, Nudelman, & Shoham, 2002). By modeling several algorithms and, at runtime, choosing the algorithm predicted to have the best performance, empirical hardness models can serve as the ba... |

40 | Zchaff2004: An Efficient SAT Solver - Mahajan, Fu, et al. - 2005 |

38 | A portfolio approach to algorithm selection
- Leyton-Brown, Nudelman, et al.
- 2003
(Show Context)
Citation Context ..., choosing the algorithm predicted to have the best performance, empirical hardness models can serve as the basis for an algorithm portfolio that solves the algorithm selection problem automatically (=-=Leyton-Brown, Nudelman, Andrew, McFadden, & Shoham, 2003-=-b, 2003a). In this work we show, for what we believe to be the first time, that such techniques can be used to build an algorithm portfolio that achieves state-of-the-art performance in a broad, pract... |

36 | W.Q.: Diversification and Determinism in Local Search for Satisfiability
- Li, Huang
- 2005
(Show Context)
Citation Context ...eten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; =-=Li & Huang, 2005-=-; Ishtaiwi, Thornton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based approaches (see, e.g., Davis & Putnam, 1960; Dechter & Rish, 1994; Bacchus, 2002b, 2002a; Bacchus & W... |

33 | SATzilla-07: The Design and Analysis of an Algorithm Portfolio for SAT
- Xu, Hutter, et al.
(Show Context)
Citation Context ...ity over this four year interval— placed 1 st in three categories, and 2 nd and 3 rd in two further categories. This solver was described, along with some preliminary analysis, in a conference paper (=-=Xu, Hutter, Hoos, & Leyton-Brown, 2007-=-c); it used hierarchical hardness models, described separately (Xu, Hoos, & Leyton-Brown, 2007a). In this work, we provide a much more detailed description of this new solver, present several new tech... |

32 | Applying machine learning to low knowledge control of optimization algorithms - Carchrae, Beck - 2008 |

31 |
Learning techniques for automatic algorithm portfolio selection
- Guerri, Milano
- 2004
(Show Context)
Citation Context ...mployed case-based reasoning to select a solution strategy for instances of a constraint programming problem. Various authors have proposed classification-based methods for algorithm selection (e.g., =-=Guerri & Milano, 2004-=-; Gebruers, Guerri, Hnich, & Milano, 2004; Guo & Hsu, 2004; and, to some extent, Horvitz, Ruan, Gomes, Kautz, Selman, & Chickering, 2001). One problem with such approaches is that they use an error me... |

29 | NiVER: Non-increasing variable elimination resolution for preprocessing SAT instances
- Subbarayan, Pradhan
- 2005
(Show Context)
Citation Context ...nton, Anbulagan, Sattar, & Pham, 2006; Hoos & Stützle, 2005), and resolution-based preprocessors (see, e.g., Davis & Putnam, 1960; Dechter & Rish, 1994; Bacchus, 2002b, 2002a; Bacchus & Winter, 2003; =-=Subbarayan & Pradhan, 2005-=-). Unfortunately, most of these SAT algorithms are highly complex, and thus have largely resisted formal, average-case analysis. Instead, empirical studies are often the only practical means for compa... |

27 | Branch and bound algorithm selection by performance prediction - Lobjois, Lemaître - 1998 |

27 | Combining multiple heuristics online - Streeter, Golovin, et al. - 2007 |

26 | Learning to select branching rules in the DPLL procedure for satisfiability - Lagoudakis, Littman - 2001 |

26 | Boosting as a Metaphor for Algorithm Design
- Brown, Nudelman, et al.
- 2003
(Show Context)
Citation Context ..., choosing the algorithm predicted to have the best performance, empirical hardness models can serve as the basis for an algorithm portfolio that solves the algorithm selection problem automatically (=-=Leyton-Brown, Nudelman, Andrew, McFadden, & Shoham, 2003-=-b, 2003a). In this work we show, for what we believe to be the first time, that such techniques can be used to build an algorithm portfolio that achieves state-of-the-art performance in a broad, pract... |

25 |
Investigating the behaviour of a SAT solver on random formulas
- Kullmann
- 2002
(Show Context)
Citation Context ... AI Access Foundation. All rights reserved.sXu, Hutter, Hoos & Leyton-Brown search algorithms (e.g., those by Davis, Logemann, & Loveland, 1962; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; =-=Kullmann, 2002-=-; Dubois & Dequen, 2001; Heule, Zwieten, Dufour, & Maaren, 2004; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, T... |

24 | Dynamic Algorithm Portfolios - Gagliolo, Schmidhuber - 2006 |

21 | Fifty-five solvers in Vancouver: The SAT 2004 competition - Berre, Simon - 2004 |

19 | Using cbr to select solution strategies in constraint programming - Gebruers, Hnich, et al. - 2005 |

18 |
March-eq: Implementing additional reasoning into an efficient lookahead SAT solver
- Huele, Zwieten, et al.
- 2004
(Show Context)
Citation Context ...ved.sXu, Hutter, Hoos & Leyton-Brown search algorithms (e.g., those by Davis, Logemann, & Loveland, 1962; Zhang, Madigan, Moskewicz, & Malik, 2001; Zhang, 2002; Kullmann, 2002; Dubois & Dequen, 2001; =-=Heule, Zwieten, Dufour, & Maaren, 2004-=-; Eén & Sörensson, 2003), local search algorithms (see, e.g., Selman, Levesque, & Mitchell, 1992; Selman, Kautz, & Cohen, 1994; Hutter, Tompkins, & Hoos, 2002; Hoos, 2002; Li & Huang, 2005; Ishtaiwi, ... |

17 | Exploring the computational tradeoff of more reasoning and less searching - Bacchus - 2002 |

17 | Learning To Solve QBF - Samulowitz, Memisevic - 2007 |

16 | Learning dynamic algorithm portfolios - Gagliolo, Schmidhuber - 2006 |