Algorithm Selection for Combinatorial Search Problems: A Survey
---
paper_title: The Algorithm Selection Problem

paper_content:

Publisher Summary The problem of selecting an effective algorithm arises in a wide variety of situations. This chapter starts with a discussion on abstract models: the basic model and associated problems, the model with selection based on features, and the model with variable performance criteria. One objective of this chapter is to explore the applicability of the approximation theory to the algorithm selection problem. There is an intimate relationship here and that the approximation theory forms an appropriate base upon which to develop a theory of algorithm selection methods. The approximation theory currently lacks much of the necessary machinery for the algorithm selection problem. There is a need to develop new results and apply known techniques to these new circumstances. The final pages of this chapter form a sort of appendix, which lists 15 specific open problems and questions in this area. There is a close relationship between the algorithm selection problem and the general optimization theory. This is not surprising since the approximation problem is a special form of the optimization problem. Most realistic algorithm selection problems are of moderate to high dimensionality and thus one should expect them to be quite complex. One consequence of this is that most straightforward approaches (even well-conceived ones) are likely to lead to enormous computations for the best selection. The single most important part of the solution of a selection problem is the appropriate choice of the form for selection mapping. It is here that theories give the least guidance and that the art of problem solving is most crucial.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: No free lunch theorems for optimization

paper_content:

A framework is developed to explore the connection between effective optimization algorithms and the problems they are solving. A number of "no free lunch" (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over another class. These theorems result in a geometric interpretation of what it means for an algorithm to be well suited to an optimization problem. Applications of the NFL theorems to information-theoretic aspects of optimization and benchmark measures of performance are also presented. Other issues addressed include time-varying optimization problems and a priori "head-to-head" minimax distinctions between optimization algorithms, distinctions that result despite the NFL theorems' enforcing of a type of uniformity over all algorithms.
---
paper_title: Learning Rules for Adaptive Planning

paper_content:

This paper presents a novel idea, which combines Planning, Machine Learning and Knowledge-Based techniques. It is concerned with the development of an adaptive planning system that can fine-tune its planning parameters based on the values of specific measurable characteristics of the given planning problem. Adaptation is guided by a rule-based system, whose knowledge has been acquired through machine learning techniques. Specifically, the algorithm of classification based on association rules was applied to a large dataset produced by results from experiments on a large number of problems used in the three AIPS Planning competitions. The paper presents experimental results with the adaptive planner, which demonstrate the boost in performance of the planning system.
---
paper_title: Confronting hardness using a hybrid approach

paper_content:

A hybrid algorithm is a collection of heuristics, paired with a polynomial time selector S that runs on the input to decide which heuristic should be executed to solve the problem. Hybrid algorithms are of particular interest in scenarios where the selector must decide between heuristics that are "good" with respect to different complexity measures.We focus on hybrid algorithms with a "hardness-defying" property: for a problem II, there is a set of complexity measures {m i } whereby II is known or conjectured to be unsolvable for each m i , but for each heuristic h i of the hybrid algorithm, one can give a complexity guarantee for h i on the instances of II that S selects for h i that is strictly better than m i . More concretely, we show that for several NP-hard problems, a given instance can either be solved exactly with substantially improved runtime (e.g. 2o(n)), or be approximated in polynomial time with an approximation ratio exceeding that of the known or conjectured inapproximability of the problem, assuming P ≠ NP.
---
paper_title: Phase transitions and the search problem

paper_content:

We describe how techniques that were originally developed in statistical mechanics can be applied to search problems that arise commonly in artificial intelligence. This approach is useful for understanding the typical behavior of classes of problems. In particular, these techniques predict that abrupt changes in computational cost, analogous to physical phase transitions, should occur universally, as heuristic effectiveness or search space topology is varied. We also present a number of open questions raised by these studies.
---
paper_title: An Attempt to Map the Performance of a Range of Algorithm and Heuristic Combinations

paper_content:

Constraint satisfaction is the core of many AI and real life problems and much research has been done in this field in recent years. Work has been done in the past in comparing the performance of different algorithms and heuristics. Much of such work has focused on finding "the best" algorithm and heuristic combination for all problems. The objective of this paper is to prove that there is no universally best algorithm and heuristic for all problems -different problems can be solved most efficiently by different algorithm and heuristic combinations. The implication of this is important because it means that instead of trying to find "the best" algorithms and heuristics, future research should try to identify the application domain of each algorithm and heuristic (i.e. when they are most effective). Furthermore our results point to future research which focuses on how to retrieve the most efficient algorithm for a given problem. The results in this paper provide a first step towards achieving such goals. * The research reported in this paper was supported by the University of Essex research promotion fund ref. R7027 and by the EPSRC research grant ref. GR/J/42878.
---
paper_title: Understanding Random SAT: Beyond the Clauses-to-Variables Ratio

paper_content:

It is well known that the ratio of the number of clauses to the number of variables in a random k-SAT instance is highly correlated with the instance's empirical hardness. We consider the problem of identifying such features of random SAT instances automatically using machine learning. We describe and analyze models for three SAT solvers - kcnfs, oksolver and satz- and for two different distributions of instances: uniform random 3-SAT with varying ratio of clauses-to-variables, and uniform random 3-SAT with fixed ratio of clauses-to-variables. We show that surprisingly accurate models can be built in all cases. Furthermore, we analyze these models to determine which features are most useful in predicting whether an instance will be hard to solve. Finally we discuss the use of our models to build SATzilla, an algorithm portfolio for SAT 1 .
---
paper_title: Where the REALLY Hard Problems Are

paper_content:

It is well known that for many NP-complete problems, such as K-Sat, etc., typical cases are easy to solve; so that computationally hard cases must be rare (assuming P = NP). This paper shows that NP-complete problems can be summarized by at least one "order parameter", and that the hard problems occur at a critical value of such a parameter. This critical value separates two regions of characteristically different properties. For example, for K-colorability, the critical value separates overconstrained from underconstrained random graphs, and it marks the value at which the probability of a solution changes abruptly from near 0 to near 1. It is the high density of well-separated almost solutions (local minima) at this boundary that cause search algorithms to "thrash". This boundary is a type of phase transition and we show that it is preserved under mappings between problems. We show that for some P problems either there is no phase transition or it occurs for bounded N (and so bounds the cost). These results suggest a way of deciding if a problem is in P or NP and why they are different.
---
paper_title: For Every Generalization Action, Is There Really An Equal And Opposite Reaction? Analysis of the Conservation Law for Generalization Performance

paper_content:

An improved vehicle rear-view mirror assembly of the type in which the mirror support arm is carried by means of at least one movable joint from the mounting part of the mirror assembly which is adapted to be secured to the bodywork of the vehicle includes a shroud of a resilient and flexible material mounted on the back of the mirror, and an extension having at least one movable joint with the end region of the extension in sealing relation with the mounting part, the extension enshrouded by a bellows-like tubular gaiter of flexible material. The shroud and the extension reduce the risk of injury to a person who may be accidentally hit by, or hit, the mirror assembly and the bellows-like gaiter also protects the movable joint or joints from dirt and moisture while readily flexing to allow the joint or joints to move or to be adjusted.
---
paper_title: The supervised learning no-free-lunch Theorems

paper_content:

This paper reviews the supervised learning versions of the no-free-lunch theorems in a simplified form. It also discusses the significance of those theorems, and their relation to other aspects of supervised learning.
---
paper_title: How to Get a Free Lunch: A Simple Cost Model for Machine Learning Applications

paper_content:

This paper proposes a simple cost model for machine learning applications based on the notion of net present value. The model extends and unifies the models used in (Pazzani et al., 1994) and (Masand & Piatetsky-Shapiro, 1996). It attempts to answer the question "Should a given machine learning system now in the prototype stage be fielded?" The model’s inputs are the system’s confusion matrix, the cash flow matrix for the application, the cost per decision, the one-time cost of deploying the system, and the rate of return on investment. Like Provost and Fawcett’s (1997) ROC convex hull method, the present model can be used for decision-making even when its input variables are not known exactly. Despite its simplicity, it has a number of non-trivial consequences. For example, under it the "no free lunch" theorems of learning theory no longer apply.
---
paper_title: No free lunch theorems for optimization

paper_content:

A framework is developed to explore the connection between effective optimization algorithms and the problems they are solving. A number of "no free lunch" (NFL) theorems are presented which establish that for any algorithm, any elevated performance over one class of problems is offset by performance over another class. These theorems result in a geometric interpretation of what it means for an algorithm to be well suited to an optimization problem. Applications of the NFL theorems to information-theoretic aspects of optimization and benchmark measures of performance are also presented. Other issues addressed include time-varying optimization problems and a priori "head-to-head" minimax distinctions between optimization algorithms, distinctions that result despite the NFL theorems' enforcing of a type of uniformity over all algorithms.
---
paper_title: Using Online Algorithms to Solve NP-Hard Problems More Efficiently in Practice

paper_content:

This thesis develops online, algorithms that can be used to solve a wide variety of NP-hard problems more efficiently in practice. The common approach taken by all our online algorithms is to improve the performance of one or more existing algorithms for a specific NP-hard problem by adapting the algorithms to the sequence of problem instance(s) they are run on. ::: We begin by presenting an algorithm for solving a specific class of online resource allocation problems. Our online algorithm can be applied in environments where abstract jobs arrive one at a time, and one can complete the jobs by investing time in a number of abstract activities. Provided the jobs and activities satisfy certain technical conditions, our online algorithm is guaranteed to perform almost as well as any fixed schedule for investing time in the various activities, according to two natural measures of performance: (i) the average time required to complete each job, and (ii) the number of jobs completed within time T, for some fixed deadline T > 0. ::: In particular, our online algorithm's guarantees apply if the job can be written as a monotone, submodular function of a set of pairs of the form (v, τ), where τ is the time invested in activity v. Under the first objective, the offline version of this problem generalizes MIN-SUM SET COVER and the related PIPELINED SET COVER problem. Under the second objective, the offline version of this problem generalizes the problem of maximizing a monotone, submodular set function subject to a knapsack constraint. Our online algorithm has potential applications in a number of areas, including the design of algorithm portfolios, database query processing, and sensor placement. ::: We apply this online algorithm to the following problem. We are given k algorithms, and are fed, one at a time, a sequence of problem instances to solve. We may solve each instance using any of the k algorithms, we may interleave the execution of the algorithms, and, if the algorithms are randomized, we may periodically restart them with a fresh random seed. Our goal is to minimize the total CPU time required to solve all the instances. Using data from eight recent solver competitions, we show that our online algorithm and its offline counterpart can be used to improve the performance of state-of-the-art solvers in a number of problem domains, including Boolean satisfiability, zero-one integer programming, constraint satisfaction, and theorem proving. ::: We next present an online algorithm that can be used to improve the performance of algorithms that solve an optimization problem by making a sequence of calls to a decision procedure that answers questions of the form "Is there a solution of cost at most k?" We present an adaptive strategy for determining the sequence of questions to ask, along with bounds on the maximum time to spend waiting for an answer to each question. Under the assumption that the time required by the decision procedure to return an answer increases as k gets closer to the optimal solution cost, our strategy's performance is near-optimal when measured in terms of a natural competitive ratio. Experimentally, we show that applying our strategy to recent algorithms for A.I. planning and job shop scheduling allows the algorithms to find approximately optimal solutions more quickly. ::: Lastly, we develop algorithms for solving the max k-armed bandit problem, a variant of the classical k-armed bandit problem in which one seeks to maximize the highest payoff received on any single trial, rather than the cumulative payoff. A strategy for solving the max k-armed bandit problem can be used to allocate trials amongmulti-start optimization heuristics. Motivated by results in extreme value theory, we present a no-regret strategy for the special case in which each arm returns payoffs drawn from a generalized extreme value distribution. We also present a heuristic strategy that solves the max k-armed bandit problem using a strategy for the classical k-armed bandit problem as a subroutine. Experimentally, we show that our max k-armed bandit strategy can be used to effectively allocate trials among multi-start heuristics for the RCPSP/max, a difficult real-world scheduling problem.
---
paper_title: Programming by optimization

paper_content:

Avoid premature commitment, seek design alternatives, and automatically generate performance-optimized software.
---
paper_title: An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

paper_content:

Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only “hard” areas but also outliers and noise.
---
paper_title: PetaBricks: a language and compiler for algorithmic choice

paper_content:

It is often impossible to obtain a one-size-fits-all solution for high performance algorithms when considering different choices for data distributions, parallelism, transformations, and blocking. The best solution to these choices is often tightly coupled to different architectures, problem sizes, data, and available system resources. In some cases, completely different algorithms may provide the best performance. Current compiler and programming language techniques are able to change some of these parameters, but today there is no simple way for the programmer to express or the compiler to choose different algorithms to handle different parts of the data. Existing solutions normally can handle only coarse-grained, library level selections or hand coded cutoffs between base cases and recursive cases. We present PetaBricks, a new implicitly parallel language and compiler where having multiple implementations of multiple algorithms to solve a problem is the natural way of programming. We make algorithmic choice a first class construct of the language. Choices are provided in a way that also allows our compiler to tune at a finer granularity. The PetaBricks compiler autotunes programs by making both fine-grained as well as algorithmic choices. Choices also include different automatic parallelization techniques, data distributions, algorithmic parameters, transformations, and blocking. Additionally, we introduce novel techniques to autotune algorithms for different convergence criteria. When choosing between various direct and iterative methods, the PetaBricks compiler is able to tune a program in such a way that delivers near-optimal efficiency for any desired level of accuracy. The compiler has the flexibility of utilizing different convergence criteria for the various components within a single algorithm, providing the user with accuracy choice alongside algorithmic choice.
---
paper_title: Popular Ensemble Methods: An Empirical Study

paper_content:

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund & Schapire, 1996; Schapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier - especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Ensemble Methods in Machine Learning

paper_content:

Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly.
---
paper_title: Cross-disciplinary perspectives on meta-learning for algorithm selection

paper_content:

The algorithm selection problem [Rice 1976] seeks to answer the question: Which algorithm is likely to perform best for my problemq Recognizing the problem as a learning task in the early 1990's, the machine learning community has developed the field of meta-learning, focused on learning about learning algorithm performance on classification problems. But there has been only limited generalization of these ideas beyond classification, and many related attempts have been made in other disciplines (such as AI and operations research) to tackle the algorithm selection problem in different ways, introducing different terminology, and overlooking the similarities of approaches. In this sense, there is much to be gained from a greater awareness of developments in meta-learning, and how these ideas can be generalized to learn about the behaviors of other (nonlearning) algorithms. In this article we present a unified framework for considering the algorithm selection problem as a learning problem, and use this framework to tie together the crossdisciplinary developments in tackling the algorithm selection problem. We discuss the generalization of meta-learning concepts to algorithms focused on tasks including sorting, forecasting, constraint satisfaction, and optimization, and the extension of these ideas to bioinformatics, cryptography, and other fields.
---
paper_title: On Algorithm Selection, with an application to combinatorial search problems

paper_content:

The Algorithm Selection Problem is to select the most appropriate way for solving a problem given a choice of different ways. Some of the most prominent and successful applications come from Artificial Intelligence and in particular combinatorial search problems. Machine Learning has established itself as the de facto way of tackling the Algorithm Selection Problem. Yet even after a decade of intensive research, there are no established guidelines as to what kind of Machine Learning to use and how. This dissertation presents an overview of the field of Algorithm Selection and associated research and highlights the fundamental questions left open and problems facing practitioners. In a series of case studies, it underlines the difficulty of doing Algorithm Selection in practice and tackles issues related to this. The case studies apply Algorithm Selection techniques to new problem domains and show how to achieve significant performance improvements. Lazy learning in constraint solving and the implementation of the all different constraint are the areas in which we improve on the performance of current state of the art systems. The case studies furthermore provide empirical evidence for the effectiveness of using the misclassification penalty as an input to Machine Learning. After having established the difficulty, we present an effective technique for reducing it. Machine Learning ensembles are a way of reducing the background knowledge and experimentation required from the researcher while increasing the robustness of the system. Ensembles do not only decrease the difficulty, but can also increase the performance of Algorithm Selection systems. They are used to much the same ends in Machine Learning itself. We finally tackle one of the great remaining challenges of Algorithm Selection – which Machine Learning technique to use in practice. Through a large-scale empirical evaluation Constraints (2015) 20:481–482 DOI 10.1007/s10601-015-9214-x * Lars Kotthoff larsko@cs.ubc.ca 1 BETA Lab, Department of Computer Science, University of British Columbia, Vancouver, BC, Canada on diverse data taken from Algorithm Selection applications in the literature, we establish recommendations for Machine Learning algorithms that are likely to perform well in Algorithm Selection for combinatorial search problems. The recommendations are based on strong empirical evidence and additional statistical simulations. The research presented in this dissertation significantly reduces the knowledge threshold for researchers who want to perform Algorithm Selection in practice. It makes major contributions to the field of Algorithm Selection by investigating fundamental issues that have been largely ignored by the research community so far. School: University of St Andrews Supervisors: Ian Miguel Ian Gent Graduated: Wednesday, June 20, 2012 Link to full text: http://www.a4cp.org/sites/default/files/lars_kotthoff_-_on_algorithm_ selection_with_an_application_to_combinatorial_search_problems.pdf 482 Constraints (2015) 20:481–482
---
paper_title: Automatic Algorithm Selection for Complex Simulation Problems

paper_content:

To select the most suitable simulation algorithm for a given task is often difficult. This is due to intricate interactions between model features, implementation details, and runtime environment, which may strongly affect the overall performance. An automated selection of simulation algorithms supports users in setting up simulation experiments without demanding expert knowledge on simulation. Roland Ewald analyzes and discusses existing approaches to solve the algorithm selection problem in the context of simulation. He introduces a framework for automatic simulation algorithm selection and describes its integration into the open-source modelling and simulation framework James II. Its selection mechanisms are able to cope with three situations: no prior knowledge is available, the impact of problem features on simulator performance is unknown, and a relationship between problem features and algorithm performance can be established empirically. The author concludes with an experimental evaluation of the developed methods.
---
paper_title: Performance prediction and automated tuning of randomized and parametric algorithms

paper_content:

Machine learning can be used to build models that predict the run-time of search algorithms for hard combinatorial problems. Such empirical hardness models have previously been studied for complete, deterministic search algorithms. In this work, we demonstrate that such models can also make surprisingly accurate predictions of the run-time distributions of incomplete and randomized search methods, such as stochastic local search algorithms. We also show for the first time how information about an algorithm's parameter settings can be incorporated into a model, and how such models can be used to automatically adjust the algorithm's parameters on a per-instance basis in order to optimize its performance. Empirical results for Novelty + and SAPS on structured and unstructured SAT instances show very good predictive performance and significant speedups of our automatically determined parameter settings when compared to the default and best fixed distribution-specific parameter settings.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: Branch and Bound Algorithm Selection by Performance Prediction

paper_content:

We propose a method called Selection by Performance Prediction (SPP) which allows one, when faced with a particular problem instance, to select a Branch and Bound algorithm from among several promising ones. This method is based on Knuth's sampling method which estimates the efficiency of a backtrack program on a particular instance by iteratively generating random paths in the search tree. We present a simple adaptation of this estimator in the field of combinatorial optimization problems, more precisely for an extension of the maximal constraint satisfaction framework. Experiments both on random and strongly structured instances show that, in most cases, the proposed method is able to select, from a candidate list, the best algorithm for solving a given instance.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: Empirical hardness models: Methodology and a case study on combinatorial auctions

paper_content:

Is it possible to predict how long an algorithm will take to solve a previously-unseen instance of an NP-complete problemq If so, what uses can be found for models that make such predictionsq This article provides answers to these questions and evaluates the answers experimentally. We propose the use of supervised machine learning to build models that predict an algorithm's runtime given a problem instance. We discuss the construction of these models and describe techniques for interpreting them to gain understanding of the characteristics that cause instances to be hard or easy. We also present two applications of our models: building algorithm portfolios that outperform their constituent algorithms, and generating test distributions that emphasize hard problems. We demonstrate the effectiveness of our techniques in a case study of the combinatorial auction winner determination problem. Our experimental results show that we can build very accurate models of an algorithm's running time, interpret our models, build an algorithm portfolio that strongly outperforms the best single algorithm, and tune a standard benchmark suite to generate much harder problem instances.
---
paper_title: Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection

paper_content:

The results of empirical comparisons of existing learning algorithms illustrate that each algorithm has a selective superiority ; it is best for some but not all tasks. Given a data set, it is often not clear beforehand which algorithm will yield the best performance. In such cases one must search the space of available algorithms to find the one that produces the best classifier. In this paper we present an approach that applies knowledge about the representational biases of a set of learning algorithms to conduct this search automatically. In addition, the approach permits the available algorithms' model classes to be mixed in a recursive tree-structured hybrid. We describe an implementation of the approach, MCS, that performs a heuristic bestfirst search for the best hybrid classifier for a set of data. An empirical comparison of MCS to each of its primitive learning algorithms, and to the computationally intensive method of cross-validation, illustrates that automatic selection of learning algorithms using knowledge can be used to solve the selective superiority problem.
---
paper_title: Confronting hardness using a hybrid approach

paper_content:

A hybrid algorithm is a collection of heuristics, paired with a polynomial time selector S that runs on the input to decide which heuristic should be executed to solve the problem. Hybrid algorithms are of particular interest in scenarios where the selector must decide between heuristics that are "good" with respect to different complexity measures.We focus on hybrid algorithms with a "hardness-defying" property: for a problem II, there is a set of complexity measures {m i } whereby II is known or conjectured to be unsolvable for each m i , but for each heuristic h i of the hybrid algorithm, one can give a complexity guarantee for h i on the instances of II that S selects for h i that is strictly better than m i . More concretely, we show that for several NP-hard problems, a given instance can either be solved exactly with substantially improved runtime (e.g. 2o(n)), or be approximated in polynomial time with an approximation ratio exceeding that of the known or conjectured inapproximability of the problem, assuming P ≠ NP.
---
paper_title: The Algorithm Selection Problem

paper_content:

Publisher Summary The problem of selecting an effective algorithm arises in a wide variety of situations. This chapter starts with a discussion on abstract models: the basic model and associated problems, the model with selection based on features, and the model with variable performance criteria. One objective of this chapter is to explore the applicability of the approximation theory to the algorithm selection problem. There is an intimate relationship here and that the approximation theory forms an appropriate base upon which to develop a theory of algorithm selection methods. The approximation theory currently lacks much of the necessary machinery for the algorithm selection problem. There is a need to develop new results and apply known techniques to these new circumstances. The final pages of this chapter form a sort of appendix, which lists 15 specific open problems and questions in this area. There is a close relationship between the algorithm selection problem and the general optimization theory. This is not surprising since the approximation problem is a special form of the optimization problem. Most realistic algorithm selection problems are of moderate to high dimensionality and thus one should expect them to be quite complex. One consequence of this is that most straightforward approaches (even well-conceived ones) are likely to lead to enormous computations for the best selection. The single most important part of the solution of a selection problem is the appropriate choice of the form for selection mapping. It is here that theories give the least guidance and that the art of problem solving is most crucial.
---
paper_title: An Attempt to Map the Performance of a Range of Algorithm and Heuristic Combinations

paper_content:

Constraint satisfaction is the core of many AI and real life problems and much research has been done in this field in recent years. Work has been done in the past in comparing the performance of different algorithms and heuristics. Much of such work has focused on finding "the best" algorithm and heuristic combination for all problems. The objective of this paper is to prove that there is no universally best algorithm and heuristic for all problems -different problems can be solved most efficiently by different algorithm and heuristic combinations. The implication of this is important because it means that instead of trying to find "the best" algorithms and heuristics, future research should try to identify the application domain of each algorithm and heuristic (i.e. when they are most effective). Furthermore our results point to future research which focuses on how to retrieve the most efficient algorithm for a given problem. The results in this paper provide a first step towards achieving such goals. * The research reported in this paper was supported by the University of Essex research promotion fund ref. R7027 and by the EPSRC research grant ref. GR/J/42878.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: The Algorithm Selection Problem

paper_content:

Publisher Summary The problem of selecting an effective algorithm arises in a wide variety of situations. This chapter starts with a discussion on abstract models: the basic model and associated problems, the model with selection based on features, and the model with variable performance criteria. One objective of this chapter is to explore the applicability of the approximation theory to the algorithm selection problem. There is an intimate relationship here and that the approximation theory forms an appropriate base upon which to develop a theory of algorithm selection methods. The approximation theory currently lacks much of the necessary machinery for the algorithm selection problem. There is a need to develop new results and apply known techniques to these new circumstances. The final pages of this chapter form a sort of appendix, which lists 15 specific open problems and questions in this area. There is a close relationship between the algorithm selection problem and the general optimization theory. This is not surprising since the approximation problem is a special form of the optimization problem. Most realistic algorithm selection problems are of moderate to high dimensionality and thus one should expect them to be quite complex. One consequence of this is that most straightforward approaches (even well-conceived ones) are likely to lead to enormous computations for the best selection. The single most important part of the solution of a selection problem is the appropriate choice of the form for selection mapping. It is here that theories give the least guidance and that the art of problem solving is most crucial.
---
paper_title: Ensemble Methods in Machine Learning

paper_content:

Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly.
---
paper_title: An Economics Approach to Hard Computational Problems

paper_content:

A general method for combining existing algorithms into new programs that are unequivocally preferable to any of the component algorithms is presented. This method, based on notions of risk in economics, offers a computational portfolio design procedure that can be used for a wide range of problems. Tested by solving a canonical NPcompleteproblem,themethodcanbeusedforproblemsrangingfromthecombinatorics of DNA sequencing to the completion of tasks in environments with resource contention, such as the World Wide Web.
---
paper_title: Isac – instance-specific algorithm configuration

paper_content:

We present a new method for instance-specific algorithm configuration (ISAC). It is based on the integration of the algorithm configuration system GGA and the recently proposed stochastic offline programming paradigm. ISAC is provided a solver with categorical, ordinal, and/or continuous parameters, a training benchmark set of input instances for that solver, and an algorithm that computes a feature vector that characterizes any given instance. ISAC then provides high quality parameter settings for any new input instance. Experiments on a variety of different constrained optimization and constraint satisfaction solvers show that automatic algorithm configuration vastly outperforms manual tuning. Moreover, we show that instance-specific tuning frequently leads to significant speed-ups over instance-oblivious configurations.
---
paper_title: SATzilla-07: The design and analysis of an algorithm portfolio for SAT

paper_content:

It has been widely observed that there is no "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe a per-instance solver portfolio for SAT, SATzilla-07, which uses socalled empirical hardness models to choose among its constituent solvers. We leverage new model-building techniques such as censored sampling and hierarchical hardness models, and demonstrate the effectiveness of our techniques by building a portfolio of state-of-the-art SAT solvers and evaluating it on several widely-studied SAT data sets. Overall, we show that our portfolio significantly outperforms its constituent algorithms on every data set. Our approach has also proven itself to be effective in practice: in the 2007 SAT competition, SATzilla-07 won three gold medals, one silver, and one bronze; it is available online at http://www.cs.ubc.ca/labs/beta/Projects/SATzilla.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: Instance-Based Selection of Policies for SAT Solvers

paper_content:

Execution of most of the modern DPLL-based SAT solvers is guided by a number of heuristics. Decisions made during the search process are usually driven by some fixed heuristic policies. Despite the outstanding progress in SAT solving in recent years, there is still an appealing lack of techniques for selecting policies appropriate for solving specific input formulae. In this paper we present a methodology for instance-based selection of solver's policies that uses a data-mining classification technique. The methodology also relies on analysis of relationships between formulae, their families, and their suitable solving strategies. The evaluation results are very good, demonstrate practical usability of the methodology, and encourage further efforts in this direction.
---
paper_title: Groups of diverse problem solvers can outperform groups of high-ability problem solvers

paper_content:

We introduce a general framework for modeling functionally diverse problem-solving agents. In this framework, problem-solving agents possess representations of problems and algorithms that they use to locate solutions. We use this framework to establish a result relevant to group composition. We find that when selecting a problem-solving team from a diverse population of intelligent agents, a team of randomly selected agents outperforms a team comprised of the best-performing agents. This result relies on the intuition that, as the initial pool of problem solvers becomes large, the best-performing agents necessarily become similar in the space of problem solvers. Their relatively greater ability is more than offset by their lack of problem-solving diversity.
---
paper_title: K.: Evaluating component solver contributions to portfolio-based algorithm selectors

paper_content:

Portfolio-based methods exploit the complementary strengths of a set of algorithms and--as evidenced in recent competitions--represent the state of the art for solving many NP-hard problems, including SAT. In this work, we argue that a state-of-the-art method for constructing portfolio-based algorithm selectors, $\texttt{SATzilla}$, also gives rise to an automated method for quantifying the importance of each of a set of available solvers. We entered a substantially improved version of $\texttt{SATzilla}$ to the inaugural "analysis track" of the 2011 SAT competition, and draw two main conclusions from the results that we obtained. First, automatically-constructed portfolios of sequential, non-portfolio competition entries perform substantially better than the winners of all three sequential categories. Second, and more importantly, a detailed analysis of these portfolios yields valuable insights into the nature of successful solver designs in the different categories. For example, we show that the solvers contributing most to $\texttt{SATzilla}$ were often not the overall best-performing solvers, but instead solvers that exploit novel solution strategies to solve instances that would remain unsolved without them.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: Understanding Random SAT: Beyond the Clauses-to-Variables Ratio

paper_content:

It is well known that the ratio of the number of clauses to the number of variables in a random k-SAT instance is highly correlated with the instance's empirical hardness. We consider the problem of identifying such features of random SAT instances automatically using machine learning. We describe and analyze models for three SAT solvers - kcnfs, oksolver and satz- and for two different distributions of instances: uniform random 3-SAT with varying ratio of clauses-to-variables, and uniform random 3-SAT with fixed ratio of clauses-to-variables. We show that surprisingly accurate models can be built in all cases. Furthermore, we analyze these models to determine which features are most useful in predicting whether an instance will be hard to solve. Finally we discuss the use of our models to build SATzilla, an algorithm portfolio for SAT 1 .
---
paper_title: A self-adaptive multi-engine solver for quantified Boolean formulas

paper_content:

In this paper we study the problem of engineering a robust solver for quantified Boolean formulas (QBFs), i.e., a tool that can efficiently solve formulas across different problem domains without the need for domain-specific tuning. The paper presents two main empirical results along this line of research. Our first result is the development of a multi-engine solver, i.e., a tool that selects among its reasoning engines the one which is more likely to yield optimal results. In particular, we show that syntactic QBF features can be correlated to the performances of existing QBF engines across a variety of domains. We also show how a multi-engine solver can be obtained by carefully picking state-of-the-art QBF solvers as basic engines, and by harnessing inductive reasoning techniques to learn engine-selection policies. Our second result is the improvement of our multi-engine solver with the capability of updating the learned policies when they fail to give good predictions. In this way the solver becomes also self-adaptive, i.e., able to adjust its internal models when the usage scenario changes substantially. The rewarding results obtained in our experiments show that our solver AQME--Adaptive QBF Multi-Engine--can be more robust and efficient than state-of-the-art single-engine solvers, even when it is confronted with previously uncharted formulas and competitors.
---
paper_title: An Economics Approach to Hard Computational Problems

paper_content:

A general method for combining existing algorithms into new programs that are unequivocally preferable to any of the component algorithms is presented. This method, based on notions of risk in economics, offers a computational portfolio design procedure that can be used for a wide range of problems. Tested by solving a canonical NPcompleteproblem,themethodcanbeusedforproblemsrangingfromthecombinatorics of DNA sequencing to the completion of tasks in environments with resource contention, such as the World Wide Web.
---
paper_title: Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

paper_content:

The AI community has achieved great success in designing high-performance algorithms for hard combinatorial problems, given both considerable domain knowledge and considerable effort by human experts. Two influential methods aim to automate this process: automated algorithm configuration and portfolio-based algorithm selection. The former has the advantage of requiring virtually no domain knowledge, but produces only a single solver; the latter exploits per-instance variation, but requires a set of relatively uncorrelated candidate solvers. Here, we introduce Hydra, a novel technique for combining these two methods, thereby realizing the benefits of both. Hydra automatically builds a set of solvers with complementary strengths by iteratively configuring new algorithms. It is primarily intended for use in problem domains for which an adequate set of candidate solvers does not already exist. Nevertheless, we tested Hydra on a widely studied domain, stochastic local search algorithms for SAT, in order to characterize its performance against a well-established and highly competitive baseline. We found that Hydra consistently achieved major improvements over the best existing individual algorithms, and always at least roughly matched—and indeed often exceeded— the performance of the best portfolios of these algorithms.
---
paper_title: Learning to solve QBF

paper_content:

We present a novel approach to solving Quantified Boolean Formulas (QBF) that combines a search-based QBF solver with marhine learning techniques. We show how classification methods can be used to predict run-times and to choose optimal heuristics both within a portfolio-based, and within a dynamic, online approach. In the dynamic method variables are set to a truth value according to a scheme that tries to maximize the probability of successfully solving the remaining sub-problem efficiently. Since each variable assignment can drastically change the problem-structure, new heuristics are chosen dynamically, and a classifier is used online to predict the usefulness of each heuristic. Experimental results on a large corpus of example problems show the usefulness of our approach in terms of run-time as well as the ability to solve previously unsolved problem instances.
---
paper_title: Integrating Heuristics for Constraint Satisfaction Problems: A Case Study

paper_content:

This paper describes a set of experiments with a system that synthesizes constraint satisfaction programs. The system, MULTI-TAC, is a CSP "expert" that can specialize a library of generic algorithms and methods for a particular application. MULTI-TAC not only proposes domain-specific versions of its generic heuristics, but also searches for the best combination of these heuristics and integrates them into a complete problem-specific program. We demonstrate MULTI-TAC's capabilities on a combinatorial problem, "Minimum Maximal Matching", and show that MULTI-TAC can synthesize programs for this problem that are on par with hand-coded programs. In synthesizing a program, MULTI-TAC bases its choice of heuristics on the instance distribution, and we show that this capability has a significant impact on the results.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: A Bayesian Approach to Tackling Hard Computational Problems

paper_content:

We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the time required to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We review the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions.
---
paper_title: Automated Discovery of Composite SAT Variable-Selection Heuristics

paper_content:

Variants of GSAT and Walksat are among the most successful SAT local search algorithms. We show that several well-known SAT local search algorithms are the result of novel combininations of a set of variable selection primitives. We describe CLASS, an automated heuristic discovery system which generates new, effective variable selection heuristic functions using a simple composition operator. New heuristics discovered by CLASS are shown to be competitive with the best Walksat variants, including Novelty+ and R-Novelty+. We also analyze the local search behavior of the learned heuristics using the depth, mobility, and coverage metrics recently proposed by Schuurmans and Southey.
---
paper_title: Knowledge-based constraint-driven software synthesis

paper_content:

A constraint-driven, real-time software synthesis architecture called RT-Syn is described. RT-Syn formulates design space constraints for each task in the real-time software system from timing requirements and a behavior description. RT-Syn then uses these constraints within a simulated-annealing-like approach, selects an abstract implementation for every data structure and algorithm required to implement the desired behavior, and transforms these selections into executable code. Experimental results covering the synthesis of two real-time software tasks that meet the desired constraints are presented. These results illustrate the effectiveness of the simulated-annealing-like approach in searching the software design space and the high reusability and maintainability provided by the use of synthesis technology. >
---
paper_title: Collaborative Learning for Constraint Solving

paper_content:

Although constraint programming offers a wealth of strong, generalpurpose methods, in practice a complex, real application demands a person who selects, combines, and refines various available techniques for constraint satisfaction and optimization. Although such tuning produces efficient code, the scarcity of human experts slows commercialization. The necessary expertise is of two forms: constraint programming expertise and problem-domain expertise. The former is in short supply, and even experts can be reduced to trial and error prototyping; the latter is difficult to extract. The project described here seeks to automate both the application of constraint programming expertise and the extraction of domain-specific expertise. It applies FORR, an architecture for learning and problem-solving, to constraint solving. FORR develops expertise from multiple heuristics. A successful case study is presented on coloring problems.
---
paper_title: The Adaptive Constraint Engine

paper_content:

The Adaptive Constraint Engine (ACE) seeks to automate the application of constraint programming expertise and the extraction of domain-specific expertise. Under the aegis of FORR, an architecture for learning and problem-solving, ACE learns search-order heuristics from problem solving experience. This paper describes ACE?s approach, as well as new experimental results on specific problem classes. ACE is both a test-bed for CSP research and a discovery environment for new algorithms.
---
paper_title: An Analytic Learning System for Specializing Heuristics

paper_content:

This paper describes how meta-level theories are used for analytic learning in MULTI-TAC. MULTI-TAC operationalizes generic heuristics for constraint-satisfaction problems, in order to create programs that are tailored to specific problems. For each of its generic heuristics, MULTI-TAC has a meta-theory specifically designed for operationalising that heuristic. We present examples of the specialisation process and discuss how the theories influence the tractability of the learning process. We also describe an empirical study showing that the specialised programs produced by MULTITAC compare favorably to hand-coded programs.
---
paper_title: Learning Rules for Adaptive Planning

paper_content:

This paper presents a novel idea, which combines Planning, Machine Learning and Knowledge-Based techniques. It is concerned with the development of an adaptive planning system that can fine-tune its planning parameters based on the values of specific measurable characteristics of the given planning problem. Adaptation is guided by a rule-based system, whose knowledge has been acquired through machine learning techniques. Specifically, the algorithm of classification based on association rules was applied to a large dataset produced by results from experiments on a large number of problems used in the three AIPS Planning competitions. The paper presents experimental results with the adaptive planner, which demonstrate the boost in performance of the planning system.
---
paper_title: Learning Effective Search Heuristics

paper_content:

SAGE.2 is a production system that improves its search strategies with practice. The program incorporates four different heuristics for assigning credit and blame, and employs a discrimination process to direct its search through the space of move-proposing rules. The system has shown its generality by learning search heuristics in five different task domains. In addition to improving its search behavior on practice problems, SAGE.2 was able to transfer its expertise to scaled-up versions of a task, and in one case transferred its acquired search strategy to problems with different initial and goal states.
---
paper_title: Automated Discovery of Local Search Heuristics for Satisfiability Testing

paper_content:

The development of successful metaheuristic algorithms such as local search for a difficult problem such as satisfiability testing (SAT) is a challenging task. We investigate an evolutionary approach to automating the discovery of new local search heuristics for SAT. We show that several well-known SAT local search algorithms such as Walksat and Novelty are composite heuristics that are derived from novel combinations of a set of building blocks. Based on this observation, we developed CLASS, a genetic programming system that uses a simple composition operator to automatically discover SAT local search heuristics. New heuristics discovered by CLASS are shown to be competitive with the best Walksat variants, including Novelty+. Evolutionary algorithms have previously been applied to directly evolve a solution for a particular SAT instance. We show that the heuristics discovered by CLASS are also competitive with these previous, direct evolutionary approaches for SAT. We also analyze the local search behavior of the learned heuristics using the depth, mobility, and coverage metrics proposed by Schuurmans and Southey.
---
paper_title: An automated approach to generating efficient constraint solvers

paper_content:

Combinatorial problems appear in numerous settings, from timetabling to industrial design. Constraint solving aims to find solutions to such problems efficiently and automatically. Current constraint solvers are monolithic in design, accepting a broad range of problems. The cost of this convenience is a complex architecture, inhibiting efficiency, extensibility and scalability. Solver components are also tightly coupled with complex restrictions on their configuration, making automated generation of solvers difficult. We describe a novel, automated, model-driven approach to generating efficient solvers tailored to individual problems and present some results from applying the approach. The main contribution of this work is a solver generation framework called Dominion, which analyses a problem and, based on its characteristics, generates a solver using components chosen from a library. The key benefit of this approach is the ability to solve larger and more difficult problems as a result of applying finer-grained optimisations and using specialised techniques as required.
---
paper_title: Isac – instance-specific algorithm configuration

paper_content:

We present a new method for instance-specific algorithm configuration (ISAC). It is based on the integration of the algorithm configuration system GGA and the recently proposed stochastic offline programming paradigm. ISAC is provided a solver with categorical, ordinal, and/or continuous parameters, a training benchmark set of input instances for that solver, and an algorithm that computes a feature vector that characterizes any given instance. ISAC then provides high quality parameter settings for any new input instance. Experiments on a variety of different constrained optimization and constraint satisfaction solvers show that automatic algorithm configuration vastly outperforms manual tuning. Moreover, we show that instance-specific tuning frequently leads to significant speed-ups over instance-oblivious configurations.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: Automated Discovery of Composite SAT Variable-Selection Heuristics

paper_content:

Variants of GSAT and Walksat are among the most successful SAT local search algorithms. We show that several well-known SAT local search algorithms are the result of novel combininations of a set of variable selection primitives. We describe CLASS, an automated heuristic discovery system which generates new, effective variable selection heuristic functions using a simple composition operator. New heuristics discovered by CLASS are shown to be competitive with the best Walksat variants, including Novelty+ and R-Novelty+. We also analyze the local search behavior of the learned heuristics using the depth, mobility, and coverage metrics recently proposed by Schuurmans and Southey.
---
paper_title: T.: Automatic algorithm configuration based on local search

paper_content:

The determination of appropriate values for free algorithm parameters is a challenging and tedious task in the design of effective algorithms for hard problems. Such parameters include categorical choices (e.g., neighborhood structure in local search or variable/value ordering heuristics in tree search), as well as numerical parameters (e.g., noise or restart timing). In practice, tuning of these parameters is largely carried out manually by applying rules of thumb and crude heuristics, while more principled approaches are only rarely used. In this paper, we present a local search approach for algorithm configuration and prove its convergence to the globally optimal parameter configuration. Our approach is very versatile: it can, e.g., be used for minimising run-time in decision problems or for maximising solution quality in optimisation problems. It further applies to arbitrary algorithms, including heuristic tree search and local search algorithms, with no limitation on the number of parameters. Experiments in four algorithm configuration scenarios demonstrate that our automatically determined parameter settings always outperform the algorithm defaults, sometimes by several orders of magnitude. Our approach also shows better performance and greater flexibility than the recent CALIBRA system. Our ParamILS code, along with instructions on how to use it for tuning your own algorithms, is available on-line at http://www.cs.ubc.ca/labs/beta/Projects/ParamILS.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: ParamILS: an automatic algorithm configuration framework

paper_content:

The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.
---
paper_title: Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

paper_content:

The AI community has achieved great success in designing high-performance algorithms for hard combinatorial problems, given both considerable domain knowledge and considerable effort by human experts. Two influential methods aim to automate this process: automated algorithm configuration and portfolio-based algorithm selection. The former has the advantage of requiring virtually no domain knowledge, but produces only a single solver; the latter exploits per-instance variation, but requires a set of relatively uncorrelated candidate solvers. Here, we introduce Hydra, a novel technique for combining these two methods, thereby realizing the benefits of both. Hydra automatically builds a set of solvers with complementary strengths by iteratively configuring new algorithms. It is primarily intended for use in problem domains for which an adequate set of candidate solvers does not already exist. Nevertheless, we tested Hydra on a widely studied domain, stochastic local search algorithms for SAT, in order to characterize its performance against a well-established and highly competitive baseline. We found that Hydra consistently achieved major improvements over the best existing individual algorithms, and always at least roughly matched—and indeed often exceeded— the performance of the best portfolios of these algorithms.
---
paper_title: An automated approach to generating efficient constraint solvers

paper_content:

Combinatorial problems appear in numerous settings, from timetabling to industrial design. Constraint solving aims to find solutions to such problems efficiently and automatically. Current constraint solvers are monolithic in design, accepting a broad range of problems. The cost of this convenience is a complex architecture, inhibiting efficiency, extensibility and scalability. Solver components are also tightly coupled with complex restrictions on their configuration, making automated generation of solvers difficult. We describe a novel, automated, model-driven approach to generating efficient solvers tailored to individual problems and present some results from applying the approach. The main contribution of this work is a solver generation framework called Dominion, which analyses a problem and, based on its characteristics, generates a solver using components chosen from a library. The key benefit of this approach is the ability to solve larger and more difficult problems as a result of applying finer-grained optimisations and using specialised techniques as required.
---
paper_title: Combining multiple heuristics on discrete resources

paper_content:

In this work we study the portfolio problem which is to find a good combination of multiple heuristics to solve given instances on parallel resources in minimum time. The resources are assumed to be discrete, it is not possible to allocate a resource to more than one heuristic. Our goal is to minimize the average completion time of the set of instances, given a set of heuristics on homogeneous discrete resources. This problem has been studied in the continuous case in [13]. We first show that the problem is hard and that there is no constant ratio polynomial approximation unless P = NP in the general case. Then, we design several approximation schemes for a restricted version of the problem where each heuristic must be used at least once. These results are obtained by using oracle with several guesses, leading to various tradeoff between the size of required information and the approximation ratio. Some additional results based on simulations are finally reported using a benchmark of instances on SAT solvers.
---
paper_title: Learning parallel portfolios of algorithms

paper_content:

A wide range of combinatorial optimization algorithms have been developed for complex reasoning tasks. Frequently, no single algorithm outperforms all the others. This has raised interest in leveraging the performance of a collection of algorithms to improve performance. We show how to accomplish this using a Parallel Portfolio of Algorithms (PPA). A PPA is a collection of diverse algorithms for solving a single problem, all running concurrently on a single processor until a solution is produced. The performance of the portfolio may be controlled by assigning different shares of processor time to each algorithm. We present an effective method for finding a PPA in which the share of processor time allocated to each algorithm is fixed. Finding the optimal static schedule is shown to be an NP-complete problem for a general class of utility functions. We present bounds on the performance of the PPA over random instances and evaluate the performance empirically on a collection of 23 state-of-the-art SAT algorithms. The results show significant performance gains over the fastest individual algorithm in the collection.
---
paper_title: Genetic algorithm portfolios

paper_content:

Comparative studies of sets of control parameter values are commonly performed when tuning an evolutionary algorithm for a class of problem instances. The standard approach is to identify the most useful set of control parameter settings for a domain. In this paper, we propose an alternative anytime algorithm portfolio technique in which computational resources are allocated among multiple sets of control parameter value settings. We show a method of optimizing such portfolios by applying a bootstrap sampling approach to a database of individual algorithm performance on instances from a problem distribution. Experiments with genetic algorithms applied to the traveling salesperson domain show that the portfolio approach can yield better performance on a distribution of problem instances than the standard approach of trying to identify the single best configuration for the problem class.
---
paper_title: Restart schedules for ensembles of problem instances

paper_content:

The mean running time of a Las Vegas algorithm can often be dramatically reduced by periodically restarting it with a fresh random seed. The optimal restart schedule depends on the Las Vegas algorithm's run length distribution, which in general is not known in advance and may differ across problem instances. We consider the problem of selecting a single restart schedule to use in solving each instance in a set of instances. We present offline algorithms for computing an (approximately) optimal restart schedule given knowledge of each instance's run length distribution, generalization bounds for learning a restart schedule from training data, and online algorithms for selecting a restart schedule adaptively as new problem instances are encountered.
---
paper_title: A Bayesian Approach to Tackling Hard Computational Problems

paper_content:

We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the time required to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We review the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions.
---
paper_title: Algorithm Selection and Scheduling

paper_content:

Algorithm portfolios aim to increase the robustness of our ability to solve problems efficiently. While recently proposed algorithm selection methods come ever closer to identifying the most appropriate solver given an input instance, they are bound to make wrong and, at times, costly decisions. Solver scheduling has been proposed to boost the performance of algorithm selection. Scheduling tries to allocate time slots to the given solvers in a portfolio so as to maximize, say, the number of solved instances within a given time limit. We show how to solve the corresponding optimization problem at a low computational cost using column generation, resulting in fast and high quality solutions. We integrate this approach with a recently introduced algorithm selector, which we also extend using other techniques. We propose various static as well as dynamic scheduling strategies, and demonstrate that in comparison to pure algorithm selection, our novel combination of scheduling and solver selection can significantly boost performance.
---
paper_title: On Portfolios for Backtracking Search in the Presence of Deadlines

paper_content:

Constraint satisfaction and prepositional satisfiability problems are often solved using backtracking search. Previous studies have shown that portfolios of backtracking algorithms - a selection of one or more algorithms plus a schedule for executing the algorithms - can dramatically improve performance on some instances. In this paper, we consider a setting that often arises in practice where the instances to be solved arise over time, the instances all belong to some class of problem instances, and a limit or deadline is placed on the computational resources that the backtracking algorithm can consume in solving any instance. For such a scenario, we present a simple scheme for learning a good portfolio of backtracking algorithms from a small sample of instances. We demonstrate the effectiveness of our approach through an extensive empirical evaluation on a real-world instruction scheduling testbed.
---
paper_title: Instance-Based Selection of Policies for SAT Solvers

paper_content:

Execution of most of the modern DPLL-based SAT solvers is guided by a number of heuristics. Decisions made during the search process are usually driven by some fixed heuristic policies. Despite the outstanding progress in SAT solving in recent years, there is still an appealing lack of techniques for selecting policies appropriate for solving specific input formulae. In this paper we present a methodology for instance-based selection of solver's policies that uses a data-mining classification technique. The methodology also relies on analysis of relationships between formulae, their families, and their suitable solving strategies. The evaluation results are very good, demonstrate practical usability of the methodology, and encourage further efforts in this direction.
---
paper_title: New Techniques for Algorithm Portfolio Design

paper_content:

We present and evaluate new techniques for designing algorithm portfolios. In our view, the problem has both a scheduling aspect and a machine learning aspect. Prior work has largely addressed one of the two aspects in isolation. Building on recent work on the scheduling aspect of the problem, we present a technique that addresses both aspects simultaneously and has attractive theoretical guarantees. Experimentally, we show that this technique can be used to improve the performance of state-of-the-art algorithms for Boolean satisfiability, zero-one integer programming, and A.I. planning.
---
paper_title: Self-Adapting Linear Algebra Algorithms and Software

paper_content:

One of the main obstacles to the efficient solution of scientific problems is the problem of tuning software, both to the available architecture and to the user problem at hand. We describe approaches for obtaining tuned high-performance kernels and for automatically choosing suitable algorithms. Specifically, we describe the generation of dense and sparse Basic Linear Algebra Subprograms (BLAS) kernels, and the selection of linear solver algorithms. However, the ideas presented here extend beyond these areas, which can be considered proof of concept.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: An automatically configurable portfolio-based planner with macroactions: PbP

paper_content:

While several powerful domain-independent planners have recently been developed, no one of these clearly outperforms all the others in every known benchmark domain. We present PbP, a multi-planner which automatically configures a portfolio of planners by (i) computing some sets of macro-actions for every planner in the portfolio, (ii) selecting a promising combination of planners in the portfolio and relative useful macro-actions, and (iii) defining some running time slots for their round-robin scheduling during planning. The configuration relies on some knowledge about the performance of the planners in the portfolio and relative macro-actions which is automatically generated from a training problem set. PbP entered the learning track of IPC-2008 and was the overall winner of this competition track. An experimental study confirms the effectiveness of PbP, and shows that the learned configuration knowledge is useful for PbP.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: Understanding Random SAT: Beyond the Clauses-to-Variables Ratio

paper_content:

It is well known that the ratio of the number of clauses to the number of variables in a random k-SAT instance is highly correlated with the instance's empirical hardness. We consider the problem of identifying such features of random SAT instances automatically using machine learning. We describe and analyze models for three SAT solvers - kcnfs, oksolver and satz- and for two different distributions of instances: uniform random 3-SAT with varying ratio of clauses-to-variables, and uniform random 3-SAT with fixed ratio of clauses-to-variables. We show that surprisingly accurate models can be built in all cases. Furthermore, we analyze these models to determine which features are most useful in predicting whether an instance will be hard to solve. Finally we discuss the use of our models to build SATzilla, an algorithm portfolio for SAT 1 .
---
paper_title: Combining multiple heuristics

paper_content:

In this work we introduce and study the question of combining multiple heuristics. Given a problem instance, each of the multiple heuristics is capable of computing the correct solution, but has a different cost. In our models the user executes multiple heuristics until one of them terminates with a solution. Given a set of problem instances, we show how to efficiently compute an optimal fixed schedule for a constant number of heuristics, and show that in general, the problem is computationally hard even to approximate (to within a constant factor). We also discuss a probabilistic configuration, in which the problem instances are drawn from some unknown fixed distribution, and show how to compute a near optimal schedule for this setup.
---
paper_title: An Economics Approach to Hard Computational Problems

paper_content:

A general method for combining existing algorithms into new programs that are unequivocally preferable to any of the component algorithms is presented. This method, based on notions of risk in economics, offers a computational portfolio design procedure that can be used for a wide range of problems. Tested by solving a canonical NPcompleteproblem,themethodcanbeusedforproblemsrangingfromthecombinatorics of DNA sequencing to the completion of tasks in environments with resource contention, such as the World Wide Web.
---
paper_title: Towards distributed algorithm portfolios

paper_content:

In recent work we have developed an online algorithm selection technique, in which a model of algorithm performance is learned incrementally while being used. The resulting exploration-exploitation trade-off is solved as a bandit problem. The candidate solvers are run in parallel on a single machine, as an algorithm portfolio, and computation time is shared among them according to their expected performances. In this paper, we extend our technique to the more interesting and practical case of multiple CPUs.
---
paper_title: The max k-armed bandit: A new model of exploration applied to search heuristic selection

paper_content:

The multiarmed bandit is often used as an analogy for the tradeoff between exploration and exploitation in search problems. The classic problem involves allocating trials to the arms of a multiarmed slot machine to maximize the expected sum of rewards. We pose a new variation of the multiarmed bandit--the Max K-Armed Bandit--in which trials must be allocated among the arms to maximize the expected best single sample reward of the series of trials. Motivation for the Max K-Armed Bandit is the allocation of restarts among a set of multistart stochastic search algorithms. We present an analysis of this Max K-Armed Bandit showing under certain assumptions that the optimal strategy allocates trials to the observed best arm at a rate increasing double exponentially relative to the other arms. This motivates an exploration strategy that follows a Boltzmann distribution with an exponentially decaying temperature parameter. We compare this exploration policy to policies that allocate trials to the observed best arm at rates faster (and slower) than double exponentially. The results confirm, for two scheduling domains, that the double exponential increase in the rate of allocations to the observed best heuristic outperfonns the other approaches.
---
paper_title: Combining multiple heuristics online

paper_content:

We present black-box techniques for learning how to interleave the execution of multiple heuristics in order to improve average-case performance. In our model, a user is given a set of heuristics whose only observable behavior is their running time. Each heuristic can compute a solution to any problem instance, but its running time varies across instances. The user solves each instance by interleaving runs of the heuristics according to a task-switching schedule. We present (i) exact and approximation algorithms for computing an optimal task-switching schedule offline, (ii) sample complexity bounds for learning a task-switching schedule from training data, and (iii) a no-regret strategy for selecting task-switching schedules online. We demonstrate the power of our results using data from recent solver competitions. We outline how to extend our results to the case in which the heuristics are randomized, and the user may periodically restart each heuristic with a fresh random seed.
---
paper_title: Learning Algorithm Portfolios for Parallel Execution

paper_content:

Portfolio-based solvers are both effective and robust, but their promise for parallel execution with constraint satisfaction solvers has received relatively little attention. This paper proposes an approach that constructs algorithm portfolios intended for parallel execution based on a combination of case-based reasoning, a greedy algorithm, and three heuristics. Empirical results show that this method is efficient, and can significantly improve performance with only a few additional processors. On problems from solver competitions, the resultant algorithm portfolios perform nearly as well as an oracle.
---
paper_title: Adaptive Online Time Allocation to Search Algorithms

paper_content:

Given is a search problem or a sequence of search problems, as well as a set of potentially useful search algorithms. We propose a general framework for online allocation of computation time to search algorithms based on experience with their performance so far. In an example instantiation, we use simple linear extrapolation of performance for allocating time to various simultaneously running genetic algorithms characterized by different parameter values. Despite the large number of searchers tested in parallel, on various tasks this rather general approach compares favorably to a more specialized state-of-the-art heuristic; in one case it is nearly two orders of magnitude faster.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: A Bayesian Approach to Tackling Hard Computational Problems

paper_content:

We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the time required to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We review the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions.
---
paper_title: Statistical Selection Among Problem-Solving Methods

paper_content:

Abstract : The choice of an appropriate problem solving method, from available methods, is a crucial skill for human experts in many areas. We describe a technique for automatic selection among methods, based on a statistical analysis of their past performances. We formalize the statistical problem involved in selecting an efficient problem solving method, derive a solution to this problem, and describe a selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We extend our basic statistical technique to account for problem sizes and for similarity between problems. We give empirical results of the use of this technique to select among search engines in the PRODIGY system. We also test the selection technique on artificially generated performance data, using several different probability distributions.
---
paper_title: How to solve it automatically: Selection among problem-solving methods

paper_content:

The choice of an appropriate problem-solving method, from available methods, is a crucial skill for experts in many areas. We describe a technique for the automatic selection among methods, which is based on a statistical analysis of their past performances. ::: ::: We formalize the statistical problem involved in selecting an efficient problem-solving method, derive a solution to this problem, and describe a method-selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We give empirical results on the use of this technique in selecting among search engines in the PRODIGY planning system.
---
paper_title: Knowledge-based constraint-driven software synthesis

paper_content:

A constraint-driven, real-time software synthesis architecture called RT-Syn is described. RT-Syn formulates design space constraints for each task in the real-time software system from timing requirements and a behavior description. RT-Syn then uses these constraints within a simulated-annealing-like approach, selects an abstract implementation for every data structure and algorithm required to implement the desired behavior, and transforms these selections into executable code. Experimental results covering the synthesis of two real-time software tasks that meet the desired constraints are presented. These results illustrate the effectiveness of the simulated-annealing-like approach in searching the software design space and the high reusability and maintainability provided by the use of synthesis technology. >
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: PRODIGY: an integrated architecture for planning and learning

paper_content:

Artificial intelligence has progressed to the point where multiple cognitive capabilities are being integrated into computational architectures, such as SOAR, PRODIGY, THEO, and ICARUS. This paper reports on the PRODIGY architecture, describing its planning and problem solving capabilities and touching upon its multiple learning methods. Learning in PRODIGY occurs at all decision points and integration in PRODIGY is at the knowledge level; the learning and reasoning modules produce mutually interpretable knowledge structures. Issues in architectural design are discussed, providing a context to examine the underlying tenets of the PRODIGY architecture.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: An adaptive algorithm selection framework

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. We present an adaptive algorithm selection framework which can collect and interpret the inputs of a particular instance of a parallel algorithm and select the best performing one from a an existing library. In this paper present the dynamic selection of parallel reduction algorithms. First we introduce a set of high-level parameters that can characterize different parallel reduction algorithms. Then we describe an off-line, systematic process to generate predictive models which can be used for run-time algorithm selection. Our experiments show that our framework: (a) selects the most appropriate algorithms in 85% of the cases studied, (b) overall delievers 98% of the optimal performance, (c) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and (d) adapts to the underlying machine architecture (tested on IBM Regatta and HP V-Class systems).
---
paper_title: A self-adaptive multi-engine solver for quantified Boolean formulas

paper_content:

In this paper we study the problem of engineering a robust solver for quantified Boolean formulas (QBFs), i.e., a tool that can efficiently solve formulas across different problem domains without the need for domain-specific tuning. The paper presents two main empirical results along this line of research. Our first result is the development of a multi-engine solver, i.e., a tool that selects among its reasoning engines the one which is more likely to yield optimal results. In particular, we show that syntactic QBF features can be correlated to the performances of existing QBF engines across a variety of domains. We also show how a multi-engine solver can be obtained by carefully picking state-of-the-art QBF solvers as basic engines, and by harnessing inductive reasoning techniques to learn engine-selection policies. Our second result is the improvement of our multi-engine solver with the capability of updating the learned policies when they fail to give good predictions. In this way the solver becomes also self-adaptive, i.e., able to adjust its internal models when the usage scenario changes substantially. The rewarding results obtained in our experiments show that our solver AQME--Adaptive QBF Multi-Engine--can be more robust and efficient than state-of-the-art single-engine solvers, even when it is confronted with previously uncharted formulas and competitors.
---
paper_title: An Adaptive Algorithm Selection Framework for Reduction Parallelization

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. In this paper, we present an adaptive algorithm selection framework which can collect and interpret the characteristics of a particular instance of parallel reduction algorithms and select the best performing one from an existing library. The framework consists of the following components: 1) an offline systematic process for characterizing the input sensitivity of parallel reduction algorithms and a method for building corresponding predictive performance models, 2) an online input characterization and algorithm selection module, and 3) a small library of parallel reduction algorithms, which represent the algorithmic choices made available at runtime. We also present one possible integration of this framework in a restructuring compiler. We validate our design experimentally and show that our framework 1) selects the most appropriate algorithms in 85 percent of the cases studied, 2) overall, delivers 98 percent of the optimal performance, 3) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and 4) adapts to the underlying machine architectures (evaluated on IBM Regatta and HP V-class systems)
---
paper_title: Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection

paper_content:

The results of empirical comparisons of existing learning algorithms illustrate that each algorithm has a selective superiority ; it is best for some but not all tasks. Given a data set, it is often not clear beforehand which algorithm will yield the best performance. In such cases one must search the space of available algorithms to find the one that produces the best classifier. In this paper we present an approach that applies knowledge about the representational biases of a set of learning algorithms to conduct this search automatically. In addition, the approach permits the available algorithms' model classes to be mixed in a recursive tree-structured hybrid. We describe an implementation of the approach, MCS, that performs a heuristic bestfirst search for the best hybrid classifier for a set of data. An empirical comparison of MCS to each of its primitive learning algorithms, and to the computationally intensive method of cross-validation, illustrates that automatic selection of learning algorithms using knowledge can be used to solve the selective superiority problem.
---
paper_title: An Instance of Adaptive Constraint Propagation

paper_content:

Constraint propagation algorithms vary in the strength of propagation they apply. This paper investigates a simple configuration for adaptive propagation—the process of varying the strength of propagation to reflect the dynamics of search. We focus on two propagation methods, Arc Consistency (AC) and Forward Checking (FC). AC-based algorithms apply a stronger form of propagation than FC-based algorithms; they invest greater computational effort to detect inconsistent values earlier. The relative payoff of maintaining AC during search as against FC may vary for different constraints and for different intermediate search states. We present a scheme for Adaptive Arc Propagation (AAP) that allows the flexible combination of the two methods. Metalevel reasoning and heuristics are used to dynamically distribute propagation effort between the two. One instance of AAP, Anti-Functional Reduction (AFR), is described in detail here. AFR achieves precisely the same propagation as a pure AC algorithm while significantly improving its average performance. The strategy is to gradually reduce the scope of AC propagation during backtrack search to exclude those arcs that may be subsequently handled as effectively by FC. Experimental results confirm the power of AFR and the validity of adaptive propagation in general.
---
paper_title: Learning to solve QBF

paper_content:

We present a novel approach to solving Quantified Boolean Formulas (QBF) that combines a search-based QBF solver with marhine learning techniques. We show how classification methods can be used to predict run-times and to choose optimal heuristics both within a portfolio-based, and within a dynamic, online approach. In the dynamic method variables are set to a truth value according to a scheme that tries to maximize the probability of successfully solving the remaining sub-problem efficiently. Since each variable assignment can drastically change the problem-structure, new heuristics are chosen dynamically, and a classifier is used online to predict the usefulness of each heuristic. Experimental results on a large corpus of example problems show the usefulness of our approach in terms of run-time as well as the ability to solve previously unsolved problem instances.
---
paper_title: Learning Effective Search Heuristics

paper_content:

SAGE.2 is a production system that improves its search strategies with practice. The program incorporates four different heuristics for assigning credit and blame, and employs a discrimination process to direct its search through the space of move-proposing rules. The system has shown its generality by learning search heuristics in five different task domains. In addition to improving its search behavior on practice problems, SAGE.2 was able to transfer its expertise to scaled-up versions of a task, and in one case transferred its acquired search strategy to problems with different initial and goal states.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: DVRP: a hard dynamic combinatorial optimisation problem tackled by an evolutionary hyper-heuristic

paper_content:

In this paper we propose and evaluate an evolutionary-based hyper-heuristic approach, called EH-DVRP, for solving hard instances of the dynamic vehicle routing problem. A hyper-heuristic is a high-level algorithm, which generates or chooses a set of low-level heuristics in a common framework, to solve the problem at hand. In our collaborative framework, we have included three different types of low-level heuristics: constructive, perturbative, and noise heuristics. Basically, the hyper-heuristic manages and evolves a sophisticated sequence of combinations of these low-level heuristics, which are sequentially applied in order to construct and improve partial solutions, i.e., partial routes. In presenting some design considerations, we have taken into account the allowance of a proper cooperation and communication among low-level heuristics, and as a result, find the most promising sequence to tackle partial states of the (dynamic) problem. Our approach has been evaluated using the Kilby's benchmarks, which comprise a large number of instances with different topologies and degrees of dynamism, and we have compared it with some well-known methods proposed in the literature. The experimental results have shown that, due to the dynamic nature of the hyper-heuristic, our proposed approach is able to adapt to dynamic scenarios more naturally than low-level heuristics. Furthermore, the hyper-heuristic can obtain high-quality solutions when compared with other (meta) heuristic-based methods. Therefore, the findings of this contribution justify the employment of hyper-heuristic techniques in such changing environments, and we believe that further contributions could be successfully proposed in related dynamic problems.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: A self-adaptive multi-engine solver for quantified Boolean formulas

paper_content:

In this paper we study the problem of engineering a robust solver for quantified Boolean formulas (QBFs), i.e., a tool that can efficiently solve formulas across different problem domains without the need for domain-specific tuning. The paper presents two main empirical results along this line of research. Our first result is the development of a multi-engine solver, i.e., a tool that selects among its reasoning engines the one which is more likely to yield optimal results. In particular, we show that syntactic QBF features can be correlated to the performances of existing QBF engines across a variety of domains. We also show how a multi-engine solver can be obtained by carefully picking state-of-the-art QBF solvers as basic engines, and by harnessing inductive reasoning techniques to learn engine-selection policies. Our second result is the improvement of our multi-engine solver with the capability of updating the learned policies when they fail to give good predictions. In this way the solver becomes also self-adaptive, i.e., able to adjust its internal models when the usage scenario changes substantially. The rewarding results obtained in our experiments show that our solver AQME--Adaptive QBF Multi-Engine--can be more robust and efficient than state-of-the-art single-engine solvers, even when it is confronted with previously uncharted formulas and competitors.
---
paper_title: Automated Discovery of Local Search Heuristics for Satisfiability Testing

paper_content:

The development of successful metaheuristic algorithms such as local search for a difficult problem such as satisfiability testing (SAT) is a challenging task. We investigate an evolutionary approach to automating the discovery of new local search heuristics for SAT. We show that several well-known SAT local search algorithms such as Walksat and Novelty are composite heuristics that are derived from novel combinations of a set of building blocks. Based on this observation, we developed CLASS, a genetic programming system that uses a simple composition operator to automatically discover SAT local search heuristics. New heuristics discovered by CLASS are shown to be competitive with the best Walksat variants, including Novelty+. Evolutionary algorithms have previously been applied to directly evolve a solution for a particular SAT instance. We show that the heuristics discovered by CLASS are also competitive with these previous, direct evolutionary approaches for SAT. We also analyze the local search behavior of the learned heuristics using the depth, mobility, and coverage metrics proposed by Schuurmans and Southey.
---
paper_title: Confronting hardness using a hybrid approach

paper_content:

A hybrid algorithm is a collection of heuristics, paired with a polynomial time selector S that runs on the input to decide which heuristic should be executed to solve the problem. Hybrid algorithms are of particular interest in scenarios where the selector must decide between heuristics that are "good" with respect to different complexity measures.We focus on hybrid algorithms with a "hardness-defying" property: for a problem II, there is a set of complexity measures {m i } whereby II is known or conjectured to be unsolvable for each m i , but for each heuristic h i of the hybrid algorithm, one can give a complexity guarantee for h i on the instances of II that S selects for h i that is strictly better than m i . More concretely, we show that for several NP-hard problems, a given instance can either be solved exactly with substantially improved runtime (e.g. 2o(n)), or be approximated in polynomial time with an approximation ratio exceeding that of the known or conjectured inapproximability of the problem, assuming P ≠ NP.
---
paper_title: Machine learning for constraint solver design -- A case study for the alldifferent constraint

paper_content:

Constraint solvers are complex pieces of software which require many design decisions to be made by the implementer based on limited information. These decisions affect the performance of the finished solver significantly. Once a design decision has been made, it cannot easily be reversed, although a different decision may be more appropriate for a particular problem. ::: We investigate using machine learning to make these decisions automatically depending on the problem to solve. We use the alldifferent constraint as a case study. Our system is capable of making non-trivial, multi-level decisions that improve over always making a default choice and can be implemented as part of a general-purpose constraint solver.
---
paper_title: Isac – instance-specific algorithm configuration

paper_content:

We present a new method for instance-specific algorithm configuration (ISAC). It is based on the integration of the algorithm configuration system GGA and the recently proposed stochastic offline programming paradigm. ISAC is provided a solver with categorical, ordinal, and/or continuous parameters, a training benchmark set of input instances for that solver, and an algorithm that computes a feature vector that characterizes any given instance. ISAC then provides high quality parameter settings for any new input instance. Experiments on a variety of different constrained optimization and constraint satisfaction solvers show that automatic algorithm configuration vastly outperforms manual tuning. Moreover, we show that instance-specific tuning frequently leads to significant speed-ups over instance-oblivious configurations.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: On Portfolios for Backtracking Search in the Presence of Deadlines

paper_content:

Constraint satisfaction and prepositional satisfiability problems are often solved using backtracking search. Previous studies have shown that portfolios of backtracking algorithms - a selection of one or more algorithms plus a schedule for executing the algorithms - can dramatically improve performance on some instances. In this paper, we consider a setting that often arises in practice where the instances to be solved arise over time, the instances all belong to some class of problem instances, and a limit or deadline is placed on the computational resources that the backtracking algorithm can consume in solving any instance. For such a scenario, we present a simple scheme for learning a good portfolio of backtracking algorithms from a small sample of instances. We demonstrate the effectiveness of our approach through an extensive empirical evaluation on a real-world instruction scheduling testbed.
---
paper_title: Impact of censored sampling on the performance of restart strategies

paper_content:

Algorithm selection, algorithm portfolios, and randomized restarts, can profit from a probabilistic model of algorithm run-time, to be estimated from data gathered by solving a set of experiments. Censored sampling offers a principled way of reducing this initial training time. We study the trade-off between training time and model precision by varying the censoring threshold, and analyzing the consequent impact on the performance of an optimal restart strategy, based on an estimated model of runtime distribution. ::: ::: We present experiments with a SAT solver on a graph-coloring benchmark. Due to the “heavy-tailed” runtime distribution, a modest censoring can already reduce training time by a few orders of magnitudes. The nature of the optimization process underlying the restart strategy renders its performance surprisingly robust, also to more aggressive censoring.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: An automatically configurable portfolio-based planner with macroactions: PbP

paper_content:

While several powerful domain-independent planners have recently been developed, no one of these clearly outperforms all the others in every known benchmark domain. We present PbP, a multi-planner which automatically configures a portfolio of planners by (i) computing some sets of macro-actions for every planner in the portfolio, (ii) selecting a promising combination of planners in the portfolio and relative useful macro-actions, and (iii) defining some running time slots for their round-robin scheduling during planning. The configuration relies on some knowledge about the performance of the planners in the portfolio and relative macro-actions which is automatically generated from a training problem set. PbP entered the learning track of IPC-2008 and was the overall winner of this competition track. An experimental study confirms the effectiveness of PbP, and shows that the learned configuration knowledge is useful for PbP.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: Rational Deployment of CSP Heuristics

paper_content:

Heuristics are crucial tools in decreasing search effort in varied fields of AI. In order to be effective, a heuristic must be efficient to compute, as well as provide useful information to the search algorithm. However, some well-known heuristics which do well in reducing backtracking are so heavy that the gain of deploying them in a search algorithm might be outweighed by their overhead. ::: ::: We propose a rational metareasoning approach to decide when to deploy heuristics, using CSP backtracking search as a case study. In particular, a value of information approach is taken to adaptive deployment of solution-count estimation heuristics for value ordering. Empirical results show that indeed the proposed mechanism successfully balances the tradeoff between decreasing backtracking and heuristic computational overhead, resulting in a significant overall search time reduction.
---
paper_title: Learning Rules for Adaptive Planning

paper_content:

This paper presents a novel idea, which combines Planning, Machine Learning and Knowledge-Based techniques. It is concerned with the development of an adaptive planning system that can fine-tune its planning parameters based on the values of specific measurable characteristics of the given planning problem. Adaptation is guided by a rule-based system, whose knowledge has been acquired through machine learning techniques. Specifically, the algorithm of classification based on association rules was applied to a large dataset produced by results from experiments on a large number of problems used in the three AIPS Planning competitions. The paper presents experimental results with the adaptive planner, which demonstrate the boost in performance of the planning system.
---
paper_title: An Instance of Adaptive Constraint Propagation

paper_content:

Constraint propagation algorithms vary in the strength of propagation they apply. This paper investigates a simple configuration for adaptive propagation—the process of varying the strength of propagation to reflect the dynamics of search. We focus on two propagation methods, Arc Consistency (AC) and Forward Checking (FC). AC-based algorithms apply a stronger form of propagation than FC-based algorithms; they invest greater computational effort to detect inconsistent values earlier. The relative payoff of maintaining AC during search as against FC may vary for different constraints and for different intermediate search states. We present a scheme for Adaptive Arc Propagation (AAP) that allows the flexible combination of the two methods. Metalevel reasoning and heuristics are used to dynamically distribute propagation effort between the two. One instance of AAP, Anti-Functional Reduction (AFR), is described in detail here. AFR achieves precisely the same propagation as a pure AC algorithm while significantly improving its average performance. The strategy is to gradually reduce the scope of AC propagation during backtrack search to exclude those arcs that may be subsequently handled as effectively by FC. Experimental results confirm the power of AFR and the validity of adaptive propagation in general.
---
paper_title: Learning dynamic algorithm portfolios

paper_content:

Algorithm selection can be performed using a model of runtime distribution, learned during a preliminary training phase. There is a trade-off between the performance of model-based algorithm selection, and the cost of learning the model. In this paper, we treat this trade-off in the context of bandit problems. We propose a fully dynamic and online algorithm selection technique, with no separate training phase: all candidate algorithms are run in parallel, while a model incrementally learns their runtime distributions. A redundant set of time allocators uses the partially trained model to propose machine time shares for the algorithms. A bandit problem solver mixes the model-based shares with a uniform share, gradually increasing the impact of the best time allocators as the model improves. We present experiments with a set of SAT solvers on a mixed SAT-UNSAT benchmark; and with a set of solvers for the Auction Winner Determination problem.
---
paper_title: Switching among Non-Weighting, Clause Weighting, and Variable Weighting in Local Search for SAT

paper_content:

One way to design a local search algorithm that is effective on many types of instances is allowing this algorithm to switch among heuristics. In this paper, we refer to the way in which non-weighting algorithm adaptG2WSAT+ selects a variable to flip, as heuristic adaptG2WSAT+, the way in which clause weighting algorithm RSAPSselects a variable to flip, as heuristic RSAPS, and the way in which variable weighting algorithm VWselects a variable to flip, as heuristic VW. We propose a new switching criterion: the evenness or unevenness of the distribution of clause weights. We apply this criterion, along with another switching criterion previously proposed, to heuristic adaptG2WSAT+, heuristic RSAPS, and heuristic VW. The resulting local search algorithm, which adaptively switches among these three heuristics in every search step according to these two criteria to intensify or diversify the search when necessary, is called NCVW(Non-, Clause, and Variable Weighting). Experimental results show that NCVWis generally effective on a wide range of instances while adaptG2WSAT+, RSAPS, VW, and gNovelty+ and adaptG2WSAT0, which won the gold and silver medals in the satisfiable random category in the SAT 2007 competition, respectively, are not.
---
paper_title: An automated approach to generating efficient constraint solvers

paper_content:

Combinatorial problems appear in numerous settings, from timetabling to industrial design. Constraint solving aims to find solutions to such problems efficiently and automatically. Current constraint solvers are monolithic in design, accepting a broad range of problems. The cost of this convenience is a complex architecture, inhibiting efficiency, extensibility and scalability. Solver components are also tightly coupled with complex restrictions on their configuration, making automated generation of solvers difficult. We describe a novel, automated, model-driven approach to generating efficient solvers tailored to individual problems and present some results from applying the approach. The main contribution of this work is a solver generation framework called Dominion, which analyses a problem and, based on its characteristics, generates a solver using components chosen from a library. The key benefit of this approach is the ability to solve larger and more difficult problems as a result of applying finer-grained optimisations and using specialised techniques as required.
---
paper_title: Learning techniques for Automatic Algorithm Portfolio Selection

paper_content:

The purpose of this paper is to show that a well known machine learning technique based on Decision Trees can be effectively used to select the best approach (in terms of efficiency) in an algorithm portfolio for a particular case study: the Bid Evaluation Problem (BEP) in Combinatorial Auctions. In particular, we are interested in deciding when to use a Constraint Programming (CP) approach and when an Integer Programming (IP) approach, on the basis of the structure of the instance considered. Different instances of the same problem present a different structure, and one aspect (e.g. feasibility or optimality) can prevail on the other. We have extracted from a set of BEP instances, a number of parameters representing the instance structure. Some of them (few indeed) precisely identify the best strategy and its corresponding tuning to be used to face that instance. We will show that this approach is very promising, since it identifies the most efficient algorithm in the 90% of the cases.
---
paper_title: Isac – instance-specific algorithm configuration

paper_content:

We present a new method for instance-specific algorithm configuration (ISAC). It is based on the integration of the algorithm configuration system GGA and the recently proposed stochastic offline programming paradigm. ISAC is provided a solver with categorical, ordinal, and/or continuous parameters, a training benchmark set of input instances for that solver, and an algorithm that computes a feature vector that characterizes any given instance. ISAC then provides high quality parameter settings for any new input instance. Experiments on a variety of different constrained optimization and constraint satisfaction solvers show that automatic algorithm configuration vastly outperforms manual tuning. Moreover, we show that instance-specific tuning frequently leads to significant speed-ups over instance-oblivious configurations.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: Learning How to Propagate Using Random Probing

paper_content:

In constraint programming there are often many choices regarding the propagation method to be used on the constraints of a problem. However, simple constraint solvers usually only apply a standard method, typically (generalized) arc consistency, on all constraints throughout search. Advanced solvers additionally allow for the modeler to choose among an array of propagators for certain (global) constraints. Since complex interactions exist among constraints, deciding in the modelling phase which propagation method to use on given constraints can be a hard task that ideally we would like to free the user from. In this paper we propose a simple technique towards the automation of this task. Our approach exploits information gathered from a random probing preprocessing phase to automatically decide on the propagation method to be used on each constraint. As we demonstrate, data gathered though probing allows for the solver to accurately differentiate between constraints that offer little pruning as opposed to ones that achieve many domain reductions, and also to detect constraints and variables that are amenable to certain propagation methods. Experimental results from an initial evaluation of the proposed method on binary CSPs demonstrate the benefits of our approach.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: A learning-based algorithm selection meta-reasoner for the real-time MPE problem

paper_content:

The algorithm selection problem aims to select the best algorithm for an input problem instance according to some characteristics of the instance This paper presents a learning-based inductive approach to build a predictive algorithm selection system from empirical algorithm performance data of the Most Probable Explanation(MPE) problem The learned model can serve as an algorithm selection meta-reasoner for the real-time MPE problem Experimental results show that the learned algorithm selection models can help integrate multiple MPE algorithms to gain a better overall performance of reasoning.
---
paper_title: Hybrid regression-classification models for algorithm selection

paper_content:

Many state of the art Algorithm Selection systems use Machine Learning to either predict the run time or a similar performance measure of each of a set of algorithms and choose the algorithm with the best predicted performance or predict the best algorithm directly. We present a technique based on the well-established Machine Learning technique of stacking that combines the two approaches into a new hybrid approach and predicts the best algorithm based on predicted run times. We demonstrate significant performance improvements of up to a factor of six compared to the previous state of the art. Our approach is widely applicable and does not place any restrictions on the performance measure used, the way to predict it or the Machine Learning used to predict the best algorithm. We investigate different ways of deriving new Machine Learning features from the predicted performance measures and evaluate their effectiveness in increasing performance further. We use five different regression algorithms for performance prediction on five data sets from the literature and present strong empirical evidence that shows the effectiveness of our approach.
---
paper_title: Algorithm Selection using Reinforcement Learning

paper_content:

Many computational problems can be solved by multiple algorithms, with different algorithms fastest for different problem sizes, input distributions, and hardware characteristics. We consider the problem of algorithm selection: dynamically choose an algorithm to attack an instance of a problem with the goal of minimizing the overall execution time. We formulate the problem as a kind of Markov decision process (MDP), and use ideas from reinforcement learning to solve it. This paper introduces a kind of MDP that models the algorithm selection problem by allowing multiple state transitions. The well known Q-learning algorithm is adapted for this case in a way that combines both Monte-Carlo and Temporal Difference methods. Also, this work uses, and extends in a way to control problems, the Least-Squares Temporal Difference algorithm (LSTD(0)) of Boyan. The experimental study focuses on the classic problems of order statistic selection and sorting. The encouraging results reveal the potential of applying learning methods to traditional computational problems.
---
paper_title: Instance-Based Selection of Policies for SAT Solvers

paper_content:

Execution of most of the modern DPLL-based SAT solvers is guided by a number of heuristics. Decisions made during the search process are usually driven by some fixed heuristic policies. Despite the outstanding progress in SAT solving in recent years, there is still an appealing lack of techniques for selecting policies appropriate for solving specific input formulae. In this paper we present a methodology for instance-based selection of solver's policies that uses a data-mining classification technique. The methodology also relies on analysis of relationships between formulae, their families, and their suitable solving strategies. The evaluation results are very good, demonstrate practical usability of the methodology, and encourage further efforts in this direction.
---
paper_title: The Adaptive Constraint Engine

paper_content:

The Adaptive Constraint Engine (ACE) seeks to automate the application of constraint programming expertise and the extraction of domain-specific expertise. Under the aegis of FORR, an architecture for learning and problem-solving, ACE learns search-order heuristics from problem solving experience. This paper describes ACE?s approach, as well as new experimental results on specific problem classes. ACE is both a test-bed for CSP research and a discovery environment for new algorithms.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: Self-Adapting Linear Algebra Algorithms and Software

paper_content:

One of the main obstacles to the efficient solution of scientific problems is the problem of tuning software, both to the available architecture and to the user problem at hand. We describe approaches for obtaining tuned high-performance kernels and for automatically choosing suitable algorithms. Specifically, we describe the generation of dense and sparse Basic Linear Algebra Subprograms (BLAS) kernels, and the selection of linear solver algorithms. However, the ideas presented here extend beyond these areas, which can be considered proof of concept.
---
paper_title: Non-model-based algorithm portfolios for SAT

paper_content:

When tackling a computationally challenging combinatorial problem, one often observes that some solution approaches work well on some instances, while other approaches work better on other instances. This observation has given rise to the idea of building algorithm portfolios [5]. Leyton-Brown et al. [1], for instance, proposed to select one of the algorithms in the portfolio based on some features of the instance to be solved. This approach has been blessed with tremendous success in the past. Especially in SAT, the SATzilla portfolios [7] have performed extremely well in past SAT Competitions [6].
---
paper_title: Low-Knowledge Algorithm Control

paper_content:

This paper addresses the question of allocating computational resources among a set of algorithms in order to achieve the best performance on a scheduling problem instance. Our primary motivation in addressing this problem is to reduce the expertise needed to apply constraint technology. Therefore, we investigate algorithm control techniques that make decision based only on observations of the improvement in solution quality achieved by each algorithm. We call our approach "low-knowledge" since it does not rely on complex prediction models. We show that such an approach results in a system that achieves significantly better performance than all of the pure algorithms without requiring additional human expertise. Furthermore the low knowledge approach achieves performance equivalent to a perfect high-knowledge classification approach.
---
paper_title: PRODIGY: an integrated architecture for planning and learning

paper_content:

Artificial intelligence has progressed to the point where multiple cognitive capabilities are being integrated into computational architectures, such as SOAR, PRODIGY, THEO, and ICARUS. This paper reports on the PRODIGY architecture, describing its planning and problem solving capabilities and touching upon its multiple learning methods. Learning in PRODIGY occurs at all decision points and integration in PRODIGY is at the knowledge level; the learning and reasoning modules produce mutually interpretable knowledge structures. Issues in architectural design are discussed, providing a context to examine the underlying tenets of the PRODIGY architecture.
---
paper_title: Statistically optimal combination of algorithms

paper_content:

Automatic specialization of algorithms to a limited domain is an interesting and industrially applicable problem. We calculate the optimal assignment of computational resources to several different solvers that solve the same problem. Optimality is considered with regard to the expected solution time on a set of problem instances from the domain of interest. We present two approaches, a static and dynamic one. The static approach leads to a simple analytically calculable solution. The dynamic approach results in formulation of the problem as a Markov Decision Process. Our tests on the SAT Problem show that the presented methods are quite effective. Therefore, both methods are attractive for applications and future research.
---
paper_title: Combining multiple heuristics

paper_content:

In this work we introduce and study the question of combining multiple heuristics. Given a problem instance, each of the multiple heuristics is capable of computing the correct solution, but has a different cost. In our models the user executes multiple heuristics until one of them terminates with a solution. Given a set of problem instances, we show how to efficiently compute an optimal fixed schedule for a constant number of heuristics, and show that in general, the problem is computationally hard even to approximate (to within a constant factor). We also discuss a probabilistic configuration, in which the problem instances are drawn from some unknown fixed distribution, and show how to compute a near optimal schedule for this setup.
---
paper_title: Learning Rules for Adaptive Planning

paper_content:

This paper presents a novel idea, which combines Planning, Machine Learning and Knowledge-Based techniques. It is concerned with the development of an adaptive planning system that can fine-tune its planning parameters based on the values of specific measurable characteristics of the given planning problem. Adaptation is guided by a rule-based system, whose knowledge has been acquired through machine learning techniques. Specifically, the algorithm of classification based on association rules was applied to a large dataset produced by results from experiments on a large number of problems used in the three AIPS Planning competitions. The paper presents experimental results with the adaptive planner, which demonstrate the boost in performance of the planning system.
---
paper_title: Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection

paper_content:

The results of empirical comparisons of existing learning algorithms illustrate that each algorithm has a selective superiority ; it is best for some but not all tasks. Given a data set, it is often not clear beforehand which algorithm will yield the best performance. In such cases one must search the space of available algorithms to find the one that produces the best classifier. In this paper we present an approach that applies knowledge about the representational biases of a set of learning algorithms to conduct this search automatically. In addition, the approach permits the available algorithms' model classes to be mixed in a recursive tree-structured hybrid. We describe an implementation of the approach, MCS, that performs a heuristic bestfirst search for the best hybrid classifier for a set of data. An empirical comparison of MCS to each of its primitive learning algorithms, and to the computationally intensive method of cross-validation, illustrates that automatic selection of learning algorithms using knowledge can be used to solve the selective superiority problem.
---
paper_title: Learning to solve QBF

paper_content:

We present a novel approach to solving Quantified Boolean Formulas (QBF) that combines a search-based QBF solver with marhine learning techniques. We show how classification methods can be used to predict run-times and to choose optimal heuristics both within a portfolio-based, and within a dynamic, online approach. In the dynamic method variables are set to a truth value according to a scheme that tries to maximize the probability of successfully solving the remaining sub-problem efficiently. Since each variable assignment can drastically change the problem-structure, new heuristics are chosen dynamically, and a classifier is used online to predict the usefulness of each heuristic. Experimental results on a large corpus of example problems show the usefulness of our approach in terms of run-time as well as the ability to solve previously unsolved problem instances.
---
paper_title: Combining multiple heuristics online

paper_content:

We present black-box techniques for learning how to interleave the execution of multiple heuristics in order to improve average-case performance. In our model, a user is given a set of heuristics whose only observable behavior is their running time. Each heuristic can compute a solution to any problem instance, but its running time varies across instances. The user solves each instance by interleaving runs of the heuristics according to a task-switching schedule. We present (i) exact and approximation algorithms for computing an optimal task-switching schedule offline, (ii) sample complexity bounds for learning a task-switching schedule from training data, and (iii) a no-regret strategy for selecting task-switching schedules online. We demonstrate the power of our results using data from recent solver competitions. We outline how to extend our results to the case in which the heuristics are randomized, and the user may periodically restart each heuristic with a fresh random seed.
---
paper_title: Heuristics for dynamically adapting propagation in constraint satisfaction problems

paper_content:

Building adaptive constraint solvers is a major challenge in constraint programming. An important line of research towards this goal is concerned with ways to dynamically adapt the propagation method applied on the constraints of the problem during search. In this paper we present a heuristic approach to this problem based on the monitoring of propagation events like value deletions and domain wipeouts. We develop a number of heuristics that allow the constraint solver to dynamically switch between a weaker and cheap local consistency and a stronger, but more expensive one, when certain conditions are met. The success of this approach is based on the observation that propagation events for individual constraints in structured problems mostly occur in clusters of nearby revisions. Hence, parts of the search space where certain constraints are highly active can be identified and exploited paving the way for the informed use of constraint propagation techniques. In this paper we first give some experimental results displaying the clustering of propagation events in structured binary CSPs. Then we present simple heuristics that exploit this clustering to efficiently switch between different local consistencies on individual constraints during search. Finally, we make an experimental study on various binary CSPs demonstrating the effectiveness of the proposed heuristics.
---
paper_title: A Neural Network Model for Inter-problem Adaptive Online Time Allocation

paper_content:

One aim of Meta-learning techniques is to minimize the time needed for problem solving, and the effort of parameter hand-tuning, by automating algorithm selection. The predictive model of algorithm performance needed for task often requires long training times. We address the problem in an online fashion, running multiple algorithms in parallel on a sequence of tasks, continually updating their relative priorities according to a neural model that maps their current state to the expected time to the solution. The model itself is updated at the end of each task, based on the actual performance of each algorithm. Censored sampling allows us to train the model effectively, without need of additional exploration after each task's solution. We present a preliminary experiment in which this new inter-problem technique learns to outperform a previously proposed intraproblem heuristic.
---
paper_title: Learning techniques for Automatic Algorithm Portfolio Selection

paper_content:

The purpose of this paper is to show that a well known machine learning technique based on Decision Trees can be effectively used to select the best approach (in terms of efficiency) in an algorithm portfolio for a particular case study: the Bid Evaluation Problem (BEP) in Combinatorial Auctions. In particular, we are interested in deciding when to use a Constraint Programming (CP) approach and when an Integer Programming (IP) approach, on the basis of the structure of the instance considered. Different instances of the same problem present a different structure, and one aspect (e.g. feasibility or optimality) can prevail on the other. We have extracted from a set of BEP instances, a number of parameters representing the instance structure. Some of them (few indeed) precisely identify the best strategy and its corresponding tuning to be used to face that instance. We will show that this approach is very promising, since it identifies the most efficient algorithm in the 90% of the cases.
---
paper_title: A Comparison of Ranking Methods for Classification Algorithm Selection

paper_content:

We investigate the problem of using past performance information to select an algorithm for a given classification problem. We present three ranking methods for that purpose: average ranks, success rate ratios and significant wins. We also analyze the problem of evaluating and comparing these methods. The evaluation technique used is based on a leave-one-out procedure. On each iteration, the method generates a ranking using the results obtained by the algorithms on the training datasets. This ranking is then evaluated by calculating its distance from the ideal ranking built using the performance information on the test dataset. The distance measure adopted here, average correlation, is based on Spearman's rank correlation coefficient. To compare ranking methods, a combination of Friedman's test and Dunn's multiple comparison procedure is adopted. When applied to the methods presented here, these tests indicate that the success rate ratios and average ranks methods perform better than significant wins.
---
paper_title: Statistical Selection Among Problem-Solving Methods

paper_content:

Abstract : The choice of an appropriate problem solving method, from available methods, is a crucial skill for human experts in many areas. We describe a technique for automatic selection among methods, based on a statistical analysis of their past performances. We formalize the statistical problem involved in selecting an efficient problem solving method, derive a solution to this problem, and describe a selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We extend our basic statistical technique to account for problem sizes and for similarity between problems. We give empirical results of the use of this technique to select among search engines in the PRODIGY system. We also test the selection technique on artificially generated performance data, using several different probability distributions.
---
paper_title: How to solve it automatically: Selection among problem-solving methods

paper_content:

The choice of an appropriate problem-solving method, from available methods, is a crucial skill for experts in many areas. We describe a technique for the automatic selection among methods, which is based on a statistical analysis of their past performances. ::: ::: We formalize the statistical problem involved in selecting an efficient problem-solving method, derive a solution to this problem, and describe a method-selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We give empirical results on the use of this technique in selecting among search engines in the PRODIGY planning system.
---
paper_title: Empirical modeling and analysis of local search algorithms for the job-shop scheduling problem

paper_content:

Local search algorithms are among the most effective approaches for locating high-quality solutions to a wide range of combinatorial optimization problems. However, our theoretical understanding of these algorithms is very limited, leading to significant problems for both researchers and practitioners. Specifically, the lack of a theory of local search impedes the development of more effective algorithms, prevents practitioners from identifying the algorithm most appropriate for a given problem, and allows widespread conjecture and misinformation regarding the benefits and/or drawbacks of particular algorithms. This thesis represents a significant step toward a theory of local search. Using empirical methods, we develop theoretical models of the behavior of four well-known local search algorithms: a random walk, tabu search, iterated local search, and simulated annealing. The analysis proceeds in the context of the well-known job-shop scheduling problem, one of the most difficult NP-hard problems encountered in practice. The large volume of prior research on the job-shop scheduling problem provides a diverse range of available algorithms and problem instances, in addition to numerous empirical observations regarding local search algorithm behavior; the latter are used to validate our behavioral models. ::: We show that all four local search algorithms can be modeled with high fidelity using straightforward variations of a generalized one-dimensional Markov chain. The states in these models represent sets of solutions a given fixed distance from the nearest optimal solution. The transition probabilities in all of the models are remarkably similar, in that search is consistently biased toward solutions that are roughly equidistant from the nearest optimal solution and solutions that are maximally distant from the nearest optimal solution. Surprisingly, the qualitative form of the transition probabilities is simply due to the structure of the representation used to encode solutions: the binary hypercube. The models account for between 96% and 99% of the variability in the cost required to locate both optimal and sub-optimal solutions to a wide range of problem instances, and provide explanations for numerous phenomena related to problem difficulty for local search in the job-shop scheduling problem. In the course of our analysis, we also disprove many conjectures regarding the behavior and benefits of particular algorithms. (Abstract shortened by UMI.)
---
paper_title: On Portfolios for Backtracking Search in the Presence of Deadlines

paper_content:

Constraint satisfaction and prepositional satisfiability problems are often solved using backtracking search. Previous studies have shown that portfolios of backtracking algorithms - a selection of one or more algorithms plus a schedule for executing the algorithms - can dramatically improve performance on some instances. In this paper, we consider a setting that often arises in practice where the instances to be solved arise over time, the instances all belong to some class of problem instances, and a limit or deadline is placed on the computational resources that the backtracking algorithm can consume in solving any instance. For such a scenario, we present a simple scheme for learning a good portfolio of backtracking algorithms from a small sample of instances. We demonstrate the effectiveness of our approach through an extensive empirical evaluation on a real-world instruction scheduling testbed.
---
paper_title: Restart Strategy Selection using Machine Learning Techniques

paper_content:

Restart strategies are an important factor in the performance of conflict-driven Davis Putnam style SAT solvers. Selecting a good restart strategy for a problem instance can enhance the performance of a solver. Inspired by recent success applying machine learning techniques to predict the runtime of SAT solvers, we present a method which uses machine learning to boost solver performance through a smart selection of the restart strategy. Based on easy to compute features, we train both a satisfiability classifier and runtime models. We use these models to choose between restart strategies. We present experimental results comparing this technique with the most commonly used restart strategies. Our results demonstrate that machine learning is effective in improving solver performance.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: Branch and Bound Algorithm Selection by Performance Prediction

paper_content:

We propose a method called Selection by Performance Prediction (SPP) which allows one, when faced with a particular problem instance, to select a Branch and Bound algorithm from among several promising ones. This method is based on Knuth's sampling method which estimates the efficiency of a backtrack program on a particular instance by iteratively generating random paths in the search tree. We present a simple adaptation of this estimator in the field of combinatorial optimization problems, more precisely for an extension of the maximal constraint satisfaction framework. Experiments both on random and strongly structured instances show that, in most cases, the proposed method is able to select, from a candidate list, the best algorithm for solving a given instance.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: An automatically configurable portfolio-based planner with macroactions: PbP

paper_content:

While several powerful domain-independent planners have recently been developed, no one of these clearly outperforms all the others in every known benchmark domain. We present PbP, a multi-planner which automatically configures a portfolio of planners by (i) computing some sets of macro-actions for every planner in the portfolio, (ii) selecting a promising combination of planners in the portfolio and relative useful macro-actions, and (iii) defining some running time slots for their round-robin scheduling during planning. The configuration relies on some knowledge about the performance of the planners in the portfolio and relative macro-actions which is automatically generated from a training problem set. PbP entered the learning track of IPC-2008 and was the overall winner of this competition track. An experimental study confirms the effectiveness of PbP, and shows that the learned configuration knowledge is useful for PbP.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: PYTHIA: a knowledge-based system to select scientific algorithms

paper_content:

Problem-solving e nvironments (PSEs) interact with theuser in a language “natural” to the associated discipline,and they provide a high-level abstraction of the underlying,computationally complex model. The knowledge-based system PYTHIAaddresses the problem of (parameter, algorithm) pair selection within ascientific computing domain assuming some minimum user-specifiedcomputational objectives and some characteristics of the given problem.PYTHIA's framework and methodology are general and applicable to anyclass of scientific problems and solvers. PYTHIA is applied in thecontext of Parallel ELLPACK where there are many alternatives for thenumerical solution of elliptic partial differential equations (PDEs).PYTHIA matches the characteristics of the given problem with those ofPDEs in an existing problem population and then uses performanceprofiles of the various solvers to select the appropriate method givenuser-specified error and solution time bounds. The profiles areautomatically generated for each solver of the Parallel ELLPACKlibrary. — Authors' Abstract
---
paper_title: Learning dynamic algorithm portfolios

paper_content:

Algorithm selection can be performed using a model of runtime distribution, learned during a preliminary training phase. There is a trade-off between the performance of model-based algorithm selection, and the cost of learning the model. In this paper, we treat this trade-off in the context of bandit problems. We propose a fully dynamic and online algorithm selection technique, with no separate training phase: all candidate algorithms are run in parallel, while a model incrementally learns their runtime distributions. A redundant set of time allocators uses the partially trained model to propose machine time shares for the algorithms. A bandit problem solver mixes the model-based shares with a uniform share, gradually increasing the impact of the best time allocators as the model improves. We present experiments with a set of SAT solvers on a mixed SAT-UNSAT benchmark; and with a set of solvers for the Auction Winner Determination problem.
---
paper_title: Optimizing time warp simulation with reinforcement learning techniques

paper_content:

Adaptive time warp protocols in the literature are usually based on a pre-defined analytic model of the system, expressed as a closed form function that maps system state to control parameter. The underlying assumption is that this model itself is optimal. In this paper we present a new approach that utilizes reinforcement learning techniques, also known as simulation-based dynamic programming. Instead of assuming an optimal control strategy, the very goal of reinforcement learning is to find the optimal strategy through simulation. A value function that captures the history of system feedbacks is used, and no prior knowledge of the system is required. Our reinforcement learning techniques were implemented in a distributed VLSI simulator with the objective of finding the optimal size of a bounded time window. Our experiments using two benchmark circuits indicated that it was successful in doing so.
---
paper_title: The max k-armed bandit: A new model of exploration applied to search heuristic selection

paper_content:

The multiarmed bandit is often used as an analogy for the tradeoff between exploration and exploitation in search problems. The classic problem involves allocating trials to the arms of a multiarmed slot machine to maximize the expected sum of rewards. We pose a new variation of the multiarmed bandit--the Max K-Armed Bandit--in which trials must be allocated among the arms to maximize the expected best single sample reward of the series of trials. Motivation for the Max K-Armed Bandit is the allocation of restarts among a set of multistart stochastic search algorithms. We present an analysis of this Max K-Armed Bandit showing under certain assumptions that the optimal strategy allocates trials to the observed best arm at a rate increasing double exponentially relative to the other arms. This motivates an exploration strategy that follows a Boltzmann distribution with an exponentially decaying temperature parameter. We compare this exploration policy to policies that allocate trials to the observed best arm at rates faster (and slower) than double exponentially. The results confirm, for two scheduling domains, that the double exponential increase in the rate of allocations to the observed best heuristic outperfonns the other approaches.
---
paper_title: Stacked Generalization

paper_content:

This paper introduces stacked generalization, a scheme for minimizing the generalization error rate of one or more generalizers. Stacked generalization works by deducing the biases of the generalizer(s) with respect to a provided learning set. This deduction proceeds by generalizing in a second space whose inputs are (for example) the guesses of the original generalizers when taught with part of the learning set and trying to guess the rest of it, and whose output is (for example) the correct guess. When used with multiple generalizers, stacked generalization can be seen as a more sophisticated version of cross-validation, exploiting a strategy more sophisticated than cross-validation's crude winner-takes-all for combining the individual generalizers. When used with a single generalizer, stacked generalization is a scheme for estimating (and then correcting for) the error of a generalizer which has been trained on a particular learning set and then asked a particular question. After introducing stacked generalization and justifying its use, this paper presents two numerical experiments. The first demonstrates how stacked generalization improves upon a set of separate generalizers for the NETtalk task of translating text to phonemes. The second demonstrates how stacked generalization improves the performance of a single surface-fitter. With the other experimental evidence in the literature, the usual arguments supporting cross-validation, and the abstract justifications presented in this paper, the conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate. This paper ends by discussing some of the variations of stacked generalization, and how it touches on other fields like chaos theory.
---
paper_title: Hybrid regression-classification models for algorithm selection

paper_content:

Many state of the art Algorithm Selection systems use Machine Learning to either predict the run time or a similar performance measure of each of a set of algorithms and choose the algorithm with the best predicted performance or predict the best algorithm directly. We present a technique based on the well-established Machine Learning technique of stacking that combines the two approaches into a new hybrid approach and predicts the best algorithm based on predicted run times. We demonstrate significant performance improvements of up to a factor of six compared to the previous state of the art. Our approach is widely applicable and does not place any restrictions on the performance measure used, the way to predict it or the Machine Learning used to predict the best algorithm. We investigate different ways of deriving new Machine Learning features from the predicted performance measures and evaluate their effectiveness in increasing performance further. We use five different regression algorithms for performance prediction on five data sets from the literature and present strong empirical evidence that shows the effectiveness of our approach.
---
paper_title: Restart Strategy Selection using Machine Learning Techniques

paper_content:

Restart strategies are an important factor in the performance of conflict-driven Davis Putnam style SAT solvers. Selecting a good restart strategy for a problem instance can enhance the performance of a solver. Inspired by recent success applying machine learning techniques to predict the runtime of SAT solvers, we present a method which uses machine learning to boost solver performance through a smart selection of the restart strategy. Based on easy to compute features, we train both a satisfiability classifier and runtime models. We use these models to choose between restart strategies. We present experimental results comparing this technique with the most commonly used restart strategies. Our results demonstrate that machine learning is effective in improving solver performance.
---
paper_title: Machine learning for constraint solver design -- A case study for the alldifferent constraint

paper_content:

Constraint solvers are complex pieces of software which require many design decisions to be made by the implementer based on limited information. These decisions affect the performance of the finished solver significantly. Once a design decision has been made, it cannot easily be reversed, although a different decision may be more appropriate for a particular problem. ::: We investigate using machine learning to make these decisions automatically depending on the problem to solve. We use the alldifferent constraint as a case study. Our system is capable of making non-trivial, multi-level decisions that improve over always making a default choice and can be implemented as part of a general-purpose constraint solver.
---
paper_title: Performance prediction and automated tuning of randomized and parametric algorithms

paper_content:

Machine learning can be used to build models that predict the run-time of search algorithms for hard combinatorial problems. Such empirical hardness models have previously been studied for complete, deterministic search algorithms. In this work, we demonstrate that such models can also make surprisingly accurate predictions of the run-time distributions of incomplete and randomized search methods, such as stochastic local search algorithms. We also show for the first time how information about an algorithm's parameter settings can be incorporated into a model, and how such models can be used to automatically adjust the algorithm's parameters on a per-instance basis in order to optimize its performance. Empirical results for Novelty + and SAPS on structured and unstructured SAT instances show very good predictive performance and significant speedups of our automatically determined parameter settings when compared to the default and best fixed distribution-specific parameter settings.
---
paper_title: Algorithm Selection for Sorting and Probabilistic Inference: A Machine Learning-Based Approach

paper_content:

The algorithm selection problem aims at selecting the best algorithm for a given computational problem instance according to some characteristics of the instance. In this dissertation, we first introduce some results from theoretical investigation of the algorithm selection problem. We show, by Rice's theorem, the nonexistence of an automatic algorithm selection program based only on the description of the input instance and the competing hardness and algorithm performance based on Kolmogorov complexity to show that algorithm selection for search is also incomputable. Driven by the theoretical results, we propose a machine learning-based inductive approach using experimental algorithmic methods and machine learning techniques to solve the algorithm selection problem. ::: Experimentally, we have applied the proposed methodology to algorithm selection for sorting and the MPE problem. In sorting, instances with an existing order are easier for some algorithms. We have studied different presortedness measures, designed algorithms to generate permutations with a specified existing order uniformly at random, and applied various learning algorithms to induce sorting algorithm selection models from runtime experimental results. In the MPE problem, the instance characteristics we have studied include size and topological type of the network, network connectedness, skewness of the distributions in Conditional Probability Tables (CPTs), and the proportion and distribution of evidence variables. The MPE algorithms considered include an exact algorithm (clique-tree propagation), two stochastic sampling algorithms (MCMC Gibbs sampling and importance forward sampling), two search-based algorithms (multi-restart hill-climbing and tabu search), and one hybrid algorithm combining both sampling and search (ant colony optimization). ::: Another major contribution of this dissertation is the discovery of multifractal properties of the joint probability distributions of Bayesian networks. With sufficient asymmetry in individual prior and conditional probability distributions, the joint distribution is not only highly skewed, but it also has clusters of high-probability instantiations at all scales. We present a two phase hybrid random sampling and search algorithm to solve the MPE problem exploiting this clustering property. Since the MPE problem (decision version) is NP-complete, the multifractal meta-heuristic can be applied to solve other NP-hard combinatorial optimization problems as well.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: A learning-based algorithm selection meta-reasoner for the real-time MPE problem

paper_content:

The algorithm selection problem aims to select the best algorithm for an input problem instance according to some characteristics of the instance This paper presents a learning-based inductive approach to build a predictive algorithm selection system from empirical algorithm performance data of the Most Probable Explanation(MPE) problem The learned model can serve as an algorithm selection meta-reasoner for the real-time MPE problem Experimental results show that the learned algorithm selection models can help integrate multiple MPE algorithms to gain a better overall performance of reasoning.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: Using CBR to select solution strategies in constraint programming

paper_content:

Constraint programming is a powerful paradigm that offers many different strategies for solving problems. Choosing a good strategy is difficult; choosing a poor strategy wastes resources and may result in a problem going unsolved. We show how Case-Based Reasoning can be used to select good strategies. We design experiments which demonstrate that, on two problems with quite different characteristics, CBR can outperform four other strategy selection techniques.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: PYTHIA: a knowledge-based system to select scientific algorithms

paper_content:

Problem-solving e nvironments (PSEs) interact with theuser in a language “natural” to the associated discipline,and they provide a high-level abstraction of the underlying,computationally complex model. The knowledge-based system PYTHIAaddresses the problem of (parameter, algorithm) pair selection within ascientific computing domain assuming some minimum user-specifiedcomputational objectives and some characteristics of the given problem.PYTHIA's framework and methodology are general and applicable to anyclass of scientific problems and solvers. PYTHIA is applied in thecontext of Parallel ELLPACK where there are many alternatives for thenumerical solution of elliptic partial differential equations (PDEs).PYTHIA matches the characteristics of the given problem with those ofPDEs in an existing problem population and then uses performanceprofiles of the various solvers to select the appropriate method givenuser-specified error and solution time bounds. The profiles areautomatically generated for each solver of the Parallel ELLPACKlibrary. — Authors' Abstract
---
paper_title: Machine learning for constraint solver design -- A case study for the alldifferent constraint

paper_content:

Constraint solvers are complex pieces of software which require many design decisions to be made by the implementer based on limited information. These decisions affect the performance of the finished solver significantly. Once a design decision has been made, it cannot easily be reversed, although a different decision may be more appropriate for a particular problem. ::: We investigate using machine learning to make these decisions automatically depending on the problem to solve. We use the alldifferent constraint as a case study. Our system is capable of making non-trivial, multi-level decisions that improve over always making a default choice and can be implemented as part of a general-purpose constraint solver.
---
paper_title: Adaptive Online Time Allocation to Search Algorithms

paper_content:

Given is a search problem or a sequence of search problems, as well as a set of potentially useful search algorithms. We propose a general framework for online allocation of computation time to search algorithms based on experience with their performance so far. In an example instantiation, we use simple linear extrapolation of performance for allocating time to various simultaneously running genetic algorithms characterized by different parameter values. Despite the large number of searchers tested in parallel, on various tasks this rather general approach compares favorably to a more specialized state-of-the-art heuristic; in one case it is nearly two orders of magnitude faster.
---
paper_title: A Comparison of Ranking Methods for Classification Algorithm Selection

paper_content:

We investigate the problem of using past performance information to select an algorithm for a given classification problem. We present three ranking methods for that purpose: average ranks, success rate ratios and significant wins. We also analyze the problem of evaluating and comparing these methods. The evaluation technique used is based on a leave-one-out procedure. On each iteration, the method generates a ranking using the results obtained by the algorithms on the training datasets. This ranking is then evaluated by calculating its distance from the ideal ranking built using the performance information on the test dataset. The distance measure adopted here, average correlation, is based on Spearman's rank correlation coefficient. To compare ranking methods, a combination of Friedman's test and Dunn's multiple comparison procedure is adopted. When applied to the methods presented here, these tests indicate that the success rate ratios and average ranks methods perform better than significant wins.
---
paper_title: A Bayesian Approach to Tackling Hard Computational Problems

paper_content:

We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the time required to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We review the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: How to solve it automatically: Selection among problem-solving methods

paper_content:

The choice of an appropriate problem-solving method, from available methods, is a crucial skill for experts in many areas. We describe a technique for the automatic selection among methods, which is based on a statistical analysis of their past performances. ::: ::: We formalize the statistical problem involved in selecting an efficient problem-solving method, derive a solution to this problem, and describe a method-selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We give empirical results on the use of this technique in selecting among search engines in the PRODIGY planning system.
---
paper_title: Knowledge-based constraint-driven software synthesis

paper_content:

A constraint-driven, real-time software synthesis architecture called RT-Syn is described. RT-Syn formulates design space constraints for each task in the real-time software system from timing requirements and a behavior description. RT-Syn then uses these constraints within a simulated-annealing-like approach, selects an abstract implementation for every data structure and algorithm required to implement the desired behavior, and transforms these selections into executable code. Experimental results covering the synthesis of two real-time software tasks that meet the desired constraints are presented. These results illustrate the effectiveness of the simulated-annealing-like approach in searching the software design space and the high reusability and maintainability provided by the use of synthesis technology. >
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: Algorithm Selection using Reinforcement Learning

paper_content:

Many computational problems can be solved by multiple algorithms, with different algorithms fastest for different problem sizes, input distributions, and hardware characteristics. We consider the problem of algorithm selection: dynamically choose an algorithm to attack an instance of a problem with the goal of minimizing the overall execution time. We formulate the problem as a kind of Markov decision process (MDP), and use ideas from reinforcement learning to solve it. This paper introduces a kind of MDP that models the algorithm selection problem by allowing multiple state transitions. The well known Q-learning algorithm is adapted for this case in a way that combines both Monte-Carlo and Temporal Difference methods. Also, this work uses, and extends in a way to control problems, the Least-Squares Temporal Difference algorithm (LSTD(0)) of Boyan. The experimental study focuses on the classic problems of order statistic selection and sorting. The encouraging results reveal the potential of applying learning methods to traditional computational problems.
---
paper_title: Instance-Based Selection of Policies for SAT Solvers

paper_content:

Execution of most of the modern DPLL-based SAT solvers is guided by a number of heuristics. Decisions made during the search process are usually driven by some fixed heuristic policies. Despite the outstanding progress in SAT solving in recent years, there is still an appealing lack of techniques for selecting policies appropriate for solving specific input formulae. In this paper we present a methodology for instance-based selection of solver's policies that uses a data-mining classification technique. The methodology also relies on analysis of relationships between formulae, their families, and their suitable solving strategies. The evaluation results are very good, demonstrate practical usability of the methodology, and encourage further efforts in this direction.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: Low-Knowledge Algorithm Control

paper_content:

This paper addresses the question of allocating computational resources among a set of algorithms in order to achieve the best performance on a scheduling problem instance. Our primary motivation in addressing this problem is to reduce the expertise needed to apply constraint technology. Therefore, we investigate algorithm control techniques that make decision based only on observations of the improvement in solution quality achieved by each algorithm. We call our approach "low-knowledge" since it does not rely on complex prediction models. We show that such an approach results in a system that achieves significantly better performance than all of the pure algorithms without requiring additional human expertise. Furthermore the low knowledge approach achieves performance equivalent to a perfect high-knowledge classification approach.
---
paper_title: Branch and Bound Algorithm Selection by Performance Prediction

paper_content:

We propose a method called Selection by Performance Prediction (SPP) which allows one, when faced with a particular problem instance, to select a Branch and Bound algorithm from among several promising ones. This method is based on Knuth's sampling method which estimates the efficiency of a backtrack program on a particular instance by iteratively generating random paths in the search tree. We present a simple adaptation of this estimator in the field of combinatorial optimization problems, more precisely for an extension of the maximal constraint satisfaction framework. Experiments both on random and strongly structured instances show that, in most cases, the proposed method is able to select, from a candidate list, the best algorithm for solving a given instance.
---
paper_title: High-level optimization via automated statistical modeling

paper_content:

We develop the use of statistical modeling for portable high-level optimizations such as data layout and algorithm selection. We build the models automatically from profiling information, which ensures robust and accurate models that reflect all aspects of the target platform. We use the models to select among several data layouts for an iterative PDE solver and to select among several sorting algorithms. The selection is correct more than 99% of the time on each of four platforms. In the few cases it selects suboptimally, the selected implementation performs nearly as well; that is, it always makes at least a very good choice. Correct selection is platform and workload dependent and can improve performance by nearly a factor of three. We also use the models to optimize parameters of these applications automatically. In all cases, the models predicted the optimal parameter setting, resulting in improvements ranging up to a factor of three. Finally, we use the models to construct portable high-level libraries , which contain multiple implementations and support for automatic selection and parameter optimization of the fastest implementation for the target platform and workload.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: Using CBR to select solution strategies in constraint programming

paper_content:

Constraint programming is a powerful paradigm that offers many different strategies for solving problems. Choosing a good strategy is difficult; choosing a poor strategy wastes resources and may result in a problem going unsolved. We show how Case-Based Reasoning can be used to select good strategies. We design experiments which demonstrate that, on two problems with quite different characteristics, CBR can outperform four other strategy selection techniques.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: PYTHIA: a knowledge-based system to select scientific algorithms

paper_content:

Problem-solving e nvironments (PSEs) interact with theuser in a language “natural” to the associated discipline,and they provide a high-level abstraction of the underlying,computationally complex model. The knowledge-based system PYTHIAaddresses the problem of (parameter, algorithm) pair selection within ascientific computing domain assuming some minimum user-specifiedcomputational objectives and some characteristics of the given problem.PYTHIA's framework and methodology are general and applicable to anyclass of scientific problems and solvers. PYTHIA is applied in thecontext of Parallel ELLPACK where there are many alternatives for thenumerical solution of elliptic partial differential equations (PDEs).PYTHIA matches the characteristics of the given problem with those ofPDEs in an existing problem population and then uses performanceprofiles of the various solvers to select the appropriate method givenuser-specified error and solution time bounds. The profiles areautomatically generated for each solver of the Parallel ELLPACKlibrary. — Authors' Abstract
---
paper_title: A Context for Constraint Satisfaction Problem Formulation Selection

paper_content:

Much research effort has been applied to finding effective ways for solving constraint satisfaction problems. However, the most fundamental aspect of constraint satisfaction problem solving, problem formulation, has received much less attention. This is important because the selection of an appropriate formulation can have dramatic effects on the efficiency of any constraint satisfaction problem solving algorithm. ::: ::: In this paper, we address the issue of problem formulation. We identify the heuristic nature of generating a good formulation and we propose a context for this process. Our work presents the research community with a focus for the many elements which affect problem formulation and this is illustrated with the example adding redundant constraints. It also provides a significant step towards the goal of automatic selection of problem formulations.
---
paper_title: Rational Deployment of CSP Heuristics

paper_content:

Heuristics are crucial tools in decreasing search effort in varied fields of AI. In order to be effective, a heuristic must be efficient to compute, as well as provide useful information to the search algorithm. However, some well-known heuristics which do well in reducing backtracking are so heavy that the gain of deploying them in a search algorithm might be outweighed by their overhead. ::: ::: We propose a rational metareasoning approach to decide when to deploy heuristics, using CSP backtracking search as a case study. In particular, a value of information approach is taken to adaptive deployment of solution-count estimation heuristics for value ordering. Empirical results show that indeed the proposed mechanism successfully balances the tradeoff between decreasing backtracking and heuristic computational overhead, resulting in a significant overall search time reduction.
---
paper_title: Statistically optimal combination of algorithms

paper_content:

Automatic specialization of algorithms to a limited domain is an interesting and industrially applicable problem. We calculate the optimal assignment of computational resources to several different solvers that solve the same problem. Optimality is considered with regard to the expected solution time on a set of problem instances from the domain of interest. We present two approaches, a static and dynamic one. The static approach leads to a simple analytically calculable solution. The dynamic approach results in formulation of the problem as a Markov Decision Process. Our tests on the SAT Problem show that the presented methods are quite effective. Therefore, both methods are attractive for applications and future research.
---
paper_title: An Instance of Adaptive Constraint Propagation

paper_content:

Constraint propagation algorithms vary in the strength of propagation they apply. This paper investigates a simple configuration for adaptive propagation—the process of varying the strength of propagation to reflect the dynamics of search. We focus on two propagation methods, Arc Consistency (AC) and Forward Checking (FC). AC-based algorithms apply a stronger form of propagation than FC-based algorithms; they invest greater computational effort to detect inconsistent values earlier. The relative payoff of maintaining AC during search as against FC may vary for different constraints and for different intermediate search states. We present a scheme for Adaptive Arc Propagation (AAP) that allows the flexible combination of the two methods. Metalevel reasoning and heuristics are used to dynamically distribute propagation effort between the two. One instance of AAP, Anti-Functional Reduction (AFR), is described in detail here. AFR achieves precisely the same propagation as a pure AC algorithm while significantly improving its average performance. The strategy is to gradually reduce the scope of AC propagation during backtrack search to exclude those arcs that may be subsequently handled as effectively by FC. Experimental results confirm the power of AFR and the validity of adaptive propagation in general.
---
paper_title: Learning dynamic algorithm portfolios

paper_content:

Algorithm selection can be performed using a model of runtime distribution, learned during a preliminary training phase. There is a trade-off between the performance of model-based algorithm selection, and the cost of learning the model. In this paper, we treat this trade-off in the context of bandit problems. We propose a fully dynamic and online algorithm selection technique, with no separate training phase: all candidate algorithms are run in parallel, while a model incrementally learns their runtime distributions. A redundant set of time allocators uses the partially trained model to propose machine time shares for the algorithms. A bandit problem solver mixes the model-based shares with a uniform share, gradually increasing the impact of the best time allocators as the model improves. We present experiments with a set of SAT solvers on a mixed SAT-UNSAT benchmark; and with a set of solvers for the Auction Winner Determination problem.
---
paper_title: A Meta-Learning Method to Select the Kernel Width in Support Vector Regression

paper_content:

The Support Vector Machine algorithm is sensitive to the choice of parameter settings. If these are not set correctly, the algorithm may have a substandard performance. Suggesting a good setting is thus an important problem. We propose a meta-learning methodology for this purpose and exploit information about the past performance of different settings. The methodology is applied to set the width of the Gaussian kernel. We carry out an extensive empirical evaluation, including comparisons with other methods (fixed default ranking; selection based on cross-validation and a heuristic method commonly used to set the width of the SVM kernel). We show that our methodology can select settings with low error while providing significant savings in time. Further work should be carried out to see how the methodology could be adapted to different parameter setting tasks.
---
paper_title: An automated approach to generating efficient constraint solvers

paper_content:

Combinatorial problems appear in numerous settings, from timetabling to industrial design. Constraint solving aims to find solutions to such problems efficiently and automatically. Current constraint solvers are monolithic in design, accepting a broad range of problems. The cost of this convenience is a complex architecture, inhibiting efficiency, extensibility and scalability. Solver components are also tightly coupled with complex restrictions on their configuration, making automated generation of solvers difficult. We describe a novel, automated, model-driven approach to generating efficient solvers tailored to individual problems and present some results from applying the approach. The main contribution of this work is a solver generation framework called Dominion, which analyses a problem and, based on its characteristics, generates a solver using components chosen from a library. The key benefit of this approach is the ability to solve larger and more difficult problems as a result of applying finer-grained optimisations and using specialised techniques as required.
---
paper_title: Learning techniques for Automatic Algorithm Portfolio Selection

paper_content:

The purpose of this paper is to show that a well known machine learning technique based on Decision Trees can be effectively used to select the best approach (in terms of efficiency) in an algorithm portfolio for a particular case study: the Bid Evaluation Problem (BEP) in Combinatorial Auctions. In particular, we are interested in deciding when to use a Constraint Programming (CP) approach and when an Integer Programming (IP) approach, on the basis of the structure of the instance considered. Different instances of the same problem present a different structure, and one aspect (e.g. feasibility or optimality) can prevail on the other. We have extracted from a set of BEP instances, a number of parameters representing the instance structure. Some of them (few indeed) precisely identify the best strategy and its corresponding tuning to be used to face that instance. We will show that this approach is very promising, since it identifies the most efficient algorithm in the 90% of the cases.
---
paper_title: Measuring instance difficulty for combinatorial optimization problems

paper_content:

Discovering the conditions under which an optimization algorithm or search heuristic will succeed or fail is critical for understanding the strengths and weaknesses of different algorithms, and for automated algorithm selection. Large scale experimental studies - studying the performance of a variety of optimization algorithms across a large collection of diverse problem instances - provide the resources to derive these conditions. Data mining techniques can be used to learn the relationships between the critical features of the instances and the performance of algorithms. This paper discusses how we can adequately characterize the features of a problem instance that have impact on difficulty in terms of algorithmic performance, and how such features can be defined and measured for various optimization problems. We provide a comprehensive survey of the research field with a focus on six combinatorial optimization problems: assignment, traveling salesman, and knapsack problems, bin-packing, graph coloring, and timetabling. For these problems - which are important abstractions of many real-world problems - we review hardness-revealing features as developed over decades of research, and we discuss the suitability of more problem-independent landscape metrics. We discuss how the features developed for one problem may be transferred to study related problems exhibiting similar structures.
---
paper_title: Adaptive Online Time Allocation to Search Algorithms

paper_content:

Given is a search problem or a sequence of search problems, as well as a set of potentially useful search algorithms. We propose a general framework for online allocation of computation time to search algorithms based on experience with their performance so far. In an example instantiation, we use simple linear extrapolation of performance for allocating time to various simultaneously running genetic algorithms characterized by different parameter values. Despite the large number of searchers tested in parallel, on various tasks this rather general approach compares favorably to a more specialized state-of-the-art heuristic; in one case it is nearly two orders of magnitude faster.
---
paper_title: A Comparison of Ranking Methods for Classification Algorithm Selection

paper_content:

We investigate the problem of using past performance information to select an algorithm for a given classification problem. We present three ranking methods for that purpose: average ranks, success rate ratios and significant wins. We also analyze the problem of evaluating and comparing these methods. The evaluation technique used is based on a leave-one-out procedure. On each iteration, the method generates a ranking using the results obtained by the algorithms on the training datasets. This ranking is then evaluated by calculating its distance from the ideal ranking built using the performance information on the test dataset. The distance measure adopted here, average correlation, is based on Spearman's rank correlation coefficient. To compare ranking methods, a combination of Friedman's test and Dunn's multiple comparison procedure is adopted. When applied to the methods presented here, these tests indicate that the success rate ratios and average ranks methods perform better than significant wins.
---
paper_title: A Bayesian Approach to Tackling Hard Computational Problems

paper_content:

We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the time required to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We review the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: How to solve it automatically: Selection among problem-solving methods

paper_content:

The choice of an appropriate problem-solving method, from available methods, is a crucial skill for experts in many areas. We describe a technique for the automatic selection among methods, which is based on a statistical analysis of their past performances. ::: ::: We formalize the statistical problem involved in selecting an efficient problem-solving method, derive a solution to this problem, and describe a method-selection algorithm. The algorithm not only chooses among available methods, but also decides when to abandon the chosen method, if it proves to take too much time. We give empirical results on the use of this technique in selecting among search engines in the PRODIGY planning system.
---
paper_title: Knowledge-based constraint-driven software synthesis

paper_content:

A constraint-driven, real-time software synthesis architecture called RT-Syn is described. RT-Syn formulates design space constraints for each task in the real-time software system from timing requirements and a behavior description. RT-Syn then uses these constraints within a simulated-annealing-like approach, selects an abstract implementation for every data structure and algorithm required to implement the desired behavior, and transforms these selections into executable code. Experimental results covering the synthesis of two real-time software tasks that meet the desired constraints are presented. These results illustrate the effectiveness of the simulated-annealing-like approach in searching the software design space and the high reusability and maintainability provided by the use of synthesis technology. >
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: Making choices using structure at the instance level within a case based reasoning framework

paper_content:

We describe using Case Based Reasoning to explore structure at the instance level as a means to distinguish whether to use CP or IP to solve instances of the Bid Evaluation Problem.
---
paper_title: Collaborative Learning for Constraint Solving

paper_content:

Although constraint programming offers a wealth of strong, generalpurpose methods, in practice a complex, real application demands a person who selects, combines, and refines various available techniques for constraint satisfaction and optimization. Although such tuning produces efficient code, the scarcity of human experts slows commercialization. The necessary expertise is of two forms: constraint programming expertise and problem-domain expertise. The former is in short supply, and even experts can be reduced to trial and error prototyping; the latter is difficult to extract. The project described here seeks to automate both the application of constraint programming expertise and the extraction of domain-specific expertise. It applies FORR, an architecture for learning and problem-solving, to constraint solving. FORR develops expertise from multiple heuristics. A successful case study is presented on coloring problems.
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: PRODIGY: an integrated architecture for planning and learning

paper_content:

Artificial intelligence has progressed to the point where multiple cognitive capabilities are being integrated into computational architectures, such as SOAR, PRODIGY, THEO, and ICARUS. This paper reports on the PRODIGY architecture, describing its planning and problem solving capabilities and touching upon its multiple learning methods. Learning in PRODIGY occurs at all decision points and integration in PRODIGY is at the knowledge level; the learning and reasoning modules produce mutually interpretable knowledge structures. Issues in architectural design are discussed, providing a context to examine the underlying tenets of the PRODIGY architecture.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: An automatically configurable portfolio-based planner with macroactions: PbP

paper_content:

While several powerful domain-independent planners have recently been developed, no one of these clearly outperforms all the others in every known benchmark domain. We present PbP, a multi-planner which automatically configures a portfolio of planners by (i) computing some sets of macro-actions for every planner in the portfolio, (ii) selecting a promising combination of planners in the portfolio and relative useful macro-actions, and (iii) defining some running time slots for their round-robin scheduling during planning. The configuration relies on some knowledge about the performance of the planners in the portfolio and relative macro-actions which is automatically generated from a training problem set. PbP entered the learning track of IPC-2008 and was the overall winner of this competition track. An experimental study confirms the effectiveness of PbP, and shows that the learned configuration knowledge is useful for PbP.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: An adaptive algorithm selection framework

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. We present an adaptive algorithm selection framework which can collect and interpret the inputs of a particular instance of a parallel algorithm and select the best performing one from a an existing library. In this paper present the dynamic selection of parallel reduction algorithms. First we introduce a set of high-level parameters that can characterize different parallel reduction algorithms. Then we describe an off-line, systematic process to generate predictive models which can be used for run-time algorithm selection. Our experiments show that our framework: (a) selects the most appropriate algorithms in 85% of the cases studied, (b) overall delievers 98% of the optimal performance, (c) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and (d) adapts to the underlying machine architecture (tested on IBM Regatta and HP V-Class systems).
---
paper_title: PYTHIA: a knowledge-based system to select scientific algorithms

paper_content:

Problem-solving e nvironments (PSEs) interact with theuser in a language “natural” to the associated discipline,and they provide a high-level abstraction of the underlying,computationally complex model. The knowledge-based system PYTHIAaddresses the problem of (parameter, algorithm) pair selection within ascientific computing domain assuming some minimum user-specifiedcomputational objectives and some characteristics of the given problem.PYTHIA's framework and methodology are general and applicable to anyclass of scientific problems and solvers. PYTHIA is applied in thecontext of Parallel ELLPACK where there are many alternatives for thenumerical solution of elliptic partial differential equations (PDEs).PYTHIA matches the characteristics of the given problem with those ofPDEs in an existing problem population and then uses performanceprofiles of the various solvers to select the appropriate method givenuser-specified error and solution time bounds. The profiles areautomatically generated for each solver of the Parallel ELLPACKlibrary. — Authors' Abstract
---
paper_title: Statistically optimal combination of algorithms

paper_content:

Automatic specialization of algorithms to a limited domain is an interesting and industrially applicable problem. We calculate the optimal assignment of computational resources to several different solvers that solve the same problem. Optimality is considered with regard to the expected solution time on a set of problem instances from the domain of interest. We present two approaches, a static and dynamic one. The static approach leads to a simple analytically calculable solution. The dynamic approach results in formulation of the problem as a Markov Decision Process. Our tests on the SAT Problem show that the presented methods are quite effective. Therefore, both methods are attractive for applications and future research.
---
paper_title: An Adaptive Algorithm Selection Framework for Reduction Parallelization

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. In this paper, we present an adaptive algorithm selection framework which can collect and interpret the characteristics of a particular instance of parallel reduction algorithms and select the best performing one from an existing library. The framework consists of the following components: 1) an offline systematic process for characterizing the input sensitivity of parallel reduction algorithms and a method for building corresponding predictive performance models, 2) an online input characterization and algorithm selection module, and 3) a small library of parallel reduction algorithms, which represent the algorithmic choices made available at runtime. We also present one possible integration of this framework in a restructuring compiler. We validate our design experimentally and show that our framework 1) selects the most appropriate algorithms in 85 percent of the cases studied, 2) overall, delivers 98 percent of the optimal performance, 3) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and 4) adapts to the underlying machine architectures (evaluated on IBM Regatta and HP V-class systems)
---
paper_title: The max k-armed bandit: A new model of exploration applied to search heuristic selection

paper_content:

The multiarmed bandit is often used as an analogy for the tradeoff between exploration and exploitation in search problems. The classic problem involves allocating trials to the arms of a multiarmed slot machine to maximize the expected sum of rewards. We pose a new variation of the multiarmed bandit--the Max K-Armed Bandit--in which trials must be allocated among the arms to maximize the expected best single sample reward of the series of trials. Motivation for the Max K-Armed Bandit is the allocation of restarts among a set of multistart stochastic search algorithms. We present an analysis of this Max K-Armed Bandit showing under certain assumptions that the optimal strategy allocates trials to the observed best arm at a rate increasing double exponentially relative to the other arms. This motivates an exploration strategy that follows a Boltzmann distribution with an exponentially decaying temperature parameter. We compare this exploration policy to policies that allocate trials to the observed best arm at rates faster (and slower) than double exponentially. The results confirm, for two scheduling domains, that the double exponential increase in the rate of allocations to the observed best heuristic outperfonns the other approaches.
---
paper_title: Simple Rules for Low-Knowledge Algorithm Selection

paper_content:

This paper addresses the question of selecting an algorithm from a predefined set that will have the best performance on a scheduling problem instance. Our goal is to reduce the expertise needed to apply constraint technology. Therefore, we investigate simple rules that make predictions based on limited problem instance knowledge. Our results indicate that it is possible to achieve superior performance over choosing the algorithm that performs best on average on the problem set. The results hold over a variety of different run lengths and on different types of scheduling problems and algorithms. We argue that low-knowledge approaches are important in reducing expertise required to exploit optimization technology.
---
paper_title: Combining multiple heuristics online

paper_content:

We present black-box techniques for learning how to interleave the execution of multiple heuristics in order to improve average-case performance. In our model, a user is given a set of heuristics whose only observable behavior is their running time. Each heuristic can compute a solution to any problem instance, but its running time varies across instances. The user solves each instance by interleaving runs of the heuristics according to a task-switching schedule. We present (i) exact and approximation algorithms for computing an optimal task-switching schedule offline, (ii) sample complexity bounds for learning a task-switching schedule from training data, and (iii) a no-regret strategy for selecting task-switching schedules online. We demonstrate the power of our results using data from recent solver competitions. We outline how to extend our results to the case in which the heuristics are randomized, and the user may periodically restart each heuristic with a fresh random seed.
---
paper_title: Machine learning for constraint solver design -- A case study for the alldifferent constraint

paper_content:

Constraint solvers are complex pieces of software which require many design decisions to be made by the implementer based on limited information. These decisions affect the performance of the finished solver significantly. Once a design decision has been made, it cannot easily be reversed, although a different decision may be more appropriate for a particular problem. ::: We investigate using machine learning to make these decisions automatically depending on the problem to solve. We use the alldifferent constraint as a case study. Our system is capable of making non-trivial, multi-level decisions that improve over always making a default choice and can be implemented as part of a general-purpose constraint solver.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: Dynamic Algorithm Selection Using Reinforcement Learning

paper_content:

It is often the case that many algorithms exist to solve a single problem, each possessing different performance characteristics. The usual approach in this situation is to manually select the algorithm which has the best average performance. However, this strategy has drawbacks in cases where the optimal algorithm changes during an invocation of the program, in response to changes in the program?s state and the computational environment. This paper presents a prototype tool that uses reinforcement learning to guide algorithm selection at runtime, matching the algorithm used to the current state of the computation. The tool is applied to a simulation similar to those used in some computational chemistry problems. It is shown that the low dimensionality of the problem enables the optimal choice of algorithm to be determined quickly, and that the learning system can react rapidly to phase changes in the target program.
---
paper_title: Branch and Bound Algorithm Selection by Performance Prediction

paper_content:

We propose a method called Selection by Performance Prediction (SPP) which allows one, when faced with a particular problem instance, to select a Branch and Bound algorithm from among several promising ones. This method is based on Knuth's sampling method which estimates the efficiency of a backtrack program on a particular instance by iteratively generating random paths in the search tree. We present a simple adaptation of this estimator in the field of combinatorial optimization problems, more precisely for an extension of the maximal constraint satisfaction framework. Experiments both on random and strongly structured instances show that, in most cases, the proposed method is able to select, from a candidate list, the best algorithm for solving a given instance.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: On Using Reinforcement Learning to Solve Sparse Linear Systems

paper_content:

This paper describes how reinforcement learning can be used to select from a wide variety of preconditioned solvers for sparse linear systems. This approach provides a simple way to consider complex metrics of goodness, and makes it easy to evaluate a wide range of preconditioned solvers. A basic implementation recommends solvers that, when they converge, generally do so with no more than a 17% overhead in time over the best solver possible within the test framework. Potential refinements of, and extensions to, the system are discussed.
---
paper_title: Simple Rules for Low-Knowledge Algorithm Selection

paper_content:

This paper addresses the question of selecting an algorithm from a predefined set that will have the best performance on a scheduling problem instance. Our goal is to reduce the expertise needed to apply constraint technology. Therefore, we investigate simple rules that make predictions based on limited problem instance knowledge. Our results indicate that it is possible to achieve superior performance over choosing the algorithm that performs best on average on the problem set. The results hold over a variety of different run lengths and on different types of scheduling problems and algorithms. We argue that low-knowledge approaches are important in reducing expertise required to exploit optimization technology.
---
paper_title: Switching among Non-Weighting, Clause Weighting, and Variable Weighting in Local Search for SAT

paper_content:

One way to design a local search algorithm that is effective on many types of instances is allowing this algorithm to switch among heuristics. In this paper, we refer to the way in which non-weighting algorithm adaptG2WSAT+ selects a variable to flip, as heuristic adaptG2WSAT+, the way in which clause weighting algorithm RSAPSselects a variable to flip, as heuristic RSAPS, and the way in which variable weighting algorithm VWselects a variable to flip, as heuristic VW. We propose a new switching criterion: the evenness or unevenness of the distribution of clause weights. We apply this criterion, along with another switching criterion previously proposed, to heuristic adaptG2WSAT+, heuristic RSAPS, and heuristic VW. The resulting local search algorithm, which adaptively switches among these three heuristics in every search step according to these two criteria to intensify or diversify the search when necessary, is called NCVW(Non-, Clause, and Variable Weighting). Experimental results show that NCVWis generally effective on a wide range of instances while adaptG2WSAT+, RSAPS, VW, and gNovelty+ and adaptG2WSAT0, which won the gold and silver medals in the satisfiable random category in the SAT 2007 competition, respectively, are not.
---
paper_title: Heuristics for dynamically adapting propagation in constraint satisfaction problems

paper_content:

Building adaptive constraint solvers is a major challenge in constraint programming. An important line of research towards this goal is concerned with ways to dynamically adapt the propagation method applied on the constraints of the problem during search. In this paper we present a heuristic approach to this problem based on the monitoring of propagation events like value deletions and domain wipeouts. We develop a number of heuristics that allow the constraint solver to dynamically switch between a weaker and cheap local consistency and a stronger, but more expensive one, when certain conditions are met. The success of this approach is based on the observation that propagation events for individual constraints in structured problems mostly occur in clusters of nearby revisions. Hence, parts of the search space where certain constraints are highly active can be identified and exploited paving the way for the informed use of constraint propagation techniques. In this paper we first give some experimental results displaying the clustering of propagation events in structured binary CSPs. Then we present simple heuristics that exploit this clustering to efficiently switch between different local consistencies on individual constraints during search. Finally, we make an experimental study on various binary CSPs demonstrating the effectiveness of the proposed heuristics.
---
paper_title: Learning techniques for Automatic Algorithm Portfolio Selection

paper_content:

The purpose of this paper is to show that a well known machine learning technique based on Decision Trees can be effectively used to select the best approach (in terms of efficiency) in an algorithm portfolio for a particular case study: the Bid Evaluation Problem (BEP) in Combinatorial Auctions. In particular, we are interested in deciding when to use a Constraint Programming (CP) approach and when an Integer Programming (IP) approach, on the basis of the structure of the instance considered. Different instances of the same problem present a different structure, and one aspect (e.g. feasibility or optimality) can prevail on the other. We have extracted from a set of BEP instances, a number of parameters representing the instance structure. Some of them (few indeed) precisely identify the best strategy and its corresponding tuning to be used to face that instance. We will show that this approach is very promising, since it identifies the most efficient algorithm in the 90% of the cases.
---
paper_title: Learning When to Use Lazy Learning in Constraint Solving

paper_content:

Learning in the context of constraint solving is a technique by which previously unknown constraints are uncovered during search and used to speed up subsequent search. Recently, lazy learning, similar to a successful idea from satisfiability modulo theories solvers, has been shown to be an effective means of incorporating constraint learning into a solver. Although a powerful technique to reduce search in some circumstances, lazy learning introduces a substantial overhead, which can outweigh its benefits. Hence, it is desirable to know beforehand whether or not it is expected to be useful. We approach this problem using machine learning (ML). We show that, in the context of a large benchmark set, standard ML approaches can be used to learn a simple, cheap classifier which performs well in identifying instances on which lazy learning should or should not be used. Furthermore, we demonstrate significant performance improvements of a system using our classifier and the lazy learning and standard constraint solvers over a standard solver. Through rigorous cross-validation across the different problem classes in our benchmark set, we show the general applicability of our learned classifier.
---
paper_title: A Multi-Engine Solver for Quantified Boolean Formulas

paper_content:

In this paper we study the problem of yielding robust performances from current state-of-the-art solvers for quantified Boolean formulas (QBFs). Building on top of existing QBF solvers, we implement a new multi-engine solver which can inductively learn its solver selection strategy. Experimental results confirm that our solver is always more robust than each single engine, that it is stable with respect to various perturbations, and that such results can be partially explained by a handful of features playing a crucial role in our solver.
---
paper_title: A learning-based algorithm selection meta-reasoner for the real-time MPE problem

paper_content:

The algorithm selection problem aims to select the best algorithm for an input problem instance according to some characteristics of the instance This paper presents a learning-based inductive approach to build a predictive algorithm selection system from empirical algorithm performance data of the Most Probable Explanation(MPE) problem The learned model can serve as an algorithm selection meta-reasoner for the real-time MPE problem Experimental results show that the learned algorithm selection models can help integrate multiple MPE algorithms to gain a better overall performance of reasoning.
---
paper_title: Hybrid regression-classification models for algorithm selection

paper_content:

Many state of the art Algorithm Selection systems use Machine Learning to either predict the run time or a similar performance measure of each of a set of algorithms and choose the algorithm with the best predicted performance or predict the best algorithm directly. We present a technique based on the well-established Machine Learning technique of stacking that combines the two approaches into a new hybrid approach and predicts the best algorithm based on predicted run times. We demonstrate significant performance improvements of up to a factor of six compared to the previous state of the art. Our approach is widely applicable and does not place any restrictions on the performance measure used, the way to predict it or the Machine Learning used to predict the best algorithm. We investigate different ways of deriving new Machine Learning features from the predicted performance measures and evaluate their effectiveness in increasing performance further. We use five different regression algorithms for performance prediction on five data sets from the literature and present strong empirical evidence that shows the effectiveness of our approach.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Towards Low-Cost, High-Accuracy Classifiers for Linear Solver Selection

paper_content:

The time to solve linear systems depends to a large extent on the choice of the solution method and the properties of the coefficient matrix. Although there are several linear solution methods, in most cases it is impossible to predict apriori which linear solver would be best suited for a given linear system. Recent investigations on selecting linear solvers for a given system have explored the use of classification techniques based on the linear system parameters for solver selection. In this paper, we present a method to develop low-cost high-accuracy classifiers. We show that the cost for constructing a classifier can be significantly reduced by focusing on the computational complexity of each feature. In particular, we filter out low information linear system parameters and then order the remaining parameters to decrease the total computation cost for classification at a prescribed accuracy. Our results indicate that the speedup factor of the time to compute the feature set using our ordering can be as high as 262. The accuracy and computation time of the feature set generated using our method is comparable to a near-optimal one, thus demonstrating the effectiveness of our technique.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: Feature Filtering for Instance-Specific Algorithm Configuration

paper_content:

Instance-Specific Algorithm Configuration (ISAC) is a novel general technique for automatically generating and tuning algorithm portfolios. The approach has been very successful in practice, but up to now it has been committed to using all the features it was provided. However, traditional feature filtering techniques are not applicable, requiring multiple computationally expensive tuning steps during the evaluation stage. To this end, we show three new evaluation functions that use precomputed runtimes of a collection of untuned solvers to quickly evaluate subsets of features. One of our proposed functions even shows how to generate such an effective collection of solvers when only one highly parameterized solver is available. Using these new functions, we show that the number of features used by ISAC can be reduced to less than a quarter of the original number while often providing significant performance gains. We present numerical results on both SAT and CP domains.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: Case-Based Reasoning as a Heuristic Selector in a Hyper-Heuristic for Course Timetabling Problems

paper_content:

This paper studies Knowledge Discovery (KD) using Tabu Search and Hill Climbing within Case-Based Reasoning (CBR) as a hyper-heuristic method for course timetabling problems. The aim of the hyper-heuristic is to choose the best heuristic(s) for given timetabling problems according to the knowledge stored in the case base. KD in CBR is a 2-stage iterative process on both case representation and the case base. Experimental results are analysed and related research issues for future work are discussed.
---
paper_title: ODEXPERT: an expert system to select numerical solvers for initial value ODE systems

paper_content:

ODEXPERT is a prototype knowledge-based system which selects the appropriate numerical solvers for initial value ordinary differential equations. It is capable of deriving some knowledge about the input problem by performing automated tests to detect properties and structures in the problem which guide the selection process.
---
paper_title: Genetic algorithm portfolios

paper_content:

Comparative studies of sets of control parameter values are commonly performed when tuning an evolutionary algorithm for a class of problem instances. The standard approach is to identify the most useful set of control parameter settings for a domain. In this paper, we propose an alternative anytime algorithm portfolio technique in which computational resources are allocated among multiple sets of control parameter value settings. We show a method of optimizing such portfolios by applying a bootstrap sampling approach to a database of individual algorithm performance on instances from a problem distribution. Experiments with genetic algorithms applied to the traveling salesperson domain show that the portfolio approach can yield better performance on a distribution of problem instances than the standard approach of trying to identify the single best configuration for the problem class.
---
paper_title: Isac – instance-specific algorithm configuration

paper_content:

We present a new method for instance-specific algorithm configuration (ISAC). It is based on the integration of the algorithm configuration system GGA and the recently proposed stochastic offline programming paradigm. ISAC is provided a solver with categorical, ordinal, and/or continuous parameters, a training benchmark set of input instances for that solver, and an algorithm that computes a feature vector that characterizes any given instance. ISAC then provides high quality parameter settings for any new input instance. Experiments on a variety of different constrained optimization and constraint satisfaction solvers show that automatic algorithm configuration vastly outperforms manual tuning. Moreover, we show that instance-specific tuning frequently leads to significant speed-ups over instance-oblivious configurations.
---
paper_title: Automatically configuring constraint satisfaction programs: A case study

paper_content:

Multi-tac is a learning system that synthesizes heuristic constraint satisfaction programs. The system takes a library of generic algorithms and heuristics and specializes them for a particular application. We present a detailed case study with three different distributions of a single combinatorial problem, “Minimum Maximal Matching”, and show that Muti-tac can synthesize programs for these different distributions that perform on par with hand-coded programs and that exceed the performance of some well-known satisfiability algorithms. In synthesizing a program, Multi-tac bases its choice of heuristics on an instance distribution, and we demonstrate that this capability has a significant impact on the results.
---
paper_title: A Portfolio Solver for Answer Set Programming: Preliminary Report

paper_content:

We propose a portfolio-based solving approach to Answer Set Programming (ASP). Our approach is homogeneous in considering several configurations of the ASP solver clasp. The selection among the configurations is realized via Support Vector Regression. The resulting portfolio-based solver claspfolio regularly outperforms clasp's default configuration as well as manual tuning.
---
paper_title: Knowledge-based constraint-driven software synthesis

paper_content:

A constraint-driven, real-time software synthesis architecture called RT-Syn is described. RT-Syn formulates design space constraints for each task in the real-time software system from timing requirements and a behavior description. RT-Syn then uses these constraints within a simulated-annealing-like approach, selects an abstract implementation for every data structure and algorithm required to implement the desired behavior, and transforms these selections into executable code. Experimental results covering the synthesis of two real-time software tasks that meet the desired constraints are presented. These results illustrate the effectiveness of the simulated-annealing-like approach in searching the software design space and the high reusability and maintainability provided by the use of synthesis technology. >
---
paper_title: A learning-based algorithm selection meta-reasoner for the real-time MPE problem

paper_content:

The algorithm selection problem aims to select the best algorithm for an input problem instance according to some characteristics of the instance This paper presents a learning-based inductive approach to build a predictive algorithm selection system from empirical algorithm performance data of the Most Probable Explanation(MPE) problem The learned model can serve as an algorithm selection meta-reasoner for the real-time MPE problem Experimental results show that the learned algorithm selection models can help integrate multiple MPE algorithms to gain a better overall performance of reasoning.
---
paper_title: Making choices using structure at the instance level within a case based reasoning framework

paper_content:

We describe using Case Based Reasoning to explore structure at the instance level as a means to distinguish whether to use CP or IP to solve instances of the Bid Evaluation Problem.
---
paper_title: Hybrid regression-classification models for algorithm selection

paper_content:

Many state of the art Algorithm Selection systems use Machine Learning to either predict the run time or a similar performance measure of each of a set of algorithms and choose the algorithm with the best predicted performance or predict the best algorithm directly. We present a technique based on the well-established Machine Learning technique of stacking that combines the two approaches into a new hybrid approach and predicts the best algorithm based on predicted run times. We demonstrate significant performance improvements of up to a factor of six compared to the previous state of the art. Our approach is widely applicable and does not place any restrictions on the performance measure used, the way to predict it or the Machine Learning used to predict the best algorithm. We investigate different ways of deriving new Machine Learning features from the predicted performance measures and evaluate their effectiveness in increasing performance further. We use five different regression algorithms for performance prediction on five data sets from the literature and present strong empirical evidence that shows the effectiveness of our approach.
---
paper_title: Selecting Simulation Algorithm Portfolios by Genetic Algorithms

paper_content:

An algorithm portfolio is a set of algorithms that are bundled together for increased overall performance. While being mostly applied to computationally hard problems so far, we investigate portfolio selection for simulation algorithms and focus on their application to adaptive simulation replication. Since the portfolio selection problem is itself hard to solve, we introduce a genetic algorithm to select the most promising portfolios from large sets of simulation algorithms. The effectiveness of this mechanism is evaluated by data from both a realistic performance study and a dedicated test environment.
---
paper_title: The Adaptive Constraint Engine

paper_content:

The Adaptive Constraint Engine (ACE) seeks to automate the application of constraint programming expertise and the extraction of domain-specific expertise. Under the aegis of FORR, an architecture for learning and problem-solving, ACE learns search-order heuristics from problem solving experience. This paper describes ACE?s approach, as well as new experimental results on specific problem classes. ACE is both a test-bed for CSP research and a discovery environment for new algorithms.
---
paper_title: Restart Strategy Selection using Machine Learning Techniques

paper_content:

Restart strategies are an important factor in the performance of conflict-driven Davis Putnam style SAT solvers. Selecting a good restart strategy for a problem instance can enhance the performance of a solver. Inspired by recent success applying machine learning techniques to predict the runtime of SAT solvers, we present a method which uses machine learning to boost solver performance through a smart selection of the restart strategy. Based on easy to compute features, we train both a satisfiability classifier and runtime models. We use these models to choose between restart strategies. We present experimental results comparing this technique with the most commonly used restart strategies. Our results demonstrate that machine learning is effective in improving solver performance.
---
paper_title: Algorithm portfolio selection as a bandit problem with unbounded losses

paper_content:

We propose a method that learns to allocate computation time to a given set of algorithms, of unknown performance, with the aim of solving a given sequence of problem instances in a minimum time. Analogous meta-learning techniques are typically based on models of algorithm performance, learned during a separate offline training sequence, which can be prohibitively expensive. We adopt instead an online approach, named GAMBLETA, in which algorithm performance models are iteratively updated, and used to guide allocation on a sequence of problem instances. GAMBLETA is a general method for selecting among two or more alternative algorithm portfolios. Each portfolio has its own way of allocating computation time to the available algorithms, possibly based on performance models, in which case its performance is expected to improve over time, as more runtime data becomes available. The resulting exploration-exploitation trade-off is represented as a bandit problem. In our previous work, the algorithms corresponded to the arms of the bandit, and allocations evaluated by the different portfolios were mixed, using a solver for the bandit problem with expert advice, but this required the setting of an arbitrary bound on algorithm runtimes, invalidating the optimal regret of the solver. In this paper, we propose a simpler version of GAMBLETA, in which the allocators correspond to the arms, such that a single portfolio is selected for each instance. The selection is represented as a bandit problem with partial information, and an unknown bound on losses. We devise a solver for this game, proving a bound on its expected regret. We present experiments based on results from several solver competitions, in various domains, comparing GAMBLETA with another online method.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: Selecting the Right Heuristic Algorithm: Runtime Performance Predictors

paper_content:

It is obvious that, given a problem instance, some heuristic algorithms can perform vastly better than others; however, in most cases the existing literature provides little guidance for choosing the best heuristic algorithm. This paper describes how runtime performance predictors can be used to identify a good algorithm for a particular problem instance. The approach is demonstrated on two families of heuristic algorithms.
---
paper_title: New Techniques for Algorithm Portfolio Design

paper_content:

We present and evaluate new techniques for designing algorithm portfolios. In our view, the problem has both a scheduling aspect and a machine learning aspect. Prior work has largely addressed one of the two aspects in isolation. Building on recent work on the scheduling aspect of the problem, we present a technique that addresses both aspects simultaneously and has attractive theoretical guarantees. Experimentally, we show that this technique can be used to improve the performance of state-of-the-art algorithms for Boolean satisfiability, zero-one integer programming, and A.I. planning.
---
paper_title: Self-Adapting Linear Algebra Algorithms and Software

paper_content:

One of the main obstacles to the efficient solution of scientific problems is the problem of tuning software, both to the available architecture and to the user problem at hand. We describe approaches for obtaining tuned high-performance kernels and for automatically choosing suitable algorithms. Specifically, we describe the generation of dense and sparse Basic Linear Algebra Subprograms (BLAS) kernels, and the selection of linear solver algorithms. However, the ideas presented here extend beyond these areas, which can be considered proof of concept.
---
paper_title: Branch and Bound Algorithm Selection by Performance Prediction

paper_content:

We propose a method called Selection by Performance Prediction (SPP) which allows one, when faced with a particular problem instance, to select a Branch and Bound algorithm from among several promising ones. This method is based on Knuth's sampling method which estimates the efficiency of a backtrack program on a particular instance by iteratively generating random paths in the search tree. We present a simple adaptation of this estimator in the field of combinatorial optimization problems, more precisely for an extension of the maximal constraint satisfaction framework. Experiments both on random and strongly structured instances show that, in most cases, the proposed method is able to select, from a candidate list, the best algorithm for solving a given instance.
---
paper_title: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions

paper_content:

We propose a new approach for understanding the algorithmspecific empirical hardness of NP-Hard problems. In this work we focus on the empirical hardness of the winner determination problem-an optimization problem arising in combinatorial auctions-when solved by ILOG's CPLEX software. We consider nine widely-used problem distributions and sample randomly from a continuum of parameter settings for each distribution. We identify a large number of distribution-nonspecific features of data instances and use statistical regression techniques to learn, evaluate and interpret a function from these features to the predicted hardness of an instance.
---
paper_title: Feature Filtering for Instance-Specific Algorithm Configuration

paper_content:

Instance-Specific Algorithm Configuration (ISAC) is a novel general technique for automatically generating and tuning algorithm portfolios. The approach has been very successful in practice, but up to now it has been committed to using all the features it was provided. However, traditional feature filtering techniques are not applicable, requiring multiple computationally expensive tuning steps during the evaluation stage. To this end, we show three new evaluation functions that use precomputed runtimes of a collection of untuned solvers to quickly evaluate subsets of features. One of our proposed functions even shows how to generate such an effective collection of solvers when only one highly parameterized solver is available. Using these new functions, we show that the number of features used by ISAC can be reduced to less than a quarter of the original number while often providing significant performance gains. We present numerical results on both SAT and CP domains.
---
paper_title: High-level optimization via automated statistical modeling

paper_content:

We develop the use of statistical modeling for portable high-level optimizations such as data layout and algorithm selection. We build the models automatically from profiling information, which ensures robust and accurate models that reflect all aspects of the target platform. We use the models to select among several data layouts for an iterative PDE solver and to select among several sorting algorithms. The selection is correct more than 99% of the time on each of four platforms. In the few cases it selects suboptimally, the selected implementation performs nearly as well; that is, it always makes at least a very good choice. Correct selection is platform and workload dependent and can improve performance by nearly a factor of three. We also use the models to optimize parameters of these applications automatically. In all cases, the models predicted the optimal parameter setting, resulting in improvements ranging up to a factor of three. Finally, we use the models to construct portable high-level libraries , which contain multiple implementations and support for automatic selection and parameter optimization of the fastest implementation for the target platform and workload.
---
paper_title: PRODIGY: an integrated architecture for planning and learning

paper_content:

Artificial intelligence has progressed to the point where multiple cognitive capabilities are being integrated into computational architectures, such as SOAR, PRODIGY, THEO, and ICARUS. This paper reports on the PRODIGY architecture, describing its planning and problem solving capabilities and touching upon its multiple learning methods. Learning in PRODIGY occurs at all decision points and integration in PRODIGY is at the knowledge level; the learning and reasoning modules produce mutually interpretable knowledge structures. Issues in architectural design are discussed, providing a context to examine the underlying tenets of the PRODIGY architecture.
---
paper_title: Exploiting Competitive Planner Performance

paper_content:

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time.
---
paper_title: SATzilla: Portfolio-based Algorithm Selection for SAT

paper_content:

It has been widely observed that there is no single"dominant"SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.
---
paper_title: An adaptive algorithm selection framework

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. We present an adaptive algorithm selection framework which can collect and interpret the inputs of a particular instance of a parallel algorithm and select the best performing one from a an existing library. In this paper present the dynamic selection of parallel reduction algorithms. First we introduce a set of high-level parameters that can characterize different parallel reduction algorithms. Then we describe an off-line, systematic process to generate predictive models which can be used for run-time algorithm selection. Our experiments show that our framework: (a) selects the most appropriate algorithms in 85% of the cases studied, (b) overall delievers 98% of the optimal performance, (c) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and (d) adapts to the underlying machine architecture (tested on IBM Regatta and HP V-Class systems).
---
paper_title: PYTHIA: a knowledge-based system to select scientific algorithms

paper_content:

Problem-solving e nvironments (PSEs) interact with theuser in a language “natural” to the associated discipline,and they provide a high-level abstraction of the underlying,computationally complex model. The knowledge-based system PYTHIAaddresses the problem of (parameter, algorithm) pair selection within ascientific computing domain assuming some minimum user-specifiedcomputational objectives and some characteristics of the given problem.PYTHIA's framework and methodology are general and applicable to anyclass of scientific problems and solvers. PYTHIA is applied in thecontext of Parallel ELLPACK where there are many alternatives for thenumerical solution of elliptic partial differential equations (PDEs).PYTHIA matches the characteristics of the given problem with those ofPDEs in an existing problem population and then uses performanceprofiles of the various solvers to select the appropriate method givenuser-specified error and solution time bounds. The profiles areautomatically generated for each solver of the Parallel ELLPACKlibrary. — Authors' Abstract
---
paper_title: A self-adaptive multi-engine solver for quantified Boolean formulas

paper_content:

In this paper we study the problem of engineering a robust solver for quantified Boolean formulas (QBFs), i.e., a tool that can efficiently solve formulas across different problem domains without the need for domain-specific tuning. The paper presents two main empirical results along this line of research. Our first result is the development of a multi-engine solver, i.e., a tool that selects among its reasoning engines the one which is more likely to yield optimal results. In particular, we show that syntactic QBF features can be correlated to the performances of existing QBF engines across a variety of domains. We also show how a multi-engine solver can be obtained by carefully picking state-of-the-art QBF solvers as basic engines, and by harnessing inductive reasoning techniques to learn engine-selection policies. Our second result is the improvement of our multi-engine solver with the capability of updating the learned policies when they fail to give good predictions. In this way the solver becomes also self-adaptive, i.e., able to adjust its internal models when the usage scenario changes substantially. The rewarding results obtained in our experiments show that our solver AQME--Adaptive QBF Multi-Engine--can be more robust and efficient than state-of-the-art single-engine solvers, even when it is confronted with previously uncharted formulas and competitors.
---
paper_title: On Using Reinforcement Learning to Solve Sparse Linear Systems

paper_content:

This paper describes how reinforcement learning can be used to select from a wide variety of preconditioned solvers for sparse linear systems. This approach provides a simple way to consider complex metrics of goodness, and makes it easy to evaluate a wide range of preconditioned solvers. A basic implementation recommends solvers that, when they converge, generally do so with no more than a 17% overhead in time over the best solver possible within the test framework. Potential refinements of, and extensions to, the system are discussed.
---
paper_title: An Adaptive Algorithm Selection Framework for Reduction Parallelization

paper_content:

Irregular and dynamic memory reference patterns can cause performance variations for low level algorithms in general and for parallel algorithms in particular. In this paper, we present an adaptive algorithm selection framework which can collect and interpret the characteristics of a particular instance of parallel reduction algorithms and select the best performing one from an existing library. The framework consists of the following components: 1) an offline systematic process for characterizing the input sensitivity of parallel reduction algorithms and a method for building corresponding predictive performance models, 2) an online input characterization and algorithm selection module, and 3) a small library of parallel reduction algorithms, which represent the algorithmic choices made available at runtime. We also present one possible integration of this framework in a restructuring compiler. We validate our design experimentally and show that our framework 1) selects the most appropriate algorithms in 85 percent of the cases studied, 2) overall, delivers 98 percent of the optimal performance, 3) adaptively selects the best algorithms for dynamic phases of a running program (resulting in performance improvements otherwise not possible), and 4) adapts to the underlying machine architectures (evaluated on IBM Regatta and HP V-class systems)
---
paper_title: Learning Rules for Adaptive Planning

paper_content:

This paper presents a novel idea, which combines Planning, Machine Learning and Knowledge-Based techniques. It is concerned with the development of an adaptive planning system that can fine-tune its planning parameters based on the values of specific measurable characteristics of the given planning problem. Adaptation is guided by a rule-based system, whose knowledge has been acquired through machine learning techniques. Specifically, the algorithm of classification based on association rules was applied to a large dataset produced by results from experiments on a large number of problems used in the three AIPS Planning competitions. The paper presents experimental results with the adaptive planner, which demonstrate the boost in performance of the planning system.
---
paper_title: Collaborative Expert Portfolio Management

paper_content:

We consider the task of assigning experts from a portfolio of specialists in order to solve a set of tasks. We apply a Bayesian model which combines collaborative filtering with a feature-based description of tasks and experts to yield a general framework for managing a portfolio of experts. The model learns an embedding of tasks and problems into a latent space in which affinity is measured by the inner product. The model can be trained incrementally and can track non-stationary data, tracking potentially changing expert and task characteristics. The approach allows us to use a principled decision theoretic framework for expert selection, allowing the user to choose a utility function that best suits their objectives. The model component for taking into account the performance feedback data is pluggable, allowing flexibility. Weapply the model to manage a portfolio of algorithms to solve hard combinatorial problems. This is a well studied area and we demonstrate a large improvement on the state of the art in one domain (constraint solving) and in a second domain (combinatorial auctions) created a portfolio that performed significantly better than any single algorithm.
---
paper_title: Learning dynamic algorithm portfolios

paper_content:

Algorithm selection can be performed using a model of runtime distribution, learned during a preliminary training phase. There is a trade-off between the performance of model-based algorithm selection, and the cost of learning the model. In this paper, we treat this trade-off in the context of bandit problems. We propose a fully dynamic and online algorithm selection technique, with no separate training phase: all candidate algorithms are run in parallel, while a model incrementally learns their runtime distributions. A redundant set of time allocators uses the partially trained model to propose machine time shares for the algorithms. A bandit problem solver mixes the model-based shares with a uniform share, gradually increasing the impact of the best time allocators as the model improves. We present experiments with a set of SAT solvers on a mixed SAT-UNSAT benchmark; and with a set of solvers for the Auction Winner Determination problem.
---
paper_title: Optimizing time warp simulation with reinforcement learning techniques

paper_content:

Adaptive time warp protocols in the literature are usually based on a pre-defined analytic model of the system, expressed as a closed form function that maps system state to control parameter. The underlying assumption is that this model itself is optimal. In this paper we present a new approach that utilizes reinforcement learning techniques, also known as simulation-based dynamic programming. Instead of assuming an optimal control strategy, the very goal of reinforcement learning is to find the optimal strategy through simulation. A value function that captures the history of system feedbacks is used, and no prior knowledge of the system is required. Our reinforcement learning techniques were implemented in a distributed VLSI simulator with the objective of finding the optimal size of a bounded time window. Our experiments using two benchmark circuits indicated that it was successful in doing so.
---
paper_title: The max k-armed bandit: A new model of exploration applied to search heuristic selection

paper_content:

The multiarmed bandit is often used as an analogy for the tradeoff between exploration and exploitation in search problems. The classic problem involves allocating trials to the arms of a multiarmed slot machine to maximize the expected sum of rewards. We pose a new variation of the multiarmed bandit--the Max K-Armed Bandit--in which trials must be allocated among the arms to maximize the expected best single sample reward of the series of trials. Motivation for the Max K-Armed Bandit is the allocation of restarts among a set of multistart stochastic search algorithms. We present an analysis of this Max K-Armed Bandit showing under certain assumptions that the optimal strategy allocates trials to the observed best arm at a rate increasing double exponentially relative to the other arms. This motivates an exploration strategy that follows a Boltzmann distribution with an exponentially decaying temperature parameter. We compare this exploration policy to policies that allocate trials to the observed best arm at rates faster (and slower) than double exponentially. The results confirm, for two scheduling domains, that the double exponential increase in the rate of allocations to the observed best heuristic outperfonns the other approaches.
---
paper_title: Simple Rules for Low-Knowledge Algorithm Selection

paper_content:

This paper addresses the question of selecting an algorithm from a predefined set that will have the best performance on a scheduling problem instance. Our goal is to reduce the expertise needed to apply constraint technology. Therefore, we investigate simple rules that make predictions based on limited problem instance knowledge. Our results indicate that it is possible to achieve superior performance over choosing the algorithm that performs best on average on the problem set. The results hold over a variety of different run lengths and on different types of scheduling problems and algorithms. We argue that low-knowledge approaches are important in reducing expertise required to exploit optimization technology.
---
paper_title: Learning to select branching rules in the DPLL procedure for satisfiability

paper_content:

Abstract The DPLL procedure is the most popular complete satisfiability (SAT) solver. While its worst case complexity is exponential, the actual running time is greatly affected by the ordering of branch variables during the search. Several branching rules have been proposed, but none is the best in all cases. This work investigates the use of automated methods for choosing the most appropriate branching rule at each node in the search tree. We consider a reinforcement-learning approach where a value function, which predicts the performance of each branching rule in each case, is learned through trial runs on a typical problem set of the target class of SAT problems. Our results indicate that, provided sufficient training on a given class, the resulting strategy performs as well as (and, in some cases, better than) the best branching rule for that class. Research supported in part by NSF grant IRI-9702576. The first author was also partially supported by the Lilian-Voudouri Foundation in Greece. The authors gratefully acknowledge the influence of Don Loveland, Ron Parr, and Henry Kautz in helping to shape this work.
---
paper_title: A Meta-Learning Method to Select the Kernel Width in Support Vector Regression

paper_content:

The Support Vector Machine algorithm is sensitive to the choice of parameter settings. If these are not set correctly, the algorithm may have a substandard performance. Suggesting a good setting is thus an important problem. We propose a meta-learning methodology for this purpose and exploit information about the past performance of different settings. The methodology is applied to set the width of the Gaussian kernel. We carry out an extensive empirical evaluation, including comparisons with other methods (fixed default ranking; selection based on cross-validation and a heuristic method commonly used to set the width of the SVM kernel). We show that our methodology can select settings with low error while providing significant savings in time. Further work should be carried out to see how the methodology could be adapted to different parameter setting tasks.
---
paper_title: Hybrid regression-classification models for algorithm selection

paper_content:

Many state of the art Algorithm Selection systems use Machine Learning to either predict the run time or a similar performance measure of each of a set of algorithms and choose the algorithm with the best predicted performance or predict the best algorithm directly. We present a technique based on the well-established Machine Learning technique of stacking that combines the two approaches into a new hybrid approach and predicts the best algorithm based on predicted run times. We demonstrate significant performance improvements of up to a factor of six compared to the previous state of the art. Our approach is widely applicable and does not place any restrictions on the performance measure used, the way to predict it or the Machine Learning used to predict the best algorithm. We investigate different ways of deriving new Machine Learning features from the predicted performance measures and evaluate their effectiveness in increasing performance further. We use five different regression algorithms for performance prediction on five data sets from the literature and present strong empirical evidence that shows the effectiveness of our approach.
---
paper_title: 1 An Evaluation of Machine Learning in Algorithm Selection for Search Problems

paper_content:

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.
---
paper_title: K.: Evaluating component solver contributions to portfolio-based algorithm selectors

paper_content:

Portfolio-based methods exploit the complementary strengths of a set of algorithms and--as evidenced in recent competitions--represent the state of the art for solving many NP-hard problems, including SAT. In this work, we argue that a state-of-the-art method for constructing portfolio-based algorithm selectors, $\texttt{SATzilla}$, also gives rise to an automated method for quantifying the importance of each of a set of available solvers. We entered a substantially improved version of $\texttt{SATzilla}$ to the inaugural "analysis track" of the 2011 SAT competition, and draw two main conclusions from the results that we obtained. First, automatically-constructed portfolios of sequential, non-portfolio competition entries perform substantially better than the winners of all three sequential categories. Second, and more importantly, a detailed analysis of these portfolios yields valuable insights into the nature of successful solver designs in the different categories. For example, we show that the solvers contributing most to $\texttt{SATzilla}$ were often not the overall best-performing solvers, but instead solvers that exploit novel solution strategies to solve instances that would remain unsolved without them.
---
paper_title: Measuring instance difficulty for combinatorial optimization problems

paper_content:

Discovering the conditions under which an optimization algorithm or search heuristic will succeed or fail is critical for understanding the strengths and weaknesses of different algorithms, and for automated algorithm selection. Large scale experimental studies - studying the performance of a variety of optimization algorithms across a large collection of diverse problem instances - provide the resources to derive these conditions. Data mining techniques can be used to learn the relationships between the critical features of the instances and the performance of algorithms. This paper discusses how we can adequately characterize the features of a problem instance that have impact on difficulty in terms of algorithmic performance, and how such features can be defined and measured for various optimization problems. We provide a comprehensive survey of the research field with a focus on six combinatorial optimization problems: assignment, traveling salesman, and knapsack problems, bin-packing, graph coloring, and timetabling. For these problems - which are important abstractions of many real-world problems - we review hardness-revealing features as developed over decades of research, and we discuss the suitability of more problem-independent landscape metrics. We discuss how the features developed for one problem may be transferred to study related problems exhibiting similar structures.
---
paper_title: Parallel algorithm configuration

paper_content:

State-of-the-art algorithms for solving hard computational problems often expose many parameters whose settings critically affect empirical performance. Manually exploring the resulting combinatorial space of parameter settings is often tedious and unsatisfactory. Automated approaches for finding good parameter settings are becoming increasingly prominent and have recently lead to substantial improvements in the state of the art for solving a variety of computationally challenging problems. However, running such automated algorithm configuration procedures is typically very costly, involving many thousands of invocations of the algorithm to be configured. Here, we study the extent to which parallel computing can come to the rescue. We compare straightforward parallelization by multiple independent runs with a more sophisticated method of parallelizing the model-based configuration procedure SMAC. Empirical results for configuring the MIP solver CPLEX demonstrate that near-optimal speedups can be obtained with up to 16 parallel workers, and that 64 workers can still accomplish challenging configuration tasks that previously took 2 days in 1---2 hours. Overall, we show that our methods make effective use of large-scale parallel resources and thus substantially expand the practical applicability of algorithm configuration methods.
---
paper_title: Towards distributed algorithm portfolios

paper_content:

In recent work we have developed an online algorithm selection technique, in which a model of algorithm performance is learned incrementally while being used. The resulting exploration-exploitation trade-off is solved as a bandit problem. The candidate solvers are run in parallel on a single machine, as an algorithm portfolio, and computation time is shared among them according to their expected performances. In this paper, we extend our technique to the more interesting and practical case of multiple CPUs.
---
paper_title: Learning Algorithm Portfolios for Parallel Execution

paper_content:

Portfolio-based solvers are both effective and robust, but their promise for parallel execution with constraint satisfaction solvers has received relatively little attention. This paper proposes an approach that constructs algorithm portfolios intended for parallel execution based on a combination of case-based reasoning, a greedy algorithm, and three heuristics. Empirical results show that this method is efficient, and can significantly improve performance with only a few additional processors. On problems from solver competitions, the resultant algorithm portfolios perform nearly as well as an oracle.
---
paper_title: A Portfolio Solver for Answer Set Programming: Preliminary Report

paper_content:

We propose a portfolio-based solving approach to Answer Set Programming (ASP). Our approach is homogeneous in considering several configurations of the ASP solver clasp. The selection among the configurations is realized via Support Vector Regression. The resulting portfolio-based solver claspfolio regularly outperforms clasp's default configuration as well as manual tuning.
---
paper_title: Programming by optimization

paper_content:

Avoid premature commitment, seek design alternatives, and automatically generate performance-optimized software.
---
paper_title: An automated approach to generating efficient constraint solvers

paper_content:

Combinatorial problems appear in numerous settings, from timetabling to industrial design. Constraint solving aims to find solutions to such problems efficiently and automatically. Current constraint solvers are monolithic in design, accepting a broad range of problems. The cost of this convenience is a complex architecture, inhibiting efficiency, extensibility and scalability. Solver components are also tightly coupled with complex restrictions on their configuration, making automated generation of solvers difficult. We describe a novel, automated, model-driven approach to generating efficient solvers tailored to individual problems and present some results from applying the approach. The main contribution of this work is a solver generation framework called Dominion, which analyses a problem and, based on its characteristics, generates a solver using components chosen from a library. The key benefit of this approach is the ability to solve larger and more difficult problems as a result of applying finer-grained optimisations and using specialised techniques as required.
---

Title: Algorithm Selection for Combinatorial Search Problems: A Survey

Section 1: Introduction
Description 1: Provide an overview of the Algorithm Selection Problem and its significance, including historical context and the basic model by Rice.

Section 2: Practical Motivation
Description 2: Discuss the motivation for studying Algorithm Selection, including insights from empirical studies and practical applications.

Section 3: No Free Lunch Theorems
Description 3: Explain the No Free Lunch theorems and their implications for Algorithm Selection.

Section 4: Scope and Related Work
Description 4: Outline the scope of the survey and discuss related work in various fields, with a focus on combinatorial search problems.

Section 5: Terminology
Description 5: Define key terminology used in the context of Algorithm Selection, highlighting variations and specific terms from different research communities.

Section 6: Organisation
Description 6: Describe the organisation of the paper, including the main criteria used for categorisation.

Section 7: Algorithm Portfolios
Description 7: Explore the concept of algorithm portfolios, differentiating between static and dynamic portfolios, and discuss examples and composition strategies.

Section 8: Problem Solving with Portfolios
Description 8: Examine how to decide which algorithms to select and when to use them, including strategies for scheduling and algorithm switching.

Section 9: Portfolio Selectors
Description 9: Review techniques for selecting the best algorithm from a portfolio, including efficiency considerations and specific selector mechanisms.

Section 10: Performance Models
Description 10: Discuss the construction and use of performance models for predicting algorithm performance, distinguishing between per-portfolio and per-algorithm models.

Section 11: Features
Description 11: Investigate the role of problem and algorithm features in Algorithm Selection, including types of features and methods for feature selection.

Section 12: Application Domains
Description 12: Provide an overview of various application domains where Algorithm Selection techniques have been applied, emphasizing combinatorial search problems.

Section 13: Current and Future Directions
Description 13: Outline current research trends and future directions in Algorithm Selection, including the use of sophisticated Machine Learning techniques and new application domains.

Section 14: Summary
Description 14: Summarize the main points of the survey, highlighting key findings and the overall approach to Algorithm Selection for combinatorial search problems.