Results 1 - 10
of
10
Input Warping for Bayesian Optimization of Non-Stationary Functions
"... Bayesian optimization has proven to be a highly effective methodology for the global optimiza-tion of unknown, expensive and multimodal functions. The ability to accurately model distri-butions over functions is critical to the effective-ness of Bayesian optimization. Although Gaus-sian processes pr ..."
Abstract
-
Cited by 9 (2 self)
- Add to MetaCart
Bayesian optimization has proven to be a highly effective methodology for the global optimiza-tion of unknown, expensive and multimodal functions. The ability to accurately model distri-butions over functions is critical to the effective-ness of Bayesian optimization. Although Gaus-sian processes provide a flexible prior over func-tions, there are various classes of functions that remain difficult to model. One of the most fre-quently occurring of these is the class of non-stationary functions. The optimization of the hy-perparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in “log-space, ” to mitigate the effects of spatially-varying length scale. We develop a methodology for automatically learning a wide family of bijective transformations or warpings of the input space using the Beta cumulative dis-tribution function. We further extend the warp-ing framework to multi-task Bayesian optimiza-tion so that multiple tasks can be warped into a jointly stationary space. On a set of challeng-ing benchmark optimization tasks, we observe that the inclusion of warping greatly improves on the state-of-the-art, producing better results faster and more reliably.
Eliasmith C. Hyperopt-sklearn: automatic hyperparameter configuration for scikit-learn
- In: Proceedings of SciPy
, 2014
"... Abstract—Hyperopt-sklearn is a new software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a singl ..."
Abstract
-
Cited by 3 (0 self)
- Add to MetaCart
(Show Context)
Abstract—Hyperopt-sklearn is a new software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a single large hyper-parameter optimization problem. We use Hyperopt to define a search space that encompasses many standard components (e.g. SVM, RF, KNN, PCA, TFIDF) and common patterns of composing them together. We demonstrate, using search algorithms in Hyperopt and standard benchmarking data sets (MNIST, 20-Newsgroups, Convex Shapes), that searching this space is practical and effective. In particular, we improve on best-known scores for the model space for both MNIST and Convex Shapes. Index Terms—bayesian optimization, model selection, hyperparameter opti-mization, scikit-learn
An entropy search portfolio for bayesian optimization. arXiv:1406.4625
, 2014
"... Bayesian optimization is a sample-efficient method for black-box global optimization. How-ever, the performance of a Bayesian optimiza-tion method very much depends on its explo-ration strategy, i.e. the choice of acquisition func-tion, and it is not clear a priori which choice will result in superi ..."
Abstract
-
Cited by 2 (2 self)
- Add to MetaCart
(Show Context)
Bayesian optimization is a sample-efficient method for black-box global optimization. How-ever, the performance of a Bayesian optimiza-tion method very much depends on its explo-ration strategy, i.e. the choice of acquisition func-tion, and it is not clear a priori which choice will result in superior performance. While portfolio methods provide an effective, principled way of combining a collection of acquisition functions, they are often based on measures of past perfor-mance which can be misleading. To address this issue, we introduce the Entropy Search Portfolio (ESP): a novel approach to portfolio construction which is motivated by information theoretic con-siderations. We show that ESP outperforms ex-isting portfolio methods on several real and syn-thetic problems, including geostatistical datasets and simulated control tasks. We not only show that ESP is able to offer performance as good as the best, but unknown, acquisition function, but surprisingly it often gives better performance. Fi-nally, over a wide range of conditions we find that ESP is robust to the inclusion of poor acquisition functions. 1
Bayesopt: a bayesian optimization library for nonlinear optimization, experimental design and bandits. arXiv preprint arXiv:1405.7430
, 2014
"... ar ..."
(Show Context)
Modular mechanisms for Bayesian optimization
"... The design of methods for Bayesian optimization involves a great number of choices that are often implicit in the overall algorithm design. In this work we argue for a modular approach to Bayesian optimization and present a Python im-plementation, pybo, that allows us to easily vary these choices. I ..."
Abstract
-
Cited by 1 (0 self)
- Add to MetaCart
(Show Context)
The design of methods for Bayesian optimization involves a great number of choices that are often implicit in the overall algorithm design. In this work we argue for a modular approach to Bayesian optimization and present a Python im-plementation, pybo, that allows us to easily vary these choices. In particular this includes selection of the acquisition function, kernel, and hyperpriors as well as less-discussed components such as the recommendation and initialization strate-gies. Ultimately this approach provides us a straightforward mechanism to exam-ine the effect of each choice both individually and in combination. 1
Heteroscedastic Treed Bayesian Optimisation
"... We propose new hierarchical models and es-timation techniques to solve the problem of heteroscedasticity in Bayesian optimisation. Our results demonstrate substantial gains in a wide range of applications, including au-tomatic machine learning and mining explo-ration. 1 ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
We propose new hierarchical models and es-timation techniques to solve the problem of heteroscedasticity in Bayesian optimisation. Our results demonstrate substantial gains in a wide range of applications, including au-tomatic machine learning and mining explo-ration. 1
ON THE POTENTIAL OF SIMPLE FRAMEWISE APPROACHES TO PIANO TRANSCRIPTION
"... ABSTRACT In an attempt at exploring the limitations of simple approaches to the task of piano transcription (as usually defined in MIR), we conduct an in-depth analysis of neural network-based framewise transcription. We systematically compare different popular input representations for transcripti ..."
Abstract
- Add to MetaCart
(Show Context)
ABSTRACT In an attempt at exploring the limitations of simple approaches to the task of piano transcription (as usually defined in MIR), we conduct an in-depth analysis of neural network-based framewise transcription. We systematically compare different popular input representations for transcription systems to determine the ones most suitable for use with neural networks. Exploiting recent advances in training techniques and new regularizers, and taking into account hyper-parameter tuning, we show that it is possible, by simple bottom-up frame-wise processing, to obtain a piano transcriber that outperforms the current published state of the art on the publicly available MAPS dataset -without any complex post-processing steps. Thus, we propose this simple approach as a new baseline for this dataset, for future transcription research to build on and improve.
Bilevel Optimization with Nonsmooth Lower Level Problems
"... Abstract. We consider a bilevel optimization approach for parameter learning in nonsmooth variational models. Existing approaches solve this problem by applying implicit differentiation to a sufficiently smooth ap-proximation of the nondifferentiable lower level problem. We propose an alternative me ..."
Abstract
- Add to MetaCart
(Show Context)
Abstract. We consider a bilevel optimization approach for parameter learning in nonsmooth variational models. Existing approaches solve this problem by applying implicit differentiation to a sufficiently smooth ap-proximation of the nondifferentiable lower level problem. We propose an alternative method based on differentiating the iterations of a nonlin-ear primal–dual algorithm. Our method computes exact (sub)gradients and can be applied also in the nonsmooth setting. We show preliminary results for the case of multi-label image segmentation. 1
AutoFolio: An Automatically Configured Algorithm Selector
"... Algorithm selection (AS) techniques – which involve choosing from a set of algorithms the one expected to solve a given problem instance most efficiently – have substantially improved the state of the art in solving many prominent AI problems, such as SAT, CSP, ASP, MAXSAT and QBF. Although several ..."
Abstract
- Add to MetaCart
(Show Context)
Algorithm selection (AS) techniques – which involve choosing from a set of algorithms the one expected to solve a given problem instance most efficiently – have substantially improved the state of the art in solving many prominent AI problems, such as SAT, CSP, ASP, MAXSAT and QBF. Although several AS procedures have been introduced, not too surprisingly, none of them dominates all others across all AS scenarios. Furthermore, these procedures have parameters whose optimal values vary across AS scenarios. This holds specifically for the machine learning techniques that form the core of current AS proce-dures, and for their hyperparameters. Therefore, to successfully apply AS to new problems, algorithms and benchmark sets, two questions need to be answered: (i) how to select an AS approach and (ii) how to set its parameters effectively. We address both of these problems simultaneously by using automated algorithm configuration. Specifically, we demonstrate that we can automatically configure claspfolio 2, which implements a large variety of different AS approaches and their respective parameters in a single, highly-parameterized algorithm framework. Our approach, dubbed AutoFolio, allows researchers and practi-
BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits
"... BayesOpt is a library with state-of-the-art Bayesian optimization methods to solve nonlin-ear optimization, stochastic bandits or sequential experimental design problems. Bayesian optimization characterized for being sample efficient as it builds a posterior distribution to capture the evidence and ..."
Abstract
- Add to MetaCart
(Show Context)
BayesOpt is a library with state-of-the-art Bayesian optimization methods to solve nonlin-ear optimization, stochastic bandits or sequential experimental design problems. Bayesian optimization characterized for being sample efficient as it builds a posterior distribution to capture the evidence and prior knowledge of the target function. Built in standard C++, the library is extremely efficient while being portable and flexible. It includes a common interface for C, C++, Python, Matlab and Octave.
