Results 1 
6 of
6
Agnostic bayesian learning of ensembles
 In Proceedings of The 31st International Conference on Machine Learning
, 2014
"... We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
We propose a method for producing ensembles of predictors based on holdout estimations of their generalization performances. This approach uses a prior directly on the performance of predictors taken from a finite set of candidates and attempts to infer which one is best. Using Bayesian inference, we can thus obtain a posterior that represents our uncertainty about that choice and construct a weighted ensemble of predictors accordingly. This approach has the advantage of not requiring that the predictors be probabilistic themselves, can deal with arbitrary measures of performance and does not assume that the data was actually generated from any of the predictors in the ensemble. Since the problem of finding the best (as opposed to the true) predictor among a class is known as agnostic PAClearning, we refer to our method as agnostic Bayesian learning. We also propose a method to address the case where the performance estimate is obtained from kfold cross validation. While being efficient and easily adjustable to any loss function, our experiments confirm that the agnostic Bayes approach is state of the art compared to common baselines such as model selection based on kfold crossvalidation or a learned linear combination of predictor outputs.
Model Averaging With Holdout Estimation of the Posterior Distribution
"... The holdout estimation of the expected loss of a model is biased and noisy. Yet, practicians often rely on it to select the model to be used for further predictions. Repeating the learning phase with small variations of the training set, reveals a variation on the selected model which then induces a ..."
Abstract
 Add to MetaCart
(Show Context)
The holdout estimation of the expected loss of a model is biased and noisy. Yet, practicians often rely on it to select the model to be used for further predictions. Repeating the learning phase with small variations of the training set, reveals a variation on the selected model which then induces an important variation of the final test performances. Thus, we propose a small modification to the kfold crossvalidation that greatly enhances the generalization performances of the final predictor. Instead of using the empirical average of the validation losses to select a single model, we propose to use bootstrap to resample the validation losses (without retraining). The variations in the selected models induce a posterior distribution that is then used for model averaging. Comparing this novel approach to the classical crossvalidation on 38 datasets with a significance test, shows that it has higher generalization performance with probability over 0.9. 1
Noname manuscript No.
"... (will be inserted by the editor) A Bayesian approach for comparing crossvalidated algorithms on multiple data sets ..."
Abstract
 Add to MetaCart
(Show Context)
(will be inserted by the editor) A Bayesian approach for comparing crossvalidated algorithms on multiple data sets
Bayesian hypothesis testing in machine learning
"... Abstract. Most hypothesis testing in machine learning is done using the frequentist nullhypothesis significance test, which has severe drawbacks. We review recent Bayesian tests which overcome the drawbacks of the frequentist ones. ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract. Most hypothesis testing in machine learning is done using the frequentist nullhypothesis significance test, which has severe drawbacks. We review recent Bayesian tests which overcome the drawbacks of the frequentist ones.
Collaborative hyperparameter tuning
"... Hyperparameter learning has traditionally been a manual task because of the limited number of trials. Today’s computing infrastructures allow bigger evaluation budgets, thus opening the way for algorithmic approaches. Recently, surrogatebased optimization was successfully applied to hyperparameter ..."
Abstract
 Add to MetaCart
(Show Context)
Hyperparameter learning has traditionally been a manual task because of the limited number of trials. Today’s computing infrastructures allow bigger evaluation budgets, thus opening the way for algorithmic approaches. Recently, surrogatebased optimization was successfully applied to hyperparameter learning for deep belief networks and to WEKA classifiers. The methods combined brute force computational power with model building about the behavior of the error function in the hyperparameter space, and they could significantly improve on manual hyperparameter tuning. What may make experienced practitioners even better at hyperparameter optimization is their ability to generalize across similar learning problems. In this paper, we propose a generic method to incorporate knowledge from previous experiments when simultaneously tuning a learning algorithm on new problems at hand. To this end, we combine surrogatebased ranking and optimization techniques for surrogatebased collaborative tuning (SCoT). We demonstrate SCoT in two experiments where it outperforms standard tuning techniques and singleproblem surrogatebased optimization. Proceedings of the 30 th
To cite this version:
, 2013
"... HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte p ..."
Abstract
 Add to MetaCart
(Show Context)
HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et a ̀ la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Collaborative hyperparameter tuning