BibTeX
@MISC{Nikolaou_fastoptimization,
author = {Nikolaos Nikolaou},
title = {Fast optimization of non-convex Machine Learning objectives},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In this project we examined the problem of non-convex optimization in the context of Machine Learning, drawing inspiration from the increasing popularity of methods such as Deep Belief Networks, which involve non-convex objectives. We focused on the task of training the Neural Autoregressive Distribution Estimator, a recently proposed variant of the Restricted Boltzmann Machine, in applications to density estimation. The aim of the project was to explore the various stages involved in implementing optimization methods and choosing the appropriate one for a given task. We examined a number of optimization methods, ranging from derivative-free to second order and from batch to stochastic. We experimented with variations of these methods, presenting along the way all the major steps and decisions involved. The challenges of the problem included the relatively large parameter space and the non-convexity of the objective function, the large size of some of the datasets we used, the multitude of hyperparameters and decisions involved in each method, as well as the ever-present danger of overfitting the data. Our results show that second order Quasi-Newton batch methods like
Keyphrases
non-convex machine learning objective fast optimization optimization method large size second order machine learning non-convex optimization objective function various stage neural autoregressive distribution estimator major step appropriate one ever-present danger non-convex objective restricted boltzmann machine second order quasi-newton batch method large parameter space deep belief network
