Results 1  10
of
20
Provable bounds for learning some deep representations.
 ArXiv:1310.6343,
, 2013
"... Abstract We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an n node multilayer network that has degree at most n γ for some γ < 1 and each edge has a random edge weight in [−1, 1]. O ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
(Show Context)
Abstract We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an n node multilayer network that has degree at most n γ for some γ < 1 and each edge has a random edge weight in [−1, 1]. Our algorithm learns almost all networks in this class with polynomial running time. The sample complexity is quadratic or cubic depending upon the details of the model. The algorithm uses layerwise learning. It is based upon a novel idea of observing correlations among features and using these to infer the underlying edge structure via a global graph recovery procedure. The analysis of the algorithm reveals interesting structure of neural nets with random edge weights.
SumofSquares Proofs and the Quest toward Optimal Algorithms
"... Abstract. In order to obtain the bestknown guarantees, algorithms are traditionally tailored to the particular problem we want to solve. Two recent developments, the Unique Games Conjecture (UGC) and the SumofSquares (SOS) method, surprisingly suggest that this tailoring is not necessary and that ..."
Abstract

Cited by 6 (0 self)
 Add to MetaCart
Abstract. In order to obtain the bestknown guarantees, algorithms are traditionally tailored to the particular problem we want to solve. Two recent developments, the Unique Games Conjecture (UGC) and the SumofSquares (SOS) method, surprisingly suggest that this tailoring is not necessary and that a single efficient algorithm could achieve best possible guarantees for a wide range of different problems. The Unique Games Conjecture (UGC) is a tantalizing conjecture in computational complexity, which, if true, will shed light on the complexity of a great many problems. In particular this conjecture predicts that a single concrete algorithm provides optimal guarantees among all efficient algorithms for a large class of computational problems. The SumofSquares (SOS) method is a general approach for solving systems of polynomial constraints. This approach is studied in several scientific disciplines, including real algebraic geometry, proof complexity, control theory, and mathematical programming, and has found applications in fields as diverse as quantum information theory, formal verification, game theory and many others. We survey some connections that were recently uncovered between the Unique Games Conjecture and the SumofSquares method. In particular, we discuss new tools to rigorously bound the running time of the SOS method for obtaining approximate solutions to hard optimization problems, and how these tools give the potential for the sumofsquares method to provide new guarantees for many problems of interest, and possibly to even refute the UGC.
Guaranteed NonOrthogonal Tensor Decomposition via Alternating Rank1 Updates. arXiv preprint arXiv:1402.5180,
, 2014
"... Abstract A simple alternating rank1 update procedure is considered for CP tensor decomposition. Local convergence guarantees are established for third order tensors of rank k in d dimensions, when k = o(d 1.5 ) and the tensor components are incoherent. We strengthen the results to global converge ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
(Show Context)
Abstract A simple alternating rank1 update procedure is considered for CP tensor decomposition. Local convergence guarantees are established for third order tensors of rank k in d dimensions, when k = o(d 1.5 ) and the tensor components are incoherent. We strengthen the results to global convergence guarantees when k = O(d) through a simple initialization procedure based on rank1 singular value decomposition of random tensor slices. Our tight perturbation analysis leads to efficient sample guarantees for unsupervised learning of discrete multiview mixtures when k = O(d), where k is the number of mixture components and d is the observed dimension. For learning overcomplete decompositions (k = ω(d)), we prove that having an extremely small number of labeled samples, scaling as polylog(k) for each label, under the semisupervised setting (where the label corresponds to the choice variable in the mixture model) leads to global convergence guarantees for learning mixture models.
Nearest Neighbors Using Compact Sparse Codes
, 2014
"... HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte p ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
(Show Context)
HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et a ̀ la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
On some provably correct cases of variational inference for topic models.
 In NIPS,
, 2015
"... Abstract Variational inference is an efficient, popular heuristic used in the context of latent variable models. We provide the first analysis of instances where variational inference algorithms converge to the global optimum, in the setting of topic models. Our initializations are natural, one of ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
Abstract Variational inference is an efficient, popular heuristic used in the context of latent variable models. We provide the first analysis of instances where variational inference algorithms converge to the global optimum, in the setting of topic models. Our initializations are natural, one of them being used in LDAc, the most popular implementation of variational inference. In addition to providing intuition into why this heuristic might work in practice, the multiplicative, rather than additive nature of the variational inference updates forces us to use nonstandard proof arguments, which we believe might be of general theoretical interest.
1Local Identification of Overcomplete Dictionaries
"... This paper presents the first theoretical results showing that stable identification of overcomplete µcoherent dictionaries Φ ∈ Rd×K is locally possible from training signals with sparsity levels S up to the order O(µ−2) and signal to noise ratios up to O( d). In particular the dictionary is recove ..."
Abstract
 Add to MetaCart
This paper presents the first theoretical results showing that stable identification of overcomplete µcoherent dictionaries Φ ∈ Rd×K is locally possible from training signals with sparsity levels S up to the order O(µ−2) and signal to noise ratios up to O( d). In particular the dictionary is recoverable as the local maximum of a new maximisation criterion that generalises the Kmeans criterion. For this maximisation criterion results for asymptotic exact recovery for sparsity levels up toO(µ−1) and stable recovery for sparsity levels up to O(µ−2) as well as signal to noise ratios up to O( d) are provided. These asymptotic results translate to finite sample size recovery results with high probability as long as the sample size N scales as O(K3dSε̃−2), where the recovery precision ε ̃ can go down to the asymptotically achievable precision. Further to actually find the local maxima of the new criterion, a very simple Iterative Thresholding & K (signed) Means algorithm (ITKM), which has complexity O(dKN) in each iteration, is presented and its local efficiency is demonstrated in several experiments. Index Terms dictionary learning, dictionary identification, sparse coding, sparse component analysis, vector quantisation, Kmeans, finite sample size, sampling complexity, maximisation criterion, sparse representation 1