Results 1  10
of
24
Greedy layerwise training of deep networks
 In NIPS
, 2007
"... Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multilayer neural networks have many levels of nonlinearities allow ..."
Abstract

Cited by 184 (32 self)
 Add to MetaCart
Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multilayer neural networks have many levels of nonlinearities allowing them to compactly represent highly nonlinear and highlyvarying functions. However, until recently it was not clear how to train such deep networks, since gradientbased optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layerwise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layerwise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are highlevel abstractions of the input, bringing better generalization.
Logics with Aggregate Operators
 Journal of the ACM
"... We study adding aggregate operators, such as summing up elements of a column of a relation, to logics with counting mechanisms. The primary motivation comes from database applications, where aggregate operators are present in all real life query languages. Unlike other features of query languages, a ..."
Abstract

Cited by 24 (12 self)
 Add to MetaCart
We study adding aggregate operators, such as summing up elements of a column of a relation, to logics with counting mechanisms. The primary motivation comes from database applications, where aggregate operators are present in all real life query languages. Unlike other features of query languages, aggregates are not adequately captured by the existing logical formalisms. Consequently, all previous approaches to analyzing the expressive power of aggregation were only capable of producing partial results, depending on the allowed class of aggregate and arithmetic operations. We consider a powerful counting logic, and extend it with the set of all aggregate operators. We show that the resulting logic satis es analogs of Hanf's and Gaifman's theorems, meaning that it can only express local properties. We consider a database query language that expresses all the standard aggregates found in commercial query languages, and show how it can be translated into the aggregate logic, thereby pro...
Representational power of restricted boltzmann machines and deep belief networks
, 2007
"... Deep Belief Networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton et al., along with a greedy layerwise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a Restricted Boltzmann Mac ..."
Abstract

Cited by 21 (6 self)
 Add to MetaCart
Deep Belief Networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton et al., along with a greedy layerwise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a Restricted Boltzmann Machine (RBM), used to represent one layer of the model. Restricted Boltzmann Machines are interesting because inference is easy in them, and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs. 1
Separating complexity classes using structural properties
 In Proceedings of the 19th IEEE Conference on Computational Complexity
, 2004
"... We study the robustness of complete sets for various complexity classes. A complete set A is robust if for any f(n)dense set S ∈ P, A − S is still complete, where f(n) ranges from log(n), polynomial, to subexponential. We show that robustness can be used to separate complexity classes: • for every ..."
Abstract

Cited by 9 (2 self)
 Add to MetaCart
We study the robustness of complete sets for various complexity classes. A complete set A is robust if for any f(n)dense set S ∈ P, A − S is still complete, where f(n) ranges from log(n), polynomial, to subexponential. We show that robustness can be used to separate complexity classes: • for every ≤ p mcomplete set A for EXP and any subexponential dense sets S ∈ P, A − S is still Turing complete and under a reasonable hardness assumption even ≤ p mcomplete. • For EXP and the delta levels of the exponential hierarchy we show that for every Turing complete set A and any logdense set S ∈ P, A − S is still Turing complete. • There exists a 3truthtable complete set A for EEXPSPACE, and a logdense set S ∈ P such that A − S is not Turing complete. This implies that settling this issue for EEXP will either separate P from PSPACE or PH from EXP. • We show that the robustness results for EXP and the delta levels of the exponential hierarchy do not relativize. 1
Unary Quantifiers, Transitive Closure, and Relations of Large Degree
"... This paper studies expressivity bounds for extensions of firstorder logic with counting and unary quantifiers in the presence of relations of large degree. There are several motivations for this work. First, it is known that firstorder logic with counting quantifiers captures uniform TC 0 over ..."
Abstract

Cited by 8 (4 self)
 Add to MetaCart
This paper studies expressivity bounds for extensions of firstorder logic with counting and unary quantifiers in the presence of relations of large degree. There are several motivations for this work. First, it is known that firstorder logic with counting quantifiers captures uniform TC 0 over ordered structures. Thus, proving expressivity bounds for firstorder with counting can be seen as an attempt to show TC 0 $ DLOG using techniques of descriptive complexity. Second, the presence of auxiliary builtin relations (e.g., order, successor) is known to make a big impact on expressivity results in finitemodel theory and database theory (where logics with counting and unary quantifiers have recently been used to model query languages with aggregation). For those logics, our goal is to extend techniques from "pure" setting to that of auxiliary relations. Until now, all known results on the limitations of expressive power of the counting and unary quantifier extensions of firstorder...
Logic meets algebra: the case of regular languages
 Logical Methods in Computer Science
"... Abstract. The study of finite automata and regular languages is a privileged meeting point of algebra and logic. Since the work of Büchi, regular languages have been classified according to their descriptive complexity, i.e. the type of logical formalism required to define them. The algebraic point ..."
Abstract

Cited by 8 (1 self)
 Add to MetaCart
Abstract. The study of finite automata and regular languages is a privileged meeting point of algebra and logic. Since the work of Büchi, regular languages have been classified according to their descriptive complexity, i.e. the type of logical formalism required to define them. The algebraic point of view on automata is an essential complement of this classification: by providing alternative, algebraic characterizations for the classes, it often yields the only opportunity for the design of algorithms that decide expressibility in some logical fragment. We survey the existing results relating the expressibility of regular languages in logical fragments of MSO[S] with algebraic properties of their minimal automata. In particular, we show that many of the best known results in this area share the same underlying mechanics and rely on a very strong relation between logical substitutions and blockproducts of pseudovarieties of monoid. We also explain the impact of these connections on circuit complexity theory. 1.
Lower Bounds for Invariant Queries in Logics with Counting
 TCS
, 2002
"... We study the expressive power of counting logics in the presence of auxiliary relations such as orders and preorders. The simplest such logic, firstorder with counting, captures the complexity class TC 0 over ordered structures. We also consider firstorder logic with arbitrary unary quantifie ..."
Abstract

Cited by 5 (2 self)
 Add to MetaCart
We study the expressive power of counting logics in the presence of auxiliary relations such as orders and preorders. The simplest such logic, firstorder with counting, captures the complexity class TC 0 over ordered structures. We also consider firstorder logic with arbitrary unary quantifiers, and infinitary extensions. We start by giving a simple direct proof that firstorder with counting, in the presence of preorders that are almosteverywhere linear orders, cannot express the transitive closure of a binary relation. The proof is based on locality of formulae. We then show that the technique cannot be extended to linear orders, and that the result does not say anything about the power of invariant queries in firstorder with counting, in the presence of those preorders, vs. the class TC 0 . In the second part of the paper we then prove a separation result showing that for all the counting logics above, a linear order is more powerful than a preorder that is a linea...
Counting and Addition cannot express Deterministic Transitive Closure
 In Proceedings of 14th IEEE Symposium on Logic in Computer Science
, 1999
"... An important open question in complexity theory is whether the circuit complexity class TC 0 is (strictly) weaker than LOGSPACE. This paper considers this question from the viewpoint of descriptive complexity theory. TC 0 can be characterized as the class of queries expressible by the logic FOC( ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
An important open question in complexity theory is whether the circuit complexity class TC 0 is (strictly) weaker than LOGSPACE. This paper considers this question from the viewpoint of descriptive complexity theory. TC 0 can be characterized as the class of queries expressible by the logic FOC(<;+;), which is firstorder logic augmented by counting quantifiers on ordered structures that have addition and multiplication predicates. We show that in firstorder logic with counting quantifiers and only an addition predicate it is not possible to express "deterministic transitive closure" on ordered structures. As this is a LOGSPACEcomplete problem, this logic therefore fails to capture LOGSPACE. It also directly follows from our proof that in the presence of counting quantifiers, multiplication cannot be expressed in terms of addition and ordering alone. 1. Introduction The interest in finite model theory from a complexity theory point of view is motivated by the fact that "descrip...
Some Pointed Questions Concerning Asymptotic Lower Bounds, And News From The Isomorphism Front
 Current Trends in Theoretical Computer Science
, 2001
"... this article, we now know that such problems are all isomorphic to the standard complete set for the complexity class, under depththree AC ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
this article, we now know that such problems are all isomorphic to the standard complete set for the complexity class, under depththree AC