Results 1 - 10 of 417

Greedy layer-wise training of deep networks

by Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle , 2006
"... Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allow ..."
Abstract - Cited by 394 (48 self) - Add to MetaCart
Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non

Deep Sparse Rectifier Neural Networks

by Xavier Glorot, Antoine Bordes, Yoshua Bengio
"... While logistic sigmoid neurons are more biologically plausible than hyperbolic tangent neurons, the latter work better for training multi-layer neural networks. This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbol ..."
Abstract - Cited by 57 (17 self) - Add to MetaCart
hyperbolic tangent networks in spite of the hard non-linearity and non-differentiability at zero, creating sparse representations with true zeros, which seem remarkably suitable for naturally sparse data. Even though they can take advantage of semi-supervised setups with extra-unlabeled data, deep rectifier

Exact solutions to the nonlinear dynamics of learning in deep linear neural network

by Andrew M. Saxe, James L. Mcclelland, Surya Ganguli - In International Conference on Learning Representations , 2014
"... networks ..."
Abstract - Cited by 15 (2 self) - Add to MetaCart
networks

Multi-column deep neural networks for image classification

by Dan Ciresan, Ueli Meier, Jürgen Schmidhuber - IN PROCEEDINGS OF THE 25TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2012 , 2012
"... Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional win ..."
Abstract - Cited by 151 (9 self) - Add to MetaCart
winnertake-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions

Sparse deep belief net model for visual area V2

by Chaitanya Ekanadham - Advances in Neural Information Processing Systems 20 , 2008
"... Abstract 1 Motivated in part by the hierarchical organization of the neocortex, a number of recently proposed algorithms have tried to learn hierarchical, or “deep, ” structure from unlabeled data. While several authors have formally or informally compared their algorithms to computations performed ..."
Abstract - Cited by 164 (19 self) - Add to MetaCart
interestingly, in a quantitative comparison, the encoding of these more complex “corner ” features matches well with the results from Ito & Komatsu’s study of neural responses to angular stimuli in area V2 of the macaque. This suggests that our sparse variant of deep belief networks holds promise

Neural Optimization

by Carsten Peterson, Bo Söderberg - The Handbook of Brain Research and Neural Networks. Bradford Books/The , 1998
"... Introduction Many combinatorial optimization problems require a more or less exhaustive search to achieve exact solutions, with the computational effort growing exponentially or worse with system size. Various kinds of heuristic methods are therefore often used to find reasonably good solutions. Th ..."
Abstract - Cited by 178 (7 self) - Add to MetaCart
. The artificial neural network (ANN) approach falls within this category. In contrast to most other methods, the ANN approach does not fully or partly explore the discrete state-space. Rather, it "feels" its way in a fuzzy manner through an interpolating, continuous space towards good solutions

Exploiting sparseness in deep neural networks for large vocabulary speech recognition

by Dong Yu, Frank Seide, Gang Li, Li Deng - in Proc. ICASSP , 2012
"... Recently, we developed context-dependent deep neural network (DNN) hidden Markov models for large vocabulary speech recognition. While reducing errors by 33 % compared to its discriminatively trained Gaussian-mixture counterpart on the switchboard benchmark task, DNN requires much more parameters. I ..."
Abstract - Cited by 15 (7 self) - Add to MetaCart
%, and computation to 14 % and 23%, respectively, on these two datasets. Index Terms — speech recognition, deep belief networks, deep neural networks, sparseness 1.

Conversational speech transcription using context-dependent deep neural networks

by Frank Seide, Gang Li, Dong Yu - in Proc. Interspeech ’11
"... We apply the recently proposed Context-Dependent Deep-Neural-Network HMMs, or CD-DNN-HMMs, to speech-to-text transcription. For single-pass speaker-independent recognition on the RT03S Fisher portion of phone-call transcription benchmark (Switchboard), the word-error rate is reduced from 27.4%, obta ..."
Abstract - Cited by 111 (20 self) - Add to MetaCart
We apply the recently proposed Context-Dependent Deep-Neural-Network HMMs, or CD-DNN-HMMs, to speech-to-text transcription. For single-pass speaker-independent recognition on the RT03S Fisher portion of phone-call transcription benchmark (Switchboard), the word-error rate is reduced from 27

Fast Exact Multiplication by the Hessian

by Barak A. Pearlmutter - Neural Computation , 1994
"... Just storing the Hessian H (the matrix of second derivatives d^2 E/dw_i dw_j of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly ca ..."
Abstract - Cited by 93 (5 self) - Add to MetaCart
Just storing the Hessian H (the matrix of second derivatives d^2 E/dw_i dw_j of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly

Image denoising and inpainting with deep neural networks

by Junyuan Xie, Linli Xu, Enhong Chen - In NIPS , 2012
"... We present a novel approach to low-level vision problems that combines sparse coding and deep networks pre-trained with denoising auto-encoder (DA). We propose an alternative training scheme that successfully adapts DA, originally designed for unsupervised feature learning, to the tasks of image den ..."
Abstract - Cited by 15 (1 self) - Add to MetaCart
We present a novel approach to low-level vision problems that combines sparse coding and deep networks pre-trained with denoising auto-encoder (DA). We propose an alternative training scheme that successfully adapts DA, originally designed for unsupervised feature learning, to the tasks of image
Results 1 - 10 of 417