## Sparse coding with an overcomplete basis set: A strategy employed by V1? (1997)

### Cached

### Download Links

Venue: | VISION RESEARCH |

Citations: | 660 - 10 self |

### BibTeX

@ARTICLE{Olshausen97sparsecoding,

author = {Bruno A. Olshausen and David J. Field},

title = {Sparse coding with an overcomplete basis set: A strategy employed by V1?},

journal = {VISION RESEARCH},

year = {1997},

volume = {37},

pages = {3311--3325}

}

### Years of Citing Articles

### OpenURL

### Abstract

The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and bandpass, comparable to the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images (Olshausen and Field, 1996a). Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete---i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input...

### Citations

2698 | A theory for multiresolution signal decomposition: the wavelet representation,” Pattern Analysis and
- Mallat
- 1989
(Show Context)
Citation Context ...ities is met (e.g., decorrelation, sparseness, etc.). This is the basis of coding strategies such as the Discrete Cosine Transform (used in JPEG image compression), or orthonormal wavelet transforms (=-=Mallat, 1989-=-). In terms of a physical implementation, such a coding scheme could be realized by a strictly feed-forward neural network, in which case the functions ~i(~) would correspond to the spatial "receptive... |

1460 | Independent component analysis, a new concept - Comon - 1994 |

1326 | Information Theory and Statistics
- Kullback
- 1959
(Show Context)
Citation Context ...een the two distributions: KL= P*(I)log dI. (9) This measures the average information gain, per image drawn from P*(/), for judging in favor of the image being drawn from P*(/) as opposed to P(/lq~) (=-=Kullback, 1959-=-). The greater the difference between the two distributions, the greater will be KL, with KL = 0 if and only if the two distributions are equal. Because P*(/) is fixed, minimizing KL amounts to maximi... |

1189 | An information-maximization approach to blind separation and blind deconvolution - Bell, Sejnowski - 1995 |

1010 | Emergence of simple-cell receptive field properties by learning a sparse code for natural images - Lewic, Olshausen, et al. - 1996 |

668 | Relations between the statistics of natural images and the response properties of cortical cells
- Field
- 1987
(Show Context)
Citation Context ...ge code that spans the joint space of spatial position, orientation, and scale in a manner similar to wavelet codes, which have previously been shown to form sparse representations of natural images (=-=Field, 1987-=-; Field,SPARSE CODING WITH AN OVERCOMPLETE BASIS SET: A STRATEGY EMPLOYED BY VI? 3319 I m II II II II II ~J /! FIGURE 7. The set of 144 basis functions learned by the sparse coding algorithm. The bas... |

539 | A new learning algorithm for blind signal separation”, NIPS 95 - Amari, Cichocki, et al. - 1996 |

467 | Shiftable multiscale transforms - Simoncelli, Freeman, et al. - 1992 |

410 |
What is the goal of sensory coding
- FIELD
- 1994
(Show Context)
Citation Context ...r the probability distribution of these events is that they are "sparse", meaning that a given image may usually be described in terms of a small number of basis functions chosen out of a larger set (=-=Field, 1994-=-), as illustrated in Fig. 1. It was shown previously that when such a code is sought for natural images, the basis functions that emerge are qualitatively similar in form to simple cell receptive fiel... |

357 |
Possible principles underlying the transformation of sensory messages
- Barlow
- 1961
(Show Context)
Citation Context ...to what extent theories of efficient coding can provide us with insights about cortical image representation. The notion of efficiency we adopt is based on Barlow's principle of redundancy reduction (=-=Barlow, 1961-=-, 1989), which states that a useful goal of sensory coding is to transform the input in such a manner that reduces the redundancy* due to complex statistical dependencies *A confusion that often arise... |

266 | Exploratory projection pursuit
- Friedman
- 1987
(Show Context)
Citation Context ...oublesome for gradient descent techniques searching for structure in this space. A standard technique to ameliorate these effects is to "sphere" the data by equalizing the variance in all directions (=-=Friedman, 1987-=-), as schematically illustrated in Fig. 6(a). Since the amplitude spectrum falls as roughly 1/f at all orientations in the 2D frequency plane, sphering may be accomplished by filtering with a circular... |

254 |
Could information theory provide an ecological theory of sensory processing
- Atick
- 1992
(Show Context)
Citation Context ...n explicit representation of the underlying independent entities that gave rise to the image, which would be useful to the survival of the organism. Atick and colleagues (Atick & Redlich, 1990, 1992; =-=Atick, 1992-=-; Dong & Atick, 1995; Dan, Atick, & Reid, 1996) have achieved considerable success in showing how the principle of redundancy reduction may be applied toward understanding the response properties of r... |

243 | Unsupervised learning - Barlow - 1989 |

202 | The Helmholtz machine
- Dayan, Hinton, et al.
- 1995
(Show Context)
Citation Context ...ge. Surely, there will be statistical dependencies among the elements of the single-stage model, and it would be desirable to have these modeled by a second and third stage in a hierarchical fashion (=-=Dayan et al., 1995-=-; Lewicki & Sejnowski, 1996). In order to do this though, nonlinearities such as those mentioned above will also need to be dealt with. Simply adding another linear image model on top of the current o... |

183 | Spatial frequency selectivity of cells in macaque visual cortex. Vision Res - DeValois, Albrecht, et al. - 1982 |

175 | What does the retina know about natural scenes - Atick, Redlich - 1992 |

152 | Forming sparse representations by local anti-Hebbian learning - Földiák - 1990 |

124 | Generative Models for Discovering Sparse Distributed Representations
- Hinton, Ghahramani
(Show Context)
Citation Context ... here is a sparse, distributed code, which actually occupies a middle ground between dense population codes at one end and local representations (i.e., grandmother cells) at the other (Foldiak, 1995; =-=Hinton & Ghahramani, 1997-=-). Note for example that the learned basis functions are broadly tuned to some stimulus dimensions (e.g., spatial frequency), as would be expected of a coarse code, while narrowly tuned to others (e.g... |

118 |
Towards a theory of early visual processing
- Atick, Redlich
- 1990
(Show Context)
Citation Context ...uch a strategy will recover an explicit representation of the underlying independent entities that gave rise to the image, which would be useful to the survival of the organism. Atick and colleagues (=-=Atick & Redlich, 1990-=-, 1992; Atick, 1992; Dong & Atick, 1995; Dan, Atick, & Reid, 1996) have achieved considerable success in showing how the principle of redundancy reduction may be applied toward understanding the respo... |

97 | E¤cient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory
- Dan, Atick, et al.
- 1996
(Show Context)
Citation Context ...underlying independent entities that gave rise to the image, which would be useful to the survival of the organism. Atick and colleagues (Atick & Redlich, 1990, 1992; Atick, 1992; Dong & Atick, 1995; =-=Dan, Atick, & Reid, 1996-=-) have achieved considerable success in showing how the principle of redundancy reduction may be applied toward understanding the response properties of retinal ganglion cells in terms of a strategy f... |

91 | Dynamic model of visual recognition predicts neural response properties in the visual cortex - Rao, Ballard - 1997 |

89 | Natural image statistics and efficient coding
- Olshausen, Field
- 1996
(Show Context)
Citation Context ... lines and edges, especially curved, fractal-like edges, give rise to statistical dependencies that are of higher-order than linear pairwise correlations (e.g., threepoint correlations) (Field, 1993; =-=Olshausen & Field, 1996-=-b), and so it is important to consider these forms of structure as well in developing an efficient code. Our goal here will be to find a linear coding strategy that is capable of reducing these higher... |

83 | Neuronal Architectures for Pattern-theoretic Problems - Mumford - 1995 |

77 |
Scale-invariance and self-similar `wavelet' transforms: an analysis of natural scenes and mammalian visual systems
- Field
- 1993
(Show Context)
Citation Context ...). y oriented lines and edges, especially curved, fractal-like edges, give rise to statistical dependencies that are of higher-order than linear pairwise correlations (e.g., threepoint correlations) (=-=Field, 1993-=-; Olshausen & Field, 1996b), and so it is important to consider these forms of structure as well in developing an efficient code. Our goal here will be to find a linear coding strategy that is capable... |

68 | Sejnowski TJ - AJ - 1995 |

68 | A multiple cause mixture model for unsupervised learning - Saund - 1995 |

63 | Relations between the statistics of natural images and the response properties of cortical cells. J Opt Soc Am A 4:2379–2394 - DJ - 1987 |

52 | Sparse coding in the primate cortex
- Foldiak, Young
- 1995
(Show Context)
Citation Context ...eparate from those that have been written about elsewhere, such as increasing capacity in associative memory (Baum, Moody, & Wilczek, 1988), minimizing wiring length and ease of forming associations (=-=Foldiak, 1995-=-), or metabolic efficiency (Baddeley, 1996). While these are obvious advantages of a sparse code, they are independent from the criteria we are considering here. If the data were actually composed fro... |

51 | Development of low entropy coding in a recurrent network - Harpur, Prager - 1996 |

50 |
Entropy reduction and decorrelation in visual coding by oriented neural receptive fields
- Daugman
- 1989
(Show Context)
Citation Context ...unctions are totally overlapping (i.e., the entire set codes for the same image patch). All have been normalized to fill the grey scale, but with zero always represented by the same grey level. 1994; =-=Daugman, 1989-=-). Shown in Fig. 8 is the distribution of the basis functions in spatial frequency and orientation. The vast majority lie within the high spatial-frequency bands, as expected of a wavelet code in orde... |

47 | Spatial frequency selectivity of cells in the macaque visual cortex. Vision Res 22: 545–549 - RL, DG, et al. - 1982 |

47 | Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus
- Dong, Atick
- 1995
(Show Context)
Citation Context ...presentation of the underlying independent entities that gave rise to the image, which would be useful to the survival of the organism. Atick and colleagues (Atick & Redlich, 1990, 1992; Atick, 1992; =-=Dong & Atick, 1995-=-; Dan, Atick, & Reid, 1996) have achieved considerable success in showing how the principle of redundancy reduction may be applied toward understanding the response properties of retinal ganglion cell... |

43 |
Feature extraction using an unsupervised neural network
- Intrator
- 1992
(Show Context)
Citation Context ... analog domain. Both of these algorithms formed the inspiration for the development of our algorithm. Another class of efficient coding methods is based on projection pursuit methods (Friedman, 1987; =-=Intrator, 1992-=-; Law & Cooper, 1994; Fyfe & Baddeley, 1995; Press & Lee, 1996; Lu, Chubb, & Sperling, 1996). Some of these were trained on natural images, but with the exception of Press & Lee (1996) and Lu et al. (... |

39 | Could information theory provide an ecological theory of sensory processing? Network: Comput Neural Syst 3:213–251 - JJ - 1992 |

35 |
Internal Representations for Associative Memory
- Baum, Moody, et al.
- 1988
(Show Context)
Citation Context ...ution such as depicted in Fig. l(b). Note that these reasons for desiring sparseness are separate from those that have been written about elsewhere, such as increasing capacity in associative memory (=-=Baum, Moody, & Wilczek, 1988-=-), minimizing wiring length and ease of forming associations (Foldiak, 1995), or metabolic efficiency (Baddeley, 1996). While these are obvious advantages of a sparse code, they are independent from t... |

29 | A theory for multiresolution signal decomposition: the wavelet representation - SG - 1989 |

28 | Redlich AN - JJ - 1992 |

26 |
Two-dimensional spatial structure of receptive ¢elds in monkey striate cortex
- Parker, Hawken
- 1988
(Show Context)
Citation Context ...o lowspatial-frequency range--i.e., around 4-8 cyc/deg in the parafoveal region, when the highest spatial-frequency band should be in the range of 16-32 cyc/deg (De Valois, Albrecht, & Thorell, 1982; =-=Parker & Hawken, 1988-=-). However, there is good reason to believe that the numberSPARSE CODING WITH AN OVERCOMPLETE BASIS SET: A STRATEGY EMPLOYED BY VI? 3321 r., m m m + + + u:z r~ ,.~ I ~=.~ ~i~il I~ "i~ o.) r., .,.., ~... |

25 | Low entropy coding with unsupervised neural networks - Harpur - 1997 |

25 | Learning linear, sparse, factorial codes
- Olshausen
- 1996
(Show Context)
Citation Context ...(1996), and Bell & Sejnowski (1995). The one most closely related to ours is that of Bell & Sejnowski (see also their article in this issue). The formal relationship described in Appendix I (see also =-=Olshausen, 1996-=-), shows that both algorithms are solving the same maximum-likelihood problem, but by making different simplifying assumptions. Bell & Sejnowski assume the weight matrix to be square and of full rank ... |

24 | Atick JJ - DW - 1995 |

20 | Bayesian unsupervised learning of higher order structure
- Lewicki, Sejnowski
- 1997
(Show Context)
Citation Context ...ll be statistical dependencies among the elements of the single-stage model, and it would be desirable to have these modeled by a second and third stage in a hierarchical fashion (Dayan et al., 1995; =-=Lewicki & Sejnowski, 1996-=-). In order to do this though, nonlinearities such as those mentioned above will also need to be dealt with. Simply adding another linear image model on top of the current one, using the same number o... |

18 | Finding compact and sparse distributed representations of visual images - Fyfe, Baddeley - 1995 |

14 |
An efficient code
- Baddeley
- 1996
(Show Context)
Citation Context ... about elsewhere, such as increasing capacity in associative memory (Baum, Moody, & Wilczek, 1988), minimizing wiring length and ease of forming associations (Foldiak, 1995), or metabolic efficiency (=-=Baddeley, 1996-=-). While these are obvious advantages of a sparse code, they are independent from the criteria we are considering here. If the data were actually composed from causes with multimodal distributions wit... |

13 |
The effect of threshold on the relationship between the receptive-field profile and the spatial-frequency tuning curve in simple cells of the cat’s striate cortex
- Tadmor, Tolhurst
- 1989
(Show Context)
Citation Context ...simple cell, although the physiological data show thatSPARSE CODING WITH AN OVERCOMPLETE BASIS SET: A STRATEGY EMPLOYED BY VI? 3313 there are interesting forms of non-linearity in these cells (e.g., =-=Tadmor & Tolhurst, 1989-=-) that are not captured by such a straightforward, linear model. An alternative way of coding images within a linear framework, which we explore in this paper, is in terms of a generative model, illus... |

11 | Unsupervised learning - HB - 1989 |

11 | Adelson EH & Heeger DJ - EP - 1991 |

11 |
Finding compact and sparse-distributed representations of visual scenes
- Fyfe, Baddeley
- 1995
(Show Context)
Citation Context ...thms formed the inspiration for the development of our algorithm. Another class of efficient coding methods is based on projection pursuit methods (Friedman, 1987; Intrator, 1992; Law & Cooper, 1994; =-=Fyfe & Baddeley, 1995-=-; Press & Lee, 1996; Lu, Chubb, & Sperling, 1996). Some of these were trained on natural images, but with the exception of Press & Lee (1996) and Lu et al. (1996), they did not show a full family of r... |

11 |
Formation of receptive fields in realistic visual environments according to the Bienenstock, Cooper and Munro (BCM) theory
- Law, Cooper
- 1994
(Show Context)
Citation Context ...Both of these algorithms formed the inspiration for the development of our algorithm. Another class of efficient coding methods is based on projection pursuit methods (Friedman, 1987; Intrator, 1992; =-=Law & Cooper, 1994-=-; Fyfe & Baddeley, 1995; Press & Lee, 1996; Lu, Chubb, & Sperling, 1996). Some of these were trained on natural images, but with the exception of Press & Lee (1996) and Lu et al. (1996), they did not ... |

10 | Two-dimensional spatial structure of receptive fields in monkey striate cortex. J Opt Soc Am A 5: 598–605 - AJ, MJ - 1988 |