The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and ban@ass, comparable with the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images. Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete--i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input-output function will deviate from being purely linear. These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in

The images we typically view, or natural scenes, constitute a minuscule fraction of the space of all possible images. It seems reasonable that the visual cortex, which has evolved and developed to effectively cope with these images, has discovered efficient coding strategies for representing their structure. Here, we explore the hypothesis that the coding strategy employed at the earliest stage of the mammalian visual cortex maximizes the sparseness of the representation. We show that a learning algorithm that attempts to find linear sparse codes for natural scenes will develop receptive fields that are localized, oriented, and bandpass, much like those in the visual system. These receptive fields produce a more efficient image representation for later stages of processing because sparseness reduces the entropies of individual outputs, which in turn reduces the redundancy due to complex statistical dependencies among unit activities. The spatial receptive fields of simple ...

tation for later stages of processing because it possesses a higher degree of statistical independence among its outputs. We start with the basic assumption that an image, I(x; y), can be represented in terms of a linear superposition of (not necessarily orthogonal) basis functions, OE i (x; y): I(x; y) = X i a i OE i (x; y) : (1) The image code is determined by the choice of basis functions, OE i . The coefficients, a i , are dynamic variables that change from one image to the next. The goal of efficient coding is to find a set of OE i that forms a complete code (i.e., spans the image space) and results in the coefficient values being as statistically independent as possible over an ensemble of natural images. The