## Information Processing by a Perceptron in an Unsupervised Learning Task (1993)

Citations: | 15 - 8 self |

### BibTeX

@MISC{Nadal93informationprocessing,

author = {Jean-pierre Nadal and Nestor Parga},

title = {Information Processing by a Perceptron in an Unsupervised Learning Task},

year = {1993}

}

### OpenURL

### Abstract

We study the ability of a simple neural network (a perceptron architecture, no hidden units, binary outputs) to process information in the context of an unsupervised learning task. The network is asked to provide the best possible neural representation of a given input distribution, according to some criterion taken from Information Theory. We compare various optimization criteria that have been proposed : maximum information transmission, minimum redundancy and closeness to factorial code. We show that for the perceptron one can compute the maximal information that the code (the output neural representation) can convey about the input. We show that one can use Statistical Mechanics techniques, such as the replica techniques, to compute the typical mutual information between input and output distributions. More precisely, for a Gaussian input source with a given correlation matrix, we compute the typical mutual information when the couplings are chosen randomly. We determine the correl...