Abstract

A new approach to unsupervised learning in a single-layer linear feedforward neural network is discussed. An optimality principle is proposed which is based upon preserving maximal information in the output units. An algorithm for unsupervised learning based upon a Hebbian learning rule, which achieves the desired optimality is presented, The algorithm finds the eigenvectors of the input correlation matrix, and it is proven to converge with probability one. An implementation which can train neural networks using only local "synaptic" modification rules is described. It is shown that the algorithm is closely related to algorithms in statistics (Factor Analysis and Principal Components Analysis) and neural networks (Self-supervised Backpropagation, or the "encoder" problem). It thus provides an explanation of certain neural network behavior in terms of classical statistical techniques. Examples of the use of a linear network for solving image coding and texture segmentation problems are presented. Also, it is shown that the algorithm can be used to find "visual receptive fields" which are qualitatively similar to those found in primate retina and visual cortex.

Citations

721	Organization of Behavior - Hebb - 1949
583	Learning representations by back-propagating errors - Rumelhart, Hinton, et al. - 1986
399	An introduction to computing with neural nets - Lippmann - 1987
336	Simplified neuron model as a principal component analyzer - Oja - 1982
312	Complete discrete 2D gabor transforms by neural networks for image analysis and compression - DAUGMAN - 1988
169	From basic network principles to neural architecture - Linsker - 1986
154	The art of adaptive pattern recognition by a self-organizing neural network - GA, Grossberg - 1988
152	Neural networks and principal component analysis: Learning from examples without local minima - Baldi, Hornik - 1989
92	Texture discrimination by Gabor functions - Turner - 1986
88	The “neural” phonetic typewriter - Kohonen - 1988
82	On stochastic approximation of the eigenvectors and the eigenvalues of the expectation of a random matrix - Oja, Karhunen - 1985
77	Phase relationships between adjacent simple cells in the visual cortex - Pollen, Ronner - 1981
73	F.: Visual Cortical Neurons as Localized Spatial Frequency Filters - Pollen, Ronner - 1983
57	Early vision and texture perception - Bergen, Adelson - 1988
29	An optimality principle for unsupervised learning - Sanger - 1989
26	Computing texture boundaries from images - Voorhees, Poggio - 1988
23	Relationship between spatial frequency selectivity and receptive field profile of simple cells - Andrews, Pollen - 1979
20	On the development of feature detectors in the visual cortex with application to learning and reaction-diffusion systems - Grossberg - 1976
14	Spatial frequency analysis in the visual system. Annu Rev Neurosci 8:547–583 - Shapley, Lennie - 1985
12	Modular learning in neural networks - Ballard - 1987
12	Finding texture boundaries in images - Voorhees - 1987
7	Spontaneous symmetry-breaking energy functions and the emergence of orientation selective cortical cells - Kammen, Yuille - 1988
7	Adaptive data orthogonalization - Owsley - 1978
6	New methods for stochastic approximation of truncated karhunen-loeve expansions - Karhunen, Oja - 1982
3	Receptive fields, binocular interactionsandfunctionalarchitectureincat’svisualcortex - Hubel, Wiesel - 1962
2	A theory for the development of neuron selectivity : Orientation selectivity and binocular inter - actions in visual cortex - unknown authors - 1982
2	Introduction to Computers and - H, Zmud - 1988
1	Learning receptive helds - Barrow - 1987
1	Auto-association by multilayet perceptrons and singular value decomposition - H, Kamp - 1988
1	On the problem of function approxmlauon bv sample set processing for an incompletely determined model, Annals" of the New York Academy of,Science - Brailovsky - 1983
1	On the problem of function system selection for function approximation based on the use of a sample set with defects, Annals of the New York Academy of Science - Brailovsky - 1983
1	On an incompletely determined model for function approximation by experimental data Annat~ of the New York Academy of Science - Brailovsky - 1985
1	Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems. Unpublished manuscript - W - 1989
1	Learning internal representations from gray-scale images: An example of extensional programming - D - 1987
1	Introduction of statistical pattern reco~mlion - Fukunaga - 1972
1	Optimal Unsupervised Learning 473 - Hinton - 1987
1	Adaptive algorithms for estimating eigenvectors of correlation type matrices - Karhuncn - 1984
1	Recursive estimation of eigenvectors q/'correlation type matrices for signal processing applications - Karhuncn - 1984
1	Simple gradient o'pe algorithms J`or dataadaptive eigenvector estimation - thesis - 1985
1	Optimal constrained representation and filtering of signals - Kazakos - 1983
1	Self-organized formation of topologically correct teaturc maps - Kohonen - 1982
1	Advanced engineering mathematics - Krcyszig - 1988
1	Stochastic approximatiop~ methods ['or constrained and unconstrained systems - unknown authors - 1978
1	Self-organization in a perceptual network - unknown authors - 1988
1	Anah,'sis of rccursive stochastic algorithms - unknown authors - 1977
1	Suh~'paee methods qf pattern recognition - Oja - 1983
1	Recursive construction of Karhuncn-Lo,Sve expansions for pattern recognition purposes - Oia, Karhunen - 1980
1	Optimal unsupervised learning - Sanger - 1988
1	Time-sequential selforganization of hierarchical neural net~,,orks - Silverman, Noetzel - 1988
1	Some remarks on the hackpropagation algorithm ]or neural net learning (Tech - D - 1988