Abstract

Most connectionist or "neural network" learning systems use some form of the back-propagation algorithm. However, back-propagation learning is too slow for many applications, and it scales up poorly as tasks become larger and more complex. The factors governing learning speed are poorly understood. I have begun a systematic, empirical study of learning speed in backprop-like algorithms, measured against a variety of benchmark problems. The goal is twofold: to develop faster learning algorithms and to contribute to the development of a methodology that will be of value in future studies of this kind. This paper is a progress report describing the results obtained during the first six months of this study. To date I have looked only at a limited set of benchmark problems, but the results on these are encouraging: I have developed a new learning algorithm that is faster than standard backprop by an order of magnitude or more and that appears to scale up very well as the problem size increases.

Citations

2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
439	Beyond Regression: New Tools for Prediction and Analysis - Werbos
283	Increased rates of convergence through learning rate adaptation - Jacobs
73	Experiments on Learning by BackPropagation - Plaut, Nowlan, et al. - 1986
50	Learning algorithms for connectionist networks: Applied gradient methods for nonlinear optimization - Watrous - 1987
42	Une procedure d’âpprentissage pour reseau a seuil asymmetrique - LeCun - 1985
36	Neural Network Simulation at Warp Speed: How We Got 17 - Pomerleau, Gusciora, et al. - 1988
36	Scaling relationships in back-propagation learning: Dependence on training set size - Tesauro - 1987
27	Optimal algorithms for adaptive networks: second order back propagation, second order direct propagation, and second order Hebbian learning - Parker - 1987
11	Successfully Using Peak Learning Rates of 10 (and Greater) in Back-Propagation Networks with the Heuristic Learning Algorithm - Cater - 1987
11	Connectionist architectures for artificial intelligence - Fahlman, Hinton - 1987
11	An improved three-layer, back propagation algorithm - Stornetta - 1987
10	Speech recognition with back propagation - Franzini - 1987
7	Accelerated learning in back-propagation nets - Schmidhuber - 1989
6	The Feasibility of Applying Numerical Optimization Techniques to Back-Propagation - Becker, LeCun - 1988