Parameter Optimization Algorithm with Improved Convergence Properties for Adaptive Learning

Parameter Optimization Algorithm with Improved Convergence Properties for Adaptive Learning

Cached

Download Links

[www.dcs.bbk.ac.uk]

by G. D. Magoulas , M. N. Vrahatis

BibTeX

@MISC{Magoulas_parameteroptimization,
    author = {G. D. Magoulas and M. N. Vrahatis},
    title = {Parameter Optimization Algorithm with Improved Convergence Properties for Adaptive Learning},
    year = {}
}

Bookmark

OpenURL

Abstract

Abstract: The error in an artificial neural network is a function of adaptive parameters (weights and biases) that needs to be minimized. Research on adaptive learning usually focuses on gradient algorithms that employ problem–dependent heuristic learning parameters. This fact usually results in a trade–off between the convergence speed and the stability of the learning algorithm. The paper investigates gradient–based adaptive algorithms and discusses their limitations. It then describes a new algorithm that does not need user–defined learning parameters. The convergence properties of this method are discussed from both theoretical and practical perspective. The algorithm has been implemented and tested on real life applications exhibiting improved stability and high performance. Keywords: Feedforward neural networks, Supervised training, Back-propagation algorithm, Heuristic learning parameters, Non-linear Jacobi process, Globally convergent algorithms,

Citations

2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
1128	Practical Optimization - Gill, Murray, et al. - 1981
763	Numerical Methods for Unconstrained Optimization and Nonlinear Equations - Dennis - 1983
726	Textural features for image classification - Haralick, Shanmugam, et al. - 1973
505	A direct adaptive method for faster back-propagation learning: the RPROP algorithm - Riedmiller, Braun - 1993
410	Iterative Solution of Nonlinear Equations - Ortega, Rheinboldt - 1970
381	Iterative Solution Methods - Axelsson - 1994
283	Increased rates of convergence through learning rate adaptation - Jacobs
266	Iterative Methods for Linear and Nonlinear Equations - Kelley - 1995
239	A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks - Moller - 1993
235	Faster-learning variations on back-propagation: An empirical study - Fahlman - 1988
126	Minimization of functions having Lipschitz continuous first partial derivatives - Armijo - 1966
108	Second-Order Methods for Learning: Between Steepest Descent and Newton's Method - Battiti, First- - 1992
100	A First Course in Numerical Analysis - Ralston, Rabinowitz - 1978
92	Dubes, “performance evaluation for four classes of texture features - Ohanian, C - 1992
83	Improving the convergence of back-propagation learning with second order methods - Becker, LeCun - 1989
75	Back-propagation algorithm which varies the number of hidden units - HIROSE, YAMSHITA, et al. - 1991
74	Advanced supervised learning in multi-layer perceptrons – From backpropagation to adaptive learning algorithms - Riedmiller - 1994
67	Theory of algorithms for unconstrained optimization - Nocedal - 1991
54	Quasi-Newton methods, motivation and theory - Dennis, More - 1977
51	Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method - Johansson, Dowla, et al. - 1992
36	Accelerated backpropagation learning: Two optimization methods - Battiti - 1989
34	Acceleration techniques for the backpropagation algorithm - Silva, Almeida - 1990
27	Optimal algorithms for adaptive networks: second order back propagation, second order direct propagation, and second order Hebbian learning - Parker - 1987
24	Fahlman: Learning with Limited Numerical Precision Using the Cascade-Correlation Algorithm - Hoehfeld, E - 1992
24	Convergence of Learning Algorithms with Constant Learning Rates - Hornik, Kuan - 1990
20	An adaptive training algorithm for back propagation networks”, Computer Speech and Language - Chan, Fallside - 1987
14	An acoustic-phonetic data base - Fisher, Zue, et al. - 1987
11	On steepest descent - Goldstein - 1965
11	An analysis of premature saturation - Lee, Oh, et al. - 1993
11	S.: Effective back^propagation with variable stepsize. Neural Networks 10 - Magoulas, Vrahatis, et al. - 1997
11	An accelerating learning algorithm for multilayer perceptron networks - Parlos, Fernandez, et al. - 1994
11	Speeding-up backpropagation — A comparison of orthogonal techniques - Pfister, Rojas - 1993
10	1997b] “Neural network supervised training based on a dimension reducing method - Magoulas, Vrahatis, et al.
10	Back-propagation heuristics: a study of the extended delta-bar-delta algorithm - Minai, Williams - 1990
8	1962] “Cauchy’s method of minimization - Goldstein
8	Sclabassi R.J.: An adaptive training algorithm for back– propagation neural networks - Hsin, Li, et al. - 1995
7	Kokkinakis “High Performance Text Independent Speaker Recognition System based - Sirigos, Fakotakis, et al.
7	Complete Solution of the Local Minima - Lisboa, Perentonis - 1991
7	S.: A new method in neural network supervised training with imprecision - Magoulas, Vrahatis, et al. - 1996
6	Gradient descent: Second-order momentum and saturating error - Pearlmutter
5	Vrahatis, “Geometry of learning: Visualizing the performance of neural network supervised training methods - Androulakis, Magoulas, et al. - 1997
5	1995] “A convergence analysis for neural networks with constant learning rates and non–stationary inputs - Liu, Dong, et al.
3	Generalized perceptron networks with nonlinear discriminant functions - Kung, Diamantaras, et al. - 1991
3	1974] Methods for Solving Systems of Nonlinear Equations - Reinboldt
1	Textures - a photographic album for artists abd designers - Brodatz - 1966