Abstract

We study and compare different neural network learning strategies: batch-mode learning, on-line learning, cyclic learning, and almost cyclic learning. Incremental learning strategies require less storage capacity than batch-mode learning. However, due to the arbitrariness in the presentation order of the training patterns, incremental learning is a stochastic process, whereas batch-mode learning is deterministic. In zeroth order, i.e., as the learning parameter j tends to zero, all learning strategies approximate the same ordinary differential equation, for convenience referred to as the "ideal behavior". Using stochastic methods valid for small learning parameters j, we derive differential equation describing the evolution of the lowest order deviations from this ideal behavior. We compute how the asymptotic misadjustment, measuring the average asymptotic distance from a stable fixed point of the ideal behavior, scales as a function of the learning parameter and the number of trainin...

Citations