Abstract

Abstract. This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units. These algorithms, called REINFORCE algorithms, are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reinforcement tasks, and they do this without explicitly computing gradient estimates or even storing information from which such estimates could be computed. Specific examples of such algorithms are presented, some of which bear a close relationship to certain existing algorithms while others are novel but potentially interesting in their own right. Also given are results that show how such algorithms can be naturally integrated with backpropagation. We close with a brief discussion of a number of additional issues surrounding the use of such algorithms, including what is known about their limiting behaviors as well as further considerations that might be used to help develop similar but potentially more powerful reinforcement learning algorithms.

Citations

2304	Learning Internal Representations by Error Propagation - Rumelhart, Hinton, et al. - 1986
1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
503	Principles of Artificial Intelligence - Nilsson - 1982
439	Beyond Regression: New Tools for Prediction and Analysis - Werbos
247	Forward models: Supervised learning with a distal teacher - Jordan, Rumelhart - 1992
245	Learning and relearning in Boltzmann machines - Hinton, Sejnowski - 1986
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
185	Learning and sequential decision making - Barto, Sutton - 1990
178	Adaptive Filtering Prediction and Control - Goodwin, Sin - 1985
157	Neuronlike elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983
143	Learning Automata: An Introduction - Narendra, Thathachar - 1989
62	A stochastic reinforcement learning algorithm for learning real-valued functions - Gullapalli - 1990
61	Pattern recognizing stochastic learning automata - Barto, Anandan - 1985
57	An Introduction to Probability Theory and Mathematical Statistics - Rohatgi - 1976
46	Learning by statistical cooperation of self-interested neuronlike computing elements." Humn Neurobiology - Barto - 1985
42	Une procedure d’âpprentissage pour reseau a seuil asymmetrique - LeCun - 1985
41	Associative search network: A reinforcement learning associative memory - Barto, Sutton, et al. - 1981
39	Toward a theory of reinforcement-learning connectionist systems - Williams - 1988
36	Gradient following without back-propagation in layered networks - Barto, Jordan - 1987
27	A dual back-propagation scheme for scalar reward learning - Munro - 1987
25	On the use of backpropagation in associative reinforcement learning - Williams - 1988
22	Reinforcement learning in connectionist networks: A mathematical analysis - Williams - 1986
22	A class of gradient-estimating algorithms for reinforcement learning in neural networks - Williams - 1987
18	Reinforcement-learning connectionist systems - Williams - 1987
17	Learning-logic - Parker - 1985
9	Structural learning in connectionist systems - Barto, Andersen - 1985
9	Reinforcement comparison - Dayan - 1990
7	Learning to generate focus trajectories for attentive vision - Schmidhuber, Huber - 1990
5	Decentralized learning in finite markov chains - Narendra - 1986
3	Learning Automata: An Introduction - Thathatchar - 1989
2	Learning-logic (Technical Report TR-47 - Parker - 1985
1	An N-player sequential stochastic game with identical payoffs - Wheeler - 1983
1	A new approach to the design of reinforcement schemes for learning automata - Thathatchar - 1985
1	1 tz Y - t~ O/x 0 2 y - ~ = ae - 01ng - 1985