Abstract

This paper presents a reinforcement learning framework for continuoustime dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for infinitehorizon, discounted reward problems, we derive algorithms for estimating value functions and for improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived and their correspondences with the conventional residual gradient, TD(0), and TD() algorithms are shown. For policy improvement, two methods, namely, a continuous actor-critic method and a value-gradient based greedy policy, are formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived....

Citations

1135	Learning from delayed rewards - Watkins - 1989
1134	Reinforcement learning: A survey - Kaelbling, Littman, et al. - 1996
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
649	Reinforcement learning - Sutton, Barto - 1998
490	Neurons with graded response have collective computational properties like those of two-state neurons - Hopfield
423	Neuron-like adaptive elements that can solve difficult learning problems - Barto, Sutton, et al. - 1983
300	Generalization in reinforcement learning: Successful examples using sparse coarse coding - Sutton - 1996
250	Improving elevator performance using reinforcement learning - Crites, Barto - 1996
248	S.: Robot learning from demonstration - Atkeson, Schaal - 1997
207	Residual Algorithms: Reinforcement Learning with Function Approximation - Baird - 1995
203	The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces - Moore, Atkeson - 1995
190	Td-gammon, a self-teaching backgammon program, achieves master-level play - Tesauro - 1994
184	An analysis of temporal-difference learning with function approximation - Tsitsiklis, Roy - 1997
172	Stable function approximation in dynamic programming - Gordon - 1995
151	Reward functions for accelerated learning - Mataric - 1994
111	Reinforcement learning for dynamic channel allocation in cellular telephone systems - Singh, Bertsekas - 1997
99	Reinforcement learning methods for continuous-time Markov decision problems - Bradtke, Duff - 1995
97	Reinforcement learning with soft state aggregation - Singh, Jaakkola, et al. - 1995
62	A stochastic reinforcement learning algorithm for learning real-valued functions - Gullapalli - 1990
55	TD models: Modeling the world at a mixture of time scales - Sutton - 1995
52	Reinforcement learning applied to linear quadratic regulation - Bradtke - 1993
50	A menu of designs for reinforcement learning over time - Werbos - 1990
41	Using Local Trajectory Optimizers to Speed up Global Optimization - Atkeson - 1994
40	Advantage updating - Baird - 1993
24	Temporal difference learning in continuous time and space - Doya - 1996
23	Action-Based Sensor Space Categorization for Robot Learning - Asada, Noda, et al. - 1996
19	Stable fitted reinforcement learning - Gordon - 1995
12	Adaptive choice of grid and time in reinforcement learning. Neural Information Processing Systems - Pareigis - 1997
10	Reinforcement learning of dynamic motor sequence: Learning to stand up - Morimoto, Doya - 1998
9	Efficient nonlinear control with actor-tutor architecture - Doya - 1997
9	Reinforcement learning applied to a differential game - Harmon, Baird, et al. - 1996
9	A convergent reinforcement learning algorithm in the continuous case based on a finite difference method - Munos - 1997
7	Reinforcement learning for continuous stochastic control problems - Munos, Bourgine - 1997
4	Improving policies without measuring merits - Dayan, Singh - 1996
1	On-line estimation of the optimal value function: HJB-estimators - Peterson - 1993