Theoretical results on reinforcement learning with temporally abstract options

Theoretical results on reinforcement learning with temporally abstract options (1998)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Doina Precup , Richard S. Sutton , Satinder Singh

Venue:	in: Proc. 10th European Conference on Machine Learning
Citations:	49 - 9 self

BibTeX

@INPROCEEDINGS{Precup98theoreticalresults,
    author = {Doina Precup and Richard S. Sutton and Satinder Singh},
    title = {Theoretical results on reinforcement learning with temporally abstract options},
    booktitle = {in: Proc. 10th European Conference on Machine Learning},
    year = {1998},
    publisher = {Springer}
}

Years of Citing Articles

Bookmark

OpenURL

Abstract

Abstract. We present new theoretical results on planning within the framework of temporally abstract reinforcement learning (Precup & Sutton, 1997; Sutton, 1995). Temporal abstraction is a key step in any decision making system that involves planning and prediction. In temporally abstract reinforcement learning, the agent is allowed to choose among ”options”, whole courses of action that may be temporally extended, stochastic, and contingent on previous events. Examples of options include closed-loop policies such as picking up an object, as well as primitive actions such as joint torques. Knowledge about the consequences of options is represented by special structures called multi-time models. In this paper we focus on the theory of planning with multi-time models. We define new Bellman equations that are satisfied for sets of multi-time models. As a consequence, multi-time models can be used interchangeably with models of primitive actions in a variety of well-known planning methods including value iteration, policy improvement and policy iteration. 1

Citations

2827	Reinforcement Learning: An Introduction - Sutton, Barto - 1998
878	Markov Decision Processes: Discrete Stochastic Dynamic Programming - Puterman - 1994
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
331	Dynamic Programming: Deterministic and Stochastic Models - Bertsekas - 1987
316	Automatic programming of behavior-based robots using reinforcement learning - Mahadevan, Connell - 1992
311	A structure for plans and behavior - Sacerdoti - 1977
307	Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition - Dietterich - 1999
275	Prioritized sweeping: Reinforcement learning with less data and less time - Moore, Atkeson - 1993
246	Chunking in Soar: The anatomy of a general learning mechanism - Laird, Rosenbloom, et al. - 1986
222	Motivated reinforcement learning - Dayan - 2002
212	Reinforcement learning with hierarchies of machines - Parr, Russell - 1998
157	Learning with delayed rewards - Watkins - 1989
94	Hierarchical learning in stochastic domains: Preliminary results - Kaelbling - 1993
85	Efficient learning and planning within the dyna framework - Peng, Williams - 1993
72	Iulti-time models for temporally abstract planning - Precup, Sutton - 1998
57	A feedback control structure for on-line learning tasks. Robotics and Autonomous Systems - Huber, Grupen - 1997
55	TD models: Modeling the world at a mixture of time scales - Sutton - 1995
54	Learning to solve problems by searching for macro-operators - Korf - 1985
38	Roles of macro-actions in accelerating reinforcement learning - McGovern, Sutton, et al. - 1997
34	The learning of world models by connectionist networks - Sutton, Pinette - 1985
23	Improving generalization for temporal difference learning: The successor representation - Dayan - 1993
14	Scaling reinforcement learning by learning variable temporal resolution models - Singh - 1992