Hidden State and Reinforcement Learning with Instance-Based State Identification

Hidden State and Reinforcement Learning with Instance-Based State Identification (0)

Cached

Download Links

[ftp.cs.rochester.edu]

by R. Andrew Mccallum

Venue:	IEEE Transations on Systems, Man, and Cybernetics
Citations:	31 - 1 self

BibTeX

@ARTICLE{Mccallum_hiddenstate,
    author = {R. Andrew Mccallum},
    title = {Hidden State and Reinforcement Learning with Instance-Based State Identification},
    journal = {IEEE Transations on Systems, Man, and Cybernetics},
    year = {},
    volume = {26},
    pages = {464--473}
}

Bookmark

OpenURL

Abstract

Real robots with real sensors are not omniscient. When a robot's next course of action depends on information that is hidden from the sensors because of problems such as occlusion, restricted range, bounded field of view and limited attention, we say the robot suffers from the hidden state problem. State identification techniques use history information to uncover hidden state. Some previous approaches to encoding history include: finite state machines [12, 28], recurrent neural networks [25] and genetic programming with indexed memory [49]. A chief disadvantage of all these techniques is their long training time. This paper presents instance-based state identification, a new approach to reinforcement learning with state identification that learns with much fewer training steps. Noting that learning with history and learning in continuous spaces both share the property that they begin without knowing the granularity of the state space, the approach applies instance-based (or "memory-ba...

Citations

6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
3341	Pattern Classification and Scene Analysis - Duda, Hart - 1973
3116	A tutorial on hidden Markov models and selected applications in speech recognition - Rabiner - 1989
2458	Genetic Programming: On the Programming of Computers by Means of Natural Selection - Koza - 1992
2289	Adaptation in Natural and Artificial Systems - Holland - 1975
1135	Learning from delayed rewards - Watkins - 1989
467	Animate vision - Ballard - 1991
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
243	Acting optimally in partially observable stochastic domains - Cassandra, Kaelbling, et al. - 1994
210	Outline for a theory of intelligence - Albus - 1991
203	The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces - Moore, Atkeson - 1995
185	Learning and sequential decision making - Barto, Sutton - 1990
181	Classifier systems and genetic algorithms - Booker, Goldberg, et al. - 1989
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
173	Reinforcement learning with perceptual aliasing: The perceptual distinctions approach - Chrisman - 1992
157	Neuronlike elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983
137	Reinforcement learning algorithm for partially observable Markov decision problems - Jaakkola, Singh, et al. - 1994
125	A distributed model for mobile robot environment-learning and navigation - Mataric - 1990
124	Hidden Markov Model Induction by Bayesian Model Merging - Stolcke, Omohundro - 1993
114	aJld �l. Colombetti. Robot shaping: Developing autonomous agents through learning - Dorigo - 1994
105	Realtime learning and control using asynchronous dynamic programmming - Baro, Bradtke, et al. - 1991
102	Dynamic Programming and Stochastic Control - Bertsekas - 1976
96	Overcoming incomplete perception with utile distinction memory - McCallum - 1993
94	Efficient Memory-based Learning for Robot Control. Unpublished - Moore - 1990
91	The Evolution of Mental Models - Teller - 1993
88	Memoryless policies: Theoretical limitations and practical results. In From Animals to Animats 3 - Littman - 1994
84	Instance-based utile distinctions for reinforcement learning with hidden state - McCallum - 1995
80	Learning to perceive and act by trial and error - Whitehead, Ballard - 1991
77	Robot juggling: an implementation of memory-based learning. Control systems magazine, pages 57–71. constraints and intrinsic motivation 65 - Schaal, Atkeson - 1994
76	Variable resolution dynamic programming: Efficiently learning action maps in multivariate real-valued state-spaces - Moore - 1991
63	Diversity-based inference of finite automata - Schapire - 1988
60	Reinforcement Learning for the adaptive control of perception and action - Whitehead - 1992
43	Memory-based approaches to approximating continuous functions - Atkeson - 1992
42	Training agents to perform sequential behavior - Colombetti, Dorigo - 1994
42	Reinforcement learning with hidden states - Lin, Mitchell - 1993
41	ALECSYS and the AutonoMouse: Learning to control a real robot by distributed classifier systems - Dorigo - 1992
38	Learning probabilistic automata with variable memory length - Ron, Singer, et al. - 1994
36	Inferring finite automata with stochastic output functions and an application to map learning - Dean, Angluin, et al. - 1992
30	Discovering the structure of a reactive environment by exploration - Mozer, Bachrach - 1989
30	Selective Perception for Robot Driving - Reece - 1992
29	A new approach to unsupervised learning in deterministic environments - Rivest - 1987
24	Learning from delayed reinforcement in a complex domain - Chapman, Kaelbling - 1990
24	Model-free reinforcement learning for non-Markovian decision problems - Singh, Jaakkola, et al. - 1994
21	Self-improvement based on reinforcement learning, planning and teaching - Lin - 1991
18	Theories of learning - Hilgard - 1956
16	First Results with Utile Distinction Memory for Reinforcement Learning - McCallum - 1992
10	Stochastic Optimal Control - Bertsekas, Shreve - 1978
10	Locally Adaptive Nearest Neighbor Algorithms - Wettschereck, Dietterich - 1994
8	Using transitional proximity for faster reinforcement learning - McCallum - 1992
8	High dimension action spaces in robot skill learning - Schneider - 1994