Abstract

This paper presents a method by which a reinforcement learning agent can solve the incomplete perception problem using memory. The agent uses a hidden Markov model (HMM) to represent its internal state space and creates memory capacity by splitting states of the HMM. The key idea is a test to determine when and how a state should be split: the agent only splits a state when doing so will help the agent predict utility. Thus the agent can create only as much memory as needed to perform the task at hand---not as much as would be required to model all the perceivable world. I call the technique UDM, for Utile Distinction Memory.

Citations

8882	Computers and Intractability: A Guide to the Theory of NP-completeness - Garey, Johnson - 1979
3116	A tutorial on hidden Markov models and selected applications in speech recognition - Rabiner - 1989
1135	Learning from delayed rewards - Watkins - 1989
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
190	Learning to coordinate behaviors - Maes, Brooks - 1990
173	Reinforcement learning with perceptual aliasing: The perceptual distinctions approach - Chrisman - 1992
113	Efficient exploration in reinforcement learning - Thrun - 1992
60	Reinforcement Learning for the adaptive control of perception and action - Whitehead - 1992
49	Kaelbling. Learning in Embedded Systems - Pack - 1990
42	Reinforcement learning with hidden states - Lin, Mitchell - 1993
24	Learning from delayed reinforcement in a complex domain - Chapman, Kaelbling - 1990
20	Learning to perceive and act - Whitehead - 1991
16	First Results with Utile Distinction Memory for Reinforcement Learning - McCallum - 1992
12	Intelligent agent design issues: Internal agent state and incomplete perception - Chrisman, Caruana, et al. - 1991
12	Cost-sensitive reinforcement learning for adaptive classification and control - Tan - 1991
7	Learning with incomplete selective perception - McCallum - 1993