Abstract

: In the last few years, many researchers have begun to study how to introduce hierarchy into reinforcement learning methods. Generally, this work has pushed the envelope in one of several important directions, but it typically has avoided the question of how to learn systems of hierarchical behavior from experience. We present a preliminary system for learning simple hierarchical systems using a constrained language for behavior specification. The system learns "do-until" macros by choosing a subgoal from and using its past experience with respect to this subgoal to build a constrained macro behavior that achieves it. The system can either be run offline and used for new but similar problems, or can be used during learning. We present an algorithm that combines macro learning with asynchronous value iteration methods such as prioritized sweeping. Macros learned by the system for a simple environment are discussed. Additionally, we present the related literature and sketch out the fut...

Citations