Abstract—This paper considers the role of shaping in applications of reinforcement learning, and proposes a formulation of shaping as a homotopy-continuation method. By considering reinforcement learning tasks as elements in an abstracted task space, we conceptualize shaping as a trajectory in task space, leading from simple tasks to harder ones. The solution of earlier, simpler tasks serves to initialize and facilitate the solution of later, harder tasks. We list the different ways reinforcement learning tasks may be modified, and review cases where continuation methods were employed (most of which were originally presented outside the context of shaping). We contrast our proposed view with previous work on computational shaping, and argue against the often-held view that equates shaping with a rich reward scheme. We conclude by discussing a proposed research agenda for the computational study of shaping in the context of reinforcement learning. I.

I would like to thank my thesis advisor, Sridhar Mahadevan. Sridhar has been such a wonderful advisor, and every aspect of this thesis has benefitted from his guidance and support throughout my graduate studies. I also like to thank Sridhar for giving me the flexibility to explore many different ideas and research topics. I am appreciative of the support offered by my other thesis committee members, Andrew McCallum, Erik Learned-Miller, and Weibo Gong. Andrew helped me on CRF, MALLET and topic modeling. Erik helped me on computer vision. Weibo has many brilliant ideas on how brains work. I got a lot of inspirations from him. I am grateful for many other professors and staff members, who helped me along. Andy Barto offered me many insightful comments and advice on my research. David Kulp and Oliver Brock helped me on bioinformatics. Stephen Scott brought me to this country, taught me machine learning/bioinformatics and offers me constant support. Vadim Gladyshev helped me on biochemistry. Mauro Maggioni helped me on diffusion wavelets. I also thank Gwyn Mitchell and Leanne Leclerc for their help with my questions over the years. I am deeply thankful to my Master thesis advisor, Zhuzhi Yuan and other teachers in Nankai University for guiding my development as