Abstract

One of the basic problems of machine learning is deciding how to act in an uncertain world. For example, if I want my robot to bring me a cup of coffee, it must be able to compute the correct sequence of electrical impulses to send to its motors to navigate from the coffee pot to my office. In fact, since the results of its actions are not completely predictable, it is not enough just to compute the correct sequence; instead the robot must sense and correct for deviations from its intended path. In order for any machine learner to act reasonably in an uncertain environment, it must solve problems like the above one quickly and reliably. Unfortunately, the world is often so complicated that it is difficult or impossible to find the optimal sequence of actions to achieve a given goal. So, in order to scale our learners up to real-world problems, we usually must settle for approximate solutions. One representation for a learner's environment and goals is a Markov decision process or MDP. ...

Citations

7321	Introduction to Algorithms - Cormen, Leiserson, et al. - 1997
2333	Convex Analysis - Rockafellar - 1970
1135	Learning from delayed rewards - Watkins - 1989
944	J: General Linear Models - McCullagh, Nelder - 1989
622	Flows in Networks - Ford, Fulkerson - 1962
556	The weighted majority algorithm - Littlestone, Warmuth - 1994
477	Tsitsiklis. Parallel and Distributed Computation: Numerical Methods - Bertsekas, N - 1989
404	Applied Nonlinear Control - Slotine, Li - 1991
342	Dynamic Programming and Optimal Control, Athena Scientific - Bertsekas - 2000
332	Iterative solution of nonlinear equations in several variables - Ortega, Rheinblodt - 1970
308	Adaptive Control Processes: a Guided Tour - Bellman - 1961
267	How to use expert advice - Cesa-Bianchi, Freund, et al. - 1997
262	Simple statistical gradient-following algorithms for connectionist reinforcement learning - Williams - 1992
251	On a routing problem - Bellman - 1958
250	Improving elevator performance using reinforcement learning - Crites, Barto - 1996
223	Generalization in reinforcement learning: safely approximating the value function - Boyan, Moore - 1995
211	Aggregating strategies - Vovk - 1990
207	Residual Algorithms: Reinforcement Learning with Function Approximation - Baird - 1995
203	The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces - Moore, Atkeson - 1995
196	Exponentiated gradient versus gradient descent for linear predictors - Kivinen, Warmuth - 1997
196	Introduction to Stochastic Dynamic Programming - Ross - 1983
190	Td-gammon, a self-teaching backgammon program, achieves master-level play - Tesauro - 1994
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
172	Stable function approximation in dynamic programming - Gordon - 1995
147	Learning to predict by the methods of temporal di erence - Sutton - 1988
139	Linear least-squares algorithms for temporal difference learning - Bradtke, Barto - 1996
133	Numerical analysis of spectral methods: Theory and Applications - Gottlieb, Orszag - 1977
130	LOQO: An interior point code for quadratic programming - Vanderbei - 1994
129	Asynchronous stochastic approximation and Q-learning - Tsitsiklis - 1994
120	Discounted Dynamic Programming - Blackwell - 1965
116	Gradient Descent for General Reinforcement Learning - Baird, Moore - 1999
112	Kendall’s Advanced Theory of Statistics - Stuart, Ord - 1994
102	Dynamic Programming and Stochastic Control - Bertsekas - 1976
97	Reinforcement learning with soft state aggregation - Singh, Jaakkola, et al. - 1995
58	Sequential prediction of individual sequences under general loss functions - Haussler, Kivinen, et al. - 1998
58	Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an application to pricing high-dimensional financial derivatives - Tsitsiklis, Roy - 1997
57	Issues in using function approximation for reinforcement learning - Thrun, Schwartz - 1993
55	Relative loss bounds for multidimensional regression problems - Kivinen, Warmuth - 2001
41	Using Local Trajectory Optimizers to Speed up Global Optimization - Atkeson - 1994
37	Conjugate priors for exponential families - Diaconis, Ylvisaker - 1979
35	Predicting a binary sequence almost as well as the optimal biased coin - Freund - 2003
32	Neurogammon: a neural network backgammon program - Tesauro - 1990
30	Polynomial approximation | a new computational technique in dynamic programming: allocation processes - Bellman, Kalaba, et al. - 1963
23	Spline approximations to value functions: A linear programming approach - Trick, Zin - 1997
22	Exponentially many local minima for single neurons - Auer, Herbster, et al. - 1996
22	Functional approximations and dynamic programming - Bellman, Dreyfus - 1959
21	Barycentric interpolators for continuous space and time reinforcement learning. Neural Information Processing Systems - Munos, Moore - 1998
21	Feature-based methods for large-scale dynamic programming - Tsitsiklis, Roy - 1994
18	Regret bounds for prediction problems - Gordon - 1999
18	An analysis of temporal-di erence learning with function approximation - Tsitsiklis, Roy - 1996