## Practical Issues in Temporal Difference Learning (1992)

### Cached

### Download Links

- [www.research.ibm.com]
- [www.research.ibm.com]
- [www-anw.cs.umass.edu]
- [www.cs.ualberta.ca]
- [webdocs.cs.ualberta.ca]
- DBLP

### Other Repositories/Bibliography

Venue: | Machine Learning |

Citations: | 368 - 2 self |

### BibTeX

@INPROCEEDINGS{Tesauro92practicalissues,

author = {Gerald Tesauro},

title = {Practical Issues in Temporal Difference Learning},

booktitle = {Machine Learning},

year = {1992},

pages = {257--277}

}

### Years of Citing Articles

### OpenURL

### Abstract

This paper examines whether temporal difference methods for training connectionist networks, such as Suttons's TD(lambda) algorithm can be successfully applied to complex real-world problems. A number of important practical issues are identified and discussed from a general theoretical perspective. These practical issues are then examined in the context of a case study in which TD(lambda) is applied to learning the game of backgammon from the outcome of self-play. This is apparently the first application of this algorithm to a complex nontrivial task. It is found that, with zero knowledge built in, the network is able to learn from scratch to play the entire game at a fairly strong intermediate level of performance which is clearly better than conventional commercial programs and which in fact surpasses comparable networks trained on a massive human expert data set. This indicates that TD learning may work better in practice than one would expect based on current theory, and it suggests that further analysis of TD methods, as well as applications in other complex domains may be worth investigating.

### Citations

2723 |
Learning internal representations by error propagation
- Rumelhart, Hinton, et al.
- 1986
(Show Context)
Citation Context ...descent method for structural credit assignment. Many supervised learning procedures use gradient-descent methods to optimize network structures; for example, the back-propagation learning procedure (=-=Rumelhart, et al., 1986-=-) uses gradient-descent to optimize the weights in a feed-forward multilayer perceptron. Equation 1 provides a way to adapt such supervised learning procedures to solve temporal credit assignment prob... |

1227 | Learning to predict by the methods of temporal differences - Sutton - 1988 |

1225 | Multilayer feedforward networks are universal approximators. Neural Netw - Hornik, Stinchcombe, et al. - 1989 |

946 | On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab - Vapnik, Chervonenkis - 1971 |

624 |
Learnability and the vapnik-chervonenkis dimension
- Blumer, Ehrenfeucht, et al.
- 1989
(Show Context)
Citation Context ...ty to noise in the teacher signal. (In contrast, with perfect teacher information, we might expect the required number of training sequences to scale roughly linearly with the network's VC dimension (=-=Blumer, et al., 1989-=-)). Another potential problem is that the quality of solution found by gradient-descent learning relative to the globally optimal solution might get progressively worse with increasing network size. O... |

613 | Some studies in machine learning using the game of checkers - Samuel - 1959 |

561 | A stochastic approximation method - ROBBINS, MONRO - 1951 |

474 | Neuronlike adaptive elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983 |

308 | Learning Efficient Classification Procedures and their ,ipplication to Chess Endgame - Quinlan |

279 | Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems - Holland - 1986 |

246 | Temporal credit assignment in reinforcement learning - Sutton - 1984 |

157 | Perceptrons - Minsky, Papert - 1969 |

83 | Stationary and nonstationary learning characteristics of the LMS adaptive filter - Widrow, McCool, et al. - 1976 |

69 | Strategy learning with multilayer conneetionist representations - Anderson - 1987 |

67 | A parallel network that learns to play backgammon - Tesauro, Sejnowski - 1989 |

54 | Two Kinds of Training Information for Evaluation Function Learning - Utgoff, Clouse - 1991 |

44 | A Pattern Classification Approach to Evaluation Function Learning - Lee, Mahajan - 1988 |

38 | The convergence of TD(λ) for general λ - Dayan - 1992 |

34 | Neurogammon: a neural network backgammon playing program - Tesauro - 1990 |

30 | Connectionist Learning of Expert Preferences by Comparison Training - Tesauro - 1989 |

18 | A Unified Theory of Heuristic Evaluation Functions and its Application to Learning - Christensen, Korf - 1986 |

14 | On the construction of evaluation functions for large domains - Berliner - 1979 |

13 | R.J.Williams, “Learning internal representations by error propagation - Rumelhart - 1986 |

11 | Experiences in Evaluation with BKG -- A Program that Plays Backgammon - Berliner - 1977 |

7 | Using features to evaluate positions in expert's and novices' Othello games - Mitchell - 1984 |

6 | A Comparison and Evaluation of Three Machine Learning Procedures as Applied to the Game of Checkers - Griffith - 1974 |

5 | Algorithmic strategies for improving the performance of game-playing programs - Frey - 1986 |

4 | Haussler and M.Warmuth, “Learnability and the Vapnik-Chervo-Nenkis Dimention - Blumer, Ehrenfeucht, et al. - 1989 |

3 | On optimal doubling - Zadeh, Kobliska - 1977 |

2 |
Backgammon
- Magriel
- 1976
(Show Context)
Citation Context ...oard, and to form blocking configurations that impede the forward movement of opponent pieces. These facts lead to a number of subtle and complex strategies and tactics at the highest levels of play (=-=Magriel, 1976-=-). Backgammon offers a number of attractive features as a test vehicle for TD learning approaches. Due to the stochastic dice rolls, the evolution of board states during the course 39s264 G. TESAURO o... |

1 |
Experiences in evaluation with BKG--a program that plays backgammon
- Berliner
- 1977
(Show Context)
Citation Context ...st move. (An interesting exception to this rule is discussed in Berliner (1977).) Another advantage of studying this situation is that it can be solved essentially exactly by conventional algorithms (=-=Berliner, 1977-=-; Zadeh, 1977). For each of the approximately 54,000 possible bearoff configurations for a given side, one can recursively compute and store in a table the exact probability of removing all men on one... |

1 | The convergence of TD0x - Dayan - 1992 |