## The rat as particle filter

Citations: | 19 - 2 self |

### BibTeX

@MISC{Daw_therat,

author = {Nathaniel D. Daw and Aaron C. Courville and Université De Montréal},

title = {The rat as particle filter},

year = {}

}

### Years of Citing Articles

### OpenURL

### Abstract

The core tenet of Bayesian modeling is that subjects represent beliefs as distributions over possible hypotheses. Such models have fruitfully been applied to the study of learning in the context of animal conditioning experiments (and analogously designed human learning tasks), where they explain phenomena such as retrospective revaluation that seem to demonstrate that subjects entertain multiple hypotheses simultaneously. However, a recent quantitative analysis of individual subject records by Gallistel and colleagues cast doubt on a very broad family of conditioning models by showing that all of the key features the models capture about even simple learning curves are artifacts of averaging over subjects. Rather than smooth learning curves (which Bayesian models interpret as revealing the gradual tradeoff from prior to posterior as data accumulate), subjects acquire suddenly, and their predictions continue to fluctuate abruptly. These data demand revisiting the model of the individual versus the ensemble, and also raise the worry that more sophisticated behaviors thought to support Bayesian models might also emerge artifactually from averaging over the simpler behavior of individuals. We suggest that the suddenness of changes in subjects ’ beliefs (as expressed in conditioned behavior) can be modeled by assuming they are conducting inference using sequential Monte Carlo sampling with a small number of samples — one, in our simulations. Ensemble behavior resembles exact Bayesian models since, as in particle filters, it averages over many samples. Further, the model is capable of exhibiting sophisticated behaviors like retrospective revaluation at the ensemble level, even given minimally sophisticated individuals that do not track uncertainty from trial to trial. These results point to the need for more sophisticated experimental analysis to test Bayesian models, and refocus theorizing on the individual, while at the same time clarifying why the ensemble may be of interest. 1

### Citations

2129 |
A new approach to linear filtering and prediction problems
- Kalman
- 1960
(Show Context)
Citation Context ...ian diffusion, P (wt+1 | wt) = N (wt, σ 2 dI) (1) then we complete the well known generative model for which Bayesian inference about the weights can be accomplished using the Kalman filter algorithm =-=[11]-=-. Given a Gaussian prior on w0, the posterior distribution P (wt | x1..t−1, r1...t−1) also takes a Gaussian form, N ( ˆwt, Σt), with the mean and covariance given by the recursive Kalman filter update... |

667 | On sequential monte carlo sampling methods for bayesian filtering
- Doucet, Andrieu, et al.
(Show Context)
Citation Context ...ruptly and fluctuates perpetually. Here we suggest that individuals’ behavior in conditioning might be understood in terms of Monte Carlo methods for sequentially sampling different hypotheses (e.g., =-=[10]-=-). Such a model preserves the insights of a statistical framing while accounting for the characteristics of individual records. Also, through the metaphor of particle filtering, it also explains why e... |

153 |
Using the SIR Algorithm to Simulate Posterior Distributions
- Rubin
- 1988
(Show Context)
Citation Context ...e level, and at the ensemble level, it can be compensated for by importance reweighting and also by resampling (here we consider standard SIR importance resampling for sequential importance samplers; =-=[13, 10]-=-). Resampling kills off conditionally unlikely particles and keeps most samples in conditionally likely parts of the space, with similar and high importance weights. Since optimal reweighting and resa... |

90 | Structure and strength in causal induction
- Griffiths, Tenenbaum
(Show Context)
Citation Context ...ng in animals (and analogously structured prediction tasks in humans). There is a rich program of reinterpreting data from such experiments (which go back a century) in terms of statistical inference =-=[1, 2, 3, 4, 5, 6]-=-. The data do appear in a number of respects to reflect key features of the Bayesian ideal — specifically, that subjects represent beliefs as distributions with uncertainty and ∗ DRAFT version: NIPS 2... |

86 |
Forward and backward blocking in human contingency judgement
- Shanks
- 1985
(Show Context)
Citation Context ...ertainty and ∗ DRAFT version: NIPS 2007 preproceedings 1sappropriately employ it in updating them in light of new evidence. Most notable in this respect are retrospective revaluation phenomena (e.g., =-=[7, 8]-=-), which demonstrate that subjects are able to revise previously favored beliefs in a way suggesting they had entertained alternative hypotheses all along [6]. However, the data addressed by such mode... |

41 |
The learning curve: implications of a quantitative analysis
- Gallistel, Fairhurst, et al.
- 2004
(Show Context)
Citation Context ...raises the question whether individuals really exhibit the sophistication attributed to them by the models, or if it instead somehow emerges from the ensemble. Recent work by Gallistel and colleagues =-=[9]-=- frames the problem particularly sharply. Whereas subject-averaged responses exhibit smooth learning curves approaching asymptote (interpreted by Bayesian modelers as reflecting the gradual tradeoff f... |

32 | Biological significance in forward and backward blocking: Resolution of a discrepancy between animal conditioning and human causal judgment
- R, Matute
- 1996
(Show Context)
Citation Context ...AB+ trials, so the association of A with reward is thereby retrospectively discounted. Backward blocking, like other retrospective revaluation phenomena, is hard to demonstrate in animals (though see =-=[16]-=-) but robust in humans [7, 8]. Kakade and Dayan [6] gave a more formal analysis of the task in terms of the Kalman filter model. In particular they point out that on the initial AB+ trials, the model ... |

31 |
A theory of Pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement
- Rescorla, Wagner
- 1972
(Show Context)
Citation Context ...mean of the sampling distribution is ˆw L t + xtκ(rt − xt · ˆwt). Here the Kalman gain κ = σ 2 d /(σ2 d + σ2 o) is constant; the expected update in ˆw, then, is just that given by the Rescorla-Wagner =-=[12]-=- model. Such seemingly peculiar behavior may be motivated by the observation that, assuming that the initial ˆw L 0 is sampled according to the prior, this process also describes the evolution of a si... |

26 | Acquisition and extinction in autoshaping
- Kakade, Dayan
- 2002
(Show Context)
Citation Context ...ng in animals (and analogously structured prediction tasks in humans). There is a rich program of reinterpreting data from such experiments (which go back a century) in terms of statistical inference =-=[1, 2, 3, 4, 5, 6]-=-. The data do appear in a number of respects to reflect key features of the Bayesian ideal — specifically, that subjects represent beliefs as distributions with uncertainty and ∗ DRAFT version: NIPS 2... |

24 |
The computational neurobiology of learning and reward
- Daw, Doya
- 2006
(Show Context)
Citation Context ...they initially exhibited a low ˆwB and the degree to which they subsequently exhibited a backward blocking effect. This would be straightforward to test. More generally, there has been a recent trend =-=[17]-=- toward comparing models against raw trial-by-trial data sets according to the cumulative log-likelihood of the data. Although this measure aggregates over trials and subjects, it measures the average... |

23 | Explaining away in weight space
- Dayan, Kakade
- 2001
(Show Context)
Citation Context ...ng in animals (and analogously structured prediction tasks in humans). There is a rich program of reinterpreting data from such experiments (which go back a century) in terms of statistical inference =-=[1, 2, 3, 4, 5, 6]-=-. The data do appear in a number of respects to reflect key features of the Bayesian ideal — specifically, that subjects represent beliefs as distributions with uncertainty and ∗ DRAFT version: NIPS 2... |

22 | Learning domain structures
- Kemp, Perfors, et al.
(Show Context)
Citation Context |

22 | Model uncertainty in classical conditioning
- Courville, Daw, et al.
- 2004
(Show Context)
Citation Context |

16 | Similarity and discrimination in classical conditioning: A latent variable account
- Courville, Daw
- 2004
(Show Context)
Citation Context |

14 |
Forward and backward blocking of causal judgment is enhanced by additivity of effect magnitude
- Lovibond, Been, et al.
- 2003
(Show Context)
Citation Context ...ertainty and ∗ DRAFT version: NIPS 2007 preproceedings 1sappropriately employ it in updating them in light of new evidence. Most notable in this respect are retrospective revaluation phenomena (e.g., =-=[7, 8]-=-), which demonstrate that subjects are able to revise previously favored beliefs in a way suggesting they had entertained alternative hypotheses all along [6]. However, the data addressed by such mode... |

9 | A: Expected and unexpected uncertainty: Ach and NE
- Dayan, Yu
(Show Context)
Citation Context ...weights over the ensemble, they are not available to our subject-as-sample. However, there are some generative models that are more forgiving of these problems. In particular, consider Yu and Dayan’s =-=[14]-=- diffusion-jump model, which replaces Equation 1 with P (wt+1 | wt) = (1 − π)N (wt, σ 2 dI) + πN (0, σ 2 j I) (3) with σj ≫ σd. Here, the weights usually diffuse as before, but occasionally (with prob... |