Citation Context

...n intermediate stimuli from fMRI-decodable image categories, allowing us to decode the prospective activation of the future path at decision time. Such activity was correlated, across both trials and subjects, with the extent to which choices successfully reflected model-based integration. Separate research in the cognitive neuroscience of memory has also suggested that there are important links between memory for the past and prospective thoughts about the future. This work started from the observation that amnesic patients with severe memory loss also have difficulty imagining future events [29,30]. Neuroimaging studies in the healthy brain also link both retrieving memory and imagining future events to activity in a common set of structures, the hippocampus and surrounding medial temporal lobe [31,32]. These studies suggest a role for hippocampal memories in prospection for decision making, though this was not tested directly. A related line of evidence does suggest that prospective thinking affects evaluation. Instructing human participants to imagine events in the future attenuates the discounting of delayed rewards in an inter-temporal choice task, an effect that is correlated with ...

Citation Context

...which will vary depending on issues like the amount of time pressure or training [7,52,53]. Prospective integration is highly flexible and can in principle be applied to any new situation. However, it depends on memory retrieval at the time of choice and therefore can delay execution. Retrospective integration is efficient and fast at decision time, because it allows new decisions to depend on integration that had been previously computed. Indeed, there is some evidence that decisions based on retrospective integration can be computed as quickly as decisions that require no integration at all [43]. This replay mechanism also provides for an ongoing integration of past and present experiences, connecting otherwise discrete experiences into a networked web of memories. However, precomputing all options is not possible and thus this approach is well suited for integration of experiences that already share overlapping elements, but not for integration in the service of truly new decisions. If replay may also happen between experience and the decisions it supports, then analogous questions — when to replay? which events to replay? — might similarly be understood in terms of rational analysi...

Citation Context

...ence toward rational cost-benefit analyses of cognitive control phenomena [55,56]. A second question, which we largely skirted, is what sorts of memory representations are operated on by the replay and preplay operations we have considered. Classic work on model-based decision making envisions that it operates over semantic representations (like maps), which may themselves arise from the integration or average over many distinct experiences [4]. Indeed, a similar replay story — in the form of systems consolidation — has been invoked to explain the formation of semantic from episodic knowledge [57]. However, there has also been increasing interest in the possibility that decisions are themselves derived more directly from episodic information — that is, by retrieving representations of the events on individual trials from working memory or episodic memory [58,59]. Interestingly, the hippocampal memory system is thought to be involved both withwww.sciencedirect.com Integrating memories to guide decisions Shohamy and Daw 89representations of individual episodes, and with more general relational links derived from multiple experiences (as in sensory preconditioning) [60]. It remains to b...

Citation Context

... not mutually exclusive, and indeed there is good evidenceCurrent Opinion in Behavioral Sciences 2015, 5:85–90 supporting each of them. This raises a larger question, though: how does the brain decide which sorts of strategies to evoke under which circumstances? Earlier work has considered rational accounts of the tradeoff between model-based and model-free decision making in terms of the relative costs (e.g., delay) and benefits (e.g., better chance of gaining rewards) of prospection at decision time, a tradeoff which will vary depending on issues like the amount of time pressure or training [7,52,53]. Prospective integration is highly flexible and can in principle be applied to any new situation. However, it depends on memory retrieval at the time of choice and therefore can delay execution. Retrospective integration is efficient and fast at decision time, because it allows new decisions to depend on integration that had been previously computed. Indeed, there is some evidence that decisions based on retrospective integration can be computed as quickly as decisions that require no integration at all [43]. This replay mechanism also provides for an ongoing integration of past and present e...

Citation Context

...t supports, then analogous questions — when to replay? which events to replay? — might similarly be understood in terms of rational analysis of costs and benefits. Such prioritization has been the product of some work in computer science [54] though — apart from a finding that place cell sequences favor unexplored locations that contain reward over those without it [39] — there is almost no neuroscientific data addressing this question. More generally, this is a promising instance of a broader trend in cognitive neuroscience toward rational cost-benefit analyses of cognitive control phenomena [55,56]. A second question, which we largely skirted, is what sorts of memory representations are operated on by the replay and preplay operations we have considered. Classic work on model-based decision making envisions that it operates over semantic representations (like maps), which may themselves arise from the integration or average over many distinct experiences [4]. Indeed, a similar replay story — in the form of systems consolidation — has been invoked to explain the formation of semantic from episodic knowledge [57]. However, there has also been increasing interest in the possibility that de...

Citation Context

...t have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking prospective neural activity to the flexible behavior it is hypothesized to support in a single task [28]. This study used a multistep reward learning task which, much like sensory preconditioning, examined to what extent subjects, when choosing an option, integrated informationwww.sciencedirect.com about rewards recently received during other, interleaved trials with different choic...

Citation Context

... been linked to the solution of behavioral tasks that demonstrably require integration. Addressing this gap, retrospective integration has also been examined in humans in the context of the sensory preconditioning task of Figure 1. Although it is possible to compute the value of A by evoking B (and then reward) at choice time — a prospective approach — it is equally possible that when B is evoked during Phase 2, this retrieves A, which is associated with reward at that point (Figure 2). fMRI and MEG studies of this task, using decodable image categories for the stimuli, support this mechanism [40,41]. These studies demonstrate that the flexible behavior at decision time — the tendency, for example, to choose A — reflects processes that happened earlier, during the learning phase. In particular, flexible decisions are correlated with activity in the hippocampus in the earlier learning phase, as well as with evidence for specific reactivation of the A stimulus at the same time as B is being rewarded. Earlier studies of several other integration tasks suggested a similar mechanism [42–44]. Similar integration might also be supported by re-activating raw experience from Phases I and II during...

Citation Context

...stinct episodes during learning/encoding, before a decision is ever confronted. In this sort of retrospective mechanism, the attribution of reward value to the blue square would have already been in place before a decision was ever required.But perhaps the most mechanistically suggestive examples arise in rodent spatial navigation, where representations of an animal’s position (as represented by patterns of activity in hippocampal place cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay ...

Citation Context

...t supports, then analogous questions — when to replay? which events to replay? — might similarly be understood in terms of rational analysis of costs and benefits. Such prioritization has been the product of some work in computer science [54] though — apart from a finding that place cell sequences favor unexplored locations that contain reward over those without it [39] — there is almost no neuroscientific data addressing this question. More generally, this is a promising instance of a broader trend in cognitive neuroscience toward rational cost-benefit analyses of cognitive control phenomena [55,56]. A second question, which we largely skirted, is what sorts of memory representations are operated on by the replay and preplay operations we have considered. Classic work on model-based decision making envisions that it operates over semantic representations (like maps), which may themselves arise from the integration or average over many distinct experiences [4]. Indeed, a similar replay story — in the form of systems consolidation — has been invoked to explain the formation of semantic from episodic knowledge [57]. However, there has also been increasing interest in the possibility that de...

Citation Context

...g [46], this mechanism might help explain why modelbased and model-free learning seem to share more neural substrates — such as dopamine and reward prediction errors — than previously expected [47–49]. Also, a recent study [50] showed that manipulating rest periods in a revaluation design analogous to Figure 1 had effects on humans’ success solving subsequent integration tests, consistent with the idea that this behavior was supported by processes occurring during rest. Night time sleep also seems to specifically enhance such integration, as shown using a transitive inference task in humans [51]. Conclusions and future questions Altogether, data support the idea that memories are retrieved and integrated to construct decision variables at a variety of times, ranging from the time of encoding to the time of decision. These mechanisms are clearly not mutually exclusive, and indeed there is good evidenceCurrent Opinion in Behavioral Sciences 2015, 5:85–90 supporting each of them. This raises a larger question, though: how does the brain decide which sorts of strategies to evoke under which circumstances? Earlier work has considered rational accounts of the tradeoff between model-based a...

Citation Context

...ar, flexible decisions are correlated with activity in the hippocampus in the earlier learning phase, as well as with evidence for specific reactivation of the A stimulus at the same time as B is being rewarded. Earlier studies of several other integration tasks suggested a similar mechanism [42–44]. Similar integration might also be supported by re-activating raw experience from Phases I and II during rest periods or sleep following Phase II, driving new learning about A via offline integration. In machine learning, such replay-driven learning has been proposed in an architecture called DYNA [45], which demonstrates how actual, remembered, and model-simulated experiences can be blended together for learning. Applied to biological learning [46], this mechanism might help explain why modelbased and model-free learning seem to share more neural substrates — such as dopamine and reward prediction errors — than previously expected [47–49]. Also, a recent study [50] showed that manipulating rest periods in a revaluation design analogous to Figure 1 had effects on humans’ success solving subsequent integration tests, consistent with the idea that this behavior was supported by processes oc...

Citation Context

...t have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking prospective neural activity to the flexible behavior it is hypothesized to support in a single task [28]. This study used a multistep reward learning task which, much like sensory preconditioning, examined to what extent subjects, when choosing an option, integrated informationwww.sciencedirect.com about rewards recently received during other, interleaved trials with different choic...

Citation Context

...ve, inferential reasoning in tasks similar to that of Figure 1 is accompanied or supported by neural events happening at the time decisions are faced [13–18]. In some of these cases, inferential reasoning is accompanied by activity in the hippocampus, presumably due to its role in supporting memory retrieval processes in service of inferences. Other work has highlighted the importance of the orbitofrontal cortex (OFC), consistent with the hypothesis that the OFC supports the construction of value when an organism is confronted with a new decision for which precomputed values are not available [15,19].www.sciencedirect.com Integrating memories to guide decisions Shohamy and Daw 87 Figure 2 ? $ $ $} Retrospective Integration Prospective Integration Current Opinion in Behavioral Sciences Schematic of possible mechanisms underlying integration of memories to guide decisions. When confronted with a new decision which cannot be wholly based on past rewards, such as predicting whether the blue square will lead to reward or not, participants’ behavior tends to reflect the integration of memory for past relevant events. This integration can happen via two distinct mechanisms. (a) One possibility i...

Citation Context

... is that the overlap in the memories themselves triggers integration of distinct episodes during learning/encoding, before a decision is ever confronted. In this sort of retrospective mechanism, the attribution of reward value to the blue square would have already been in place before a decision was ever required.But perhaps the most mechanistically suggestive examples arise in rodent spatial navigation, where representations of an animal’s position (as represented by patterns of activity in hippocampal place cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective c...

Citation Context

...on of reward value to the blue square would have already been in place before a decision was ever required.But perhaps the most mechanistically suggestive examples arise in rodent spatial navigation, where representations of an animal’s position (as represented by patterns of activity in hippocampal place cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were...

Citation Context

...arise in rodent spatial navigation, where representations of an animal’s position (as represented by patterns of activity in hippocampal place cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking ...

Citation Context

...ace cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking prospective neural activity to the flexible behavior it is hypothesized to support in a single task [28]. This study used a multistep reward l...

Citation Context

...ace cells) can ‘‘run ahead’’ of the animal’s position. This can be seen at choice points [20], can predict the direction of the animal’s own subsequent locomotion [21], and can be integrative in the sense of knitting together ‘‘subroutes’’ that have not actually yet been experienced sequentially [22]. A related oscillation in spatial representation, theta phase procession, was also shown to modulate in a way that reflected the animal’s current spatial goal [23]. Similar anticipatory representations in hippocampus and prefrontal cortex have been observed in statistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking prospective neural activity to the flexible behavior it is hypothesized to support in a single task [28]. This study used a multistep reward l...

Citation Context

...atistical learning tasks in humans [24,25]. Such anticipatory or ‘‘preplay’’ activity is a promising substrate for prospective computation of the value of actions. However, although hippocampal preplay appears to support correct navigational performance on simple spatial tasks [26,27], it has not so far been studied while animals were performing behavioral tasks that specifically require or demonstrate integrative, model-based decisions. A recent experiment aimed to close this circle by linking prospective neural activity to the flexible behavior it is hypothesized to support in a single task [28]. This study used a multistep reward learning task which, much like sensory preconditioning, examined to what extent subjects, when choosing an option, integrated informationwww.sciencedirect.com about rewards recently received during other, interleaved trials with different choice options. Choices resulted in intermediate stimuli from fMRI-decodable image categories, allowing us to decode the prospective activation of the future path at decision time. Such activity was correlated, across both trials and subjects, with the extent to which choices successfully reflected model-based integration....

Citation Context

...n intermediate stimuli from fMRI-decodable image categories, allowing us to decode the prospective activation of the future path at decision time. Such activity was correlated, across both trials and subjects, with the extent to which choices successfully reflected model-based integration. Separate research in the cognitive neuroscience of memory has also suggested that there are important links between memory for the past and prospective thoughts about the future. This work started from the observation that amnesic patients with severe memory loss also have difficulty imagining future events [29,30]. Neuroimaging studies in the healthy brain also link both retrieving memory and imagining future events to activity in a common set of structures, the hippocampus and surrounding medial temporal lobe [31,32]. These studies suggest a role for hippocampal memories in prospection for decision making, though this was not tested directly. A related line of evidence does suggest that prospective thinking affects evaluation. Instructing human participants to imagine events in the future attenuates the discounting of delayed rewards in an inter-temporal choice task, an effect that is correlated with ...

Citation Context

... been linked to the solution of behavioral tasks that demonstrably require integration. Addressing this gap, retrospective integration has also been examined in humans in the context of the sensory preconditioning task of Figure 1. Although it is possible to compute the value of A by evoking B (and then reward) at choice time — a prospective approach — it is equally possible that when B is evoked during Phase 2, this retrieves A, which is associated with reward at that point (Figure 2). fMRI and MEG studies of this task, using decodable image categories for the stimuli, support this mechanism [40,41]. These studies demonstrate that the flexible behavior at decision time — the tendency, for example, to choose A — reflects processes that happened earlier, during the learning phase. In particular, flexible decisions are correlated with activity in the hippocampus in the earlier learning phase, as well as with evidence for specific reactivation of the A stimulus at the same time as B is being rewarded. Earlier studies of several other integration tasks suggested a similar mechanism [42–44]. Similar integration might also be supported by re-activating raw experience from Phases I and II during...

Citation Context

...on of the A stimulus at the same time as B is being rewarded. Earlier studies of several other integration tasks suggested a similar mechanism [42–44]. Similar integration might also be supported by re-activating raw experience from Phases I and II during rest periods or sleep following Phase II, driving new learning about A via offline integration. In machine learning, such replay-driven learning has been proposed in an architecture called DYNA [45], which demonstrates how actual, remembered, and model-simulated experiences can be blended together for learning. Applied to biological learning [46], this mechanism might help explain why modelbased and model-free learning seem to share more neural substrates — such as dopamine and reward prediction errors — than previously expected [47–49]. Also, a recent study [50] showed that manipulating rest periods in a revaluation design analogous to Figure 1 had effects on humans’ success solving subsequent integration tests, consistent with the idea that this behavior was supported by processes occurring during rest. Night time sleep also seems to specifically enhance such integration, as shown using a transitive inference task in humans [51]. ...

Citation Context

...onnecting otherwise discrete experiences into a networked web of memories. However, precomputing all options is not possible and thus this approach is well suited for integration of experiences that already share overlapping elements, but not for integration in the service of truly new decisions. If replay may also happen between experience and the decisions it supports, then analogous questions — when to replay? which events to replay? — might similarly be understood in terms of rational analysis of costs and benefits. Such prioritization has been the product of some work in computer science [54] though — apart from a finding that place cell sequences favor unexplored locations that contain reward over those without it [39] — there is almost no neuroscientific data addressing this question. More generally, this is a promising instance of a broader trend in cognitive neuroscience toward rational cost-benefit analyses of cognitive control phenomena [55,56]. A second question, which we largely skirted, is what sorts of memory representations are operated on by the replay and preplay operations we have considered. Classic work on model-based decision making envisions that it operates over...

Citation Context

...isodic knowledge [57]. However, there has also been increasing interest in the possibility that decisions are themselves derived more directly from episodic information — that is, by retrieving representations of the events on individual trials from working memory or episodic memory [58,59]. Interestingly, the hippocampal memory system is thought to be involved both withwww.sciencedirect.com Integrating memories to guide decisions Shohamy and Daw 89representations of individual episodes, and with more general relational links derived from multiple experiences (as in sensory preconditioning) [60]. It remains to be understood to what extent the decision phenomena we have examined are supported by one or the other or both of these underlying representations [61]. Conflict of interest Nothing declared. Acknowledgements This work was supported by NINDS, grant R01NS078784 (DS, ND), NIDA grant R01DA038891 (ND, DS), by Google DeepMind (ND), and by NSF, grant 0955494(DS). References and recommended reading Papers of particular interest, published within the period of review, have been highlighted as: of special interest of outstanding interest 1. Houk JC, Adams JL, Barto AG: A model of h...