Abstract: A wealth of research focuses on the decision-making processes that animals and humans employ when selecting actions in the face of reward and punishment. Initially such work stemmed from psychological investigations of conditioned behavior, and explanations of these in terms of computational models. Increasingly, analysis at the computational level has drawn on ideas from reinforcement learning, which provide a normative framework within which decision-making can be analyzed. More recently, the fruits of these extensive lines of research have made contact with investigations into the neural basis of decision making. Converging evidence now links reinforcement learning to specific neural substrates, assigning them precise computational roles. Specifically, electrophysiological recordings in behaving animals and functional imaging of human decision-making have revealed in the brain the existence of a key reinforcement learning signal, the temporal difference reward prediction error. Here, we first introduce the formal reinforcement learning framework. We then review the multiple lines of evidence linking reinforcement learning to the function of dopaminergic neurons in the mammalian midbrain and

The framework provided by Claude Shannon’s (1948) theory of information leads to a far-reaching, more quantitatively oriented reconceptualization of the processes that mediate what is commonly called associative learning. The focus shifts from processes set in motion by individual events to processes sensitive to the information carried by the flow of events. The conception of what properties of the conditioned and unconditioned stimuli are important shifts from the tangible properties that excite sensory receptors to the abstract and intangible properties of number, duration, frequency and contingency, which are the carriers of the information. Frequency is an abstraction built on abstractions—one intangible, number, divided by another intangible, time. A contingent frequency raises the pyramid of abstractions still higher. It is the rate at which an event occurs following the onset or offset of a conditioning event. Sensitivity to contingent frequency requires frequency estimation together with the estimation of contingency, which is itself a forbidding abstraction

Many theories of category learning assume that learning is driven by a need to minimize classification error. When there is no classification error, therefore, learning of individual features should be negligible. The authors tested this hypothesis by conducting three category-learning experiments adapted from an associative learning blocking paradigm. Contrary to an error-driven account of learning, participants learned a wide range of information when they learned about categories, and blocking effects were difficult to obtain. Conversely, when participants learned to predict an outcome in a task with the same formal structure and materials, blocking effects were robust and followed the predictions of error-driven learning. The authors discuss their findings in relation to models of category learning and the usefulness of category knowledge in the environment.

Research has shown that learning a concept via standard supervised classification leads to a focus on diagnostic features, whereas learning by inferring missing features promotes the acquisition of withincategory information. Accordingly, we predicted that classification learning would produce a deficit in people’s ability to draw novel contrasts—distinctions that were not part of training—compared with feature inference learning. Two experiments confirmed that classification learners were at a disadvantage at making novel distinctions. Eye movement data indicated that this conceptual inflexibility was due to (a) a narrower attention profile that reduces the encoding of many category features and (b) learned inattention that inhibits the reallocation of attention to newly relevant information. Implications of these costs of supervised classification learning for views of conceptual structure are discussed.

Bayesian treatments of animal conditioning start from a generative model that specifies precisely a set of assumptions about the structure of the learning task. Optimal rules for learning are direct mathematical consequences of these assumptions. In terms of Marr’s (1982) levels of analyses, the main task at the computational level

The effects of altering the contingency between the conditioned stimulus (CS) and the unconditioned stimulus (US) on the acquisition of autoshaped responding was investigated by changing the frequency of unsignaled USs during the intertrial interval. The addition of the unsignaled USs had an effect on acquisition speed comparable with that of massing trials. The effects of these manipulations can be understood in terms of their effect on the amount of information (number of bits) that the average CS conveys to the subject about the timing of the next US. The number of reinforced CSs prior to acquisition is inversely related to the information content of the CS.

Research has shown that category learning is affected by (a) attention, which selects which aspects of stimuli are available for further processing, and (b) the existing semantic knowledge that learners bring to the task. However, little is known about how knowledge affects what is attended. Using eyetracking, we found that (a) knowledge indeed changes what features are attended, with knowledgerelevant features being fixated more often than irrelevant ones, (b) this effect was not due to an initial attentional bias toward relevant dimensions but rather emerged as a result of observing category members, and (c) this effect grew even after a learning criterion was reached, that is, despite the absence of error feedback. We argue that models of knowledge-based learning will remain incomplete until they include mechanisms that dynamically select prior knowledge in response to observed category members and which then directs attention to knowledge-relevant dimensions and away from irrelevant ones. Knowledge and Attention in Category Learning 3 How Prior Knowledge Affects Selective Attention During Category Learning:

The authors investigated people's ability to restructure their knowledge when additional information about a categorization task is revealed. In 2 experiments, people first learned to rely on a fairly accurate (but imperfect) predictor. At various points in training, a complex relationship between 2 other predictors was revealed in a schematic diagram that could support perfect performance. In Experiment 1, people adopted the complex strategy when it was revealed at the outset but were unable to restructure their knowledge after the expedient predictor had been learned. In Experiment 2, expedient knowledge persisted even with an adaptive display. The persistence of expedient knowledge is explained by associative blocking of potential alternative cues. A 3rd experiment analyzed the strategies people use with and without the diagram. The study confirmed that the diagram, when presented at the outset, significantly alters people's approach to the task. Knowledge restructuring is often observed when people develop new skills, both in the real world and in the laboratory; paradoxically, expertise at these skills in some cases also seems to prevent restructuring of knowledge. Little is known about the processes involved in knowledge restructuring, and our investigation formed