2006 Special IssueNeural systems implicated in delayed and probabilistic reinforcement
Section snippets
Delayed and uncertain reinforcement: The problems of learning and choice
Natural and artificial learning agents must grapple with the problem of selecting actions to achieve the best possible outcome under their value system. However, the outcome of a given action is not always certain and immediate. Outcomes are frequently uncertain: agents do not always obtain that for which they work. Furthermore, when an agent acts to obtain reward or reinforcement, there is often a delay between its action and the ultimate outcome. This applies both to positive reinforcers
Individual differences: Risk taking and impulsivity
Individual differences in responsivity to uncertain or delayed reinforcement are also of considerable interest. When making decisions under conditions of uncertainty, individuals vary as to how much uncertainty or risk they are willing to tolerate. Formally, individuals differ in how much they discount the value of reinforcers as the uncertainty of the reinforcer increases (i.e. as the probability of the reinforcer declines, or the odds against obtaining the reinforcer increase) (Ho, Mobini,
Learning with delayed reinforcement in normal animals
Delays can hamper both Pavlovian and instrumental conditioning (Dickinson, 1980, Dickinson, 1994, Gallistel, 1994, Hall, 1994, Mackintosh, 1983): for example, instrumental conditioning has long been observed to be systematically impaired as the outcome is delayed (Dickinson et al., 1992, Grice, 1948, Harker, 1956, Lattal and Gleeson, 1990, Perin, 1943, Skinner, 1938). Despite this, normal rats have been shown to acquire free-operant responding with programmed response–reinforcer delays of up to
Delayed and probabilistic reinforcement: Equivalent or distinct processes?
It has been suggested that delay (or temporal) discounting, the process by which delayed reinforcers lose value, and probability (or odds) discounting, the process by which uncertain reinforcers lose value, reflect the same underlying process (Green and Myerson, 1996, Mischel, 1966, Mazur, 1989, Mazur, 1995, Mazur, 1997, Rachlin et al., 1987, Rachlin et al., 1986, Rachlin et al., 1991, Rotter, 1954, Sozou, 1998, Stevenson, 1986). For example, choosing an uncertain reinforcer five times but only
Systemic pharmacological studies
Given the importance of impulsive choice in disorders such as addiction (Bickel et al., 1999, Evenden, 1999a, Heyman, 1996, Mitchell, 1999, Poulos et al., 1995) and attention-deficit/hyperactivity disorder (ADHD) (Sagvolden et al., 1998, Sagvolden and Sergeant, 1998), a number of groups have studied the effects on impulsive choice of manipulating neurochemical and neuroanatomical systems implicated in these disorders. I will review pharmacological and neurochemical studies first. To date, more
Neuroanatomically specific studies
In recent years, a number of studies have examined the effects of focal excitotoxic or neurochemical lesions on choice and learning involving delayed or uncertain rewards, in additional to correlational studies using functional imaging, microdialysis, and electrophysiology. These studies centre on interconnected structures forming part of the limbic corticostriatal loop (Fig. 2).
Conclusions
A number of limbic corticostriatal structures, together with the major forebrain neuromodulatory systems, play a role in learning and choice involving delayed and probabilistic rewards. The contribution of these structures is best understood for delayed reward (Fig. 2), although recent functional imaging and lesion studies have examined the neuroanatomical basis of choice involving uncertain reward.
To summarize, many structures have been implicated in the processing of delayed and/or
Acknowledgements
This work was supported by the UK Medical Research Council (MRC) and the Wellcome Trust within the University of Cambridge Behavioural and Clinical Neuroscience Institute. I thank two anonymous referees for their helpful comments.
References (373)
- et al.
Functional architecture of basal ganglia circuits: Neural substrates of parallel processing
Trends in Neurosciences
(1990) - et al.
Changes of cerebrospinal fluid monoamine metabolites during long-term antidepressant treatment
European Neuropsychopharmacology
(2000) - et al.
Effects of ibotenic acid lesions of the nucleus accumbens on instrumental action
Behavioural Brain Research
(1994) - et al.
Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates
Neuropharmacology
(1998) - et al.
Differential responsiveness of dopamine transmission to food-stimuli in nucleus accumbens shell/core compartments
Neuroscience
(1999) - et al.
Insensitivity to future consequences following damage to human prefrontal cortex
Cognition
(1994) Measuring hedonic impact in animals and infants: Microstructure of affective taste reactivity patterns
Neuroscience and Biobehavioral Reviews
(2000)- et al.
Rapid depletion of serum tryptophan, brain tryptophan, serotonin and 5-hydroxyindoleacetic acid by a trytophan-free diet
Life Sciences
(1974) - et al.
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Neuron
(2001) - et al.
Effects of dorsal and ventral striatal lesions on delayed matching trained with retractable levers
Behavioural Brain Research
(2001)
Anterior cingulate cortex dysfunction in attention-deficit/hyperactivity disorder revealed by fMRI and the Counting Stroop
Biological Psychiatry
Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex
Neuroscience and Biobehavioral Reviews
Differential distribution, affinity and plasticity of dopamine D-1 and D-2 receptors in the target sites of the mesolimbic system in an animal model of ADHD
Behavioural Brain Research
Effects of psychotropic drugs on rat responding in an operant paradigm involving choice between delayed reinforcers
Pharmacology, Biochemistry and Behavior
The contributions of lesion laterality and lesion volume to decision-making impairment following frontal lobe damage
Neuropsychologia
Effects of 6-hydroxydopamine lesions of the nucleus accumbens septi on performance of a 5-choice serial reaction time task in rats: Implications for theories of selective attention and arousal
Behavioural Brain Research
Unilateral amygdala inactivation after training attenuates memory for reduced reward
Behavioural Brain Research
5-Hydroxyindoleacetic acid in cerebrospinal fluid and prediction of suicidal behaviour in schizophrenia
Lancet
Understanding impulsivity in severe depression? A psychometrical contribution
Progress in Neuro-Psychopharmacology and Biological Psychiatry
Nucleus accumbens dopamine depletions alter relative response allocation in a T-maze cost/benefit task
Behavioural Brain Research
Effect of tryptophan depletion on impulsive behavior in men with or without a family history of alcoholism
Behavioural Brain Research
CSF 5-HIAA levels are lower in impulsive as compared to nonimpulsive violent suicide attempters and control subjects
Biological Psychiatry
Deficits in impulse control associated with tonically-elevated serotonergic function in rat prefrontal cortex
Neuropsychopharmacology
Acute administration of d-amphetamine decreases impulsivity in healthy volunteers
Neuropsychopharmacology
Neuroendocrine and behavioral effects of dietary tryptophan restriction in healthy subjects
Life Sciences
Instrumental conditioning
Lesions of the medial prefrontal cortex or nucleus accumbens core do not impair inhibitory control in rats performing a stop-signal reaction time task
Behavioural Brain Research
Choice selection and reward anticipation: An fMRI study
Neuropsychologia
Specious reward: A behavioral theory of impulsiveness and impulse control
Psychological Bulletin
Breakdown of will
Anterior cingulate error-related activity is modulated by predicted reward
European Journal of Neuroscience
Temporally graded retrograde amnesia of contextual fear after hippocampal damage in rats: Within-subjects examination
Journal of Neuroscience
The effect of acute tryptophan depletion on probabilistic choice
Journal of Psychopharmacology
Diagnostic and statistical manual of mental disorders
Anxiety, impulsivity and depressed mood in relation to suicidal and violent behavior
Acta Psychiatrica Scandinavica
Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning
Journal of Neurophysiology
Neurotransmitters and suicidal behavior. The evidence from cerebrospinal fluid studies
Annals of the New York Academy of Sciences
5-HIAA in the cerebrospinal fluid: A biochemical suicide predictor
Archives of General Psychiatry
The locus coeruleus, A5 and A7 noradrenergic cell groups
Adaptive critics and the basal ganglia
Feeding induced by GABA(A) receptor stimulation within the nucleus accumbens shell: Regional mapping and characterization of macronutrient and taste preference
Behavioral Neuroscience
Effects of STN lesions on simple vs choice reaction time tasks in the rat: Preserved motor readiness, but impaired response selection
European Journal of Neuroscience
Bilateral lesions of the subthalamic nucleus induce multiple deficits in an attentional task in rats
European Journal of Neuroscience
Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making
Journal of Neuroscience
Deciding advantageously before knowing the advantageous strategy
Science
Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex
Cerebral Cortex
The connections of the medial part of the subthalamic nucleus in the rat: Evidence for a parallel organization
Impulsivity and cigarette smoking: Delay discounting in current, never, and ex-smokers
Psychopharmacology
Cited by (263)
A behavioral timing intervention upregulates striatal serotonergic markers and reduces impulsive action in adult male mice
2023, Behavioural Brain ResearchEvent-based control of autonomic and emotional states by the anterior cingulate cortex
2022, Neuroscience and Biobehavioral ReviewsCentral but not peripheral oxytocin administration reduces risk-based decision-making in male rats
2020, Hormones and BehaviorThe role of dopamine pharmacotherapy and addiction-like behaviors in Parkinson's disease
2020, Progress in Neuro-Psychopharmacology and Biological Psychiatry