Elsevier

Neural Networks

Volume 19, Issue 8, October 2006, Pages 1277-1301
Neural Networks

2006 Special Issue
Neural systems implicated in delayed and probabilistic reinforcement

https://doi.org/10.1016/j.neunet.2006.03.004Get rights and content

Abstract

This review considers the theoretical problems facing agents that must learn and choose on the basis of reward or reinforcement that is uncertain or delayed, in implicit or procedural (stimulus–response) representational systems and in explicit or declarative (action–outcome–value) representational systems. Individual differences in sensitivity to delays and uncertainty may contribute to impulsivity and risk taking. Learning and choice with delayed and uncertain reinforcement are related but in some cases dissociable processes. The contributions to delay and uncertainty discounting of neuromodulators including serotonin, dopamine, and noradrenaline, and of specific neural structures including the nucleus accumbens core, nucleus accumbens shell, orbitofrontal cortex, basolateral amygdala, anterior cingulate cortex, medial prefrontal (prelimbic/infralimbic) cortex, insula, subthalamic nucleus, and hippocampus are examined.

Section snippets

Delayed and uncertain reinforcement: The problems of learning and choice

Natural and artificial learning agents must grapple with the problem of selecting actions to achieve the best possible outcome under their value system. However, the outcome of a given action is not always certain and immediate. Outcomes are frequently uncertain: agents do not always obtain that for which they work. Furthermore, when an agent acts to obtain reward or reinforcement, there is often a delay between its action and the ultimate outcome. This applies both to positive reinforcers

Individual differences: Risk taking and impulsivity

Individual differences in responsivity to uncertain or delayed reinforcement are also of considerable interest. When making decisions under conditions of uncertainty, individuals vary as to how much uncertainty or risk they are willing to tolerate. Formally, individuals differ in how much they discount the value of reinforcers as the uncertainty of the reinforcer increases (i.e. as the probability of the reinforcer declines, or the odds against obtaining the reinforcer increase) (Ho, Mobini,

Learning with delayed reinforcement in normal animals

Delays can hamper both Pavlovian and instrumental conditioning (Dickinson, 1980, Dickinson, 1994, Gallistel, 1994, Hall, 1994, Mackintosh, 1983): for example, instrumental conditioning has long been observed to be systematically impaired as the outcome is delayed (Dickinson et al., 1992, Grice, 1948, Harker, 1956, Lattal and Gleeson, 1990, Perin, 1943, Skinner, 1938). Despite this, normal rats have been shown to acquire free-operant responding with programmed response–reinforcer delays of up to

Delayed and probabilistic reinforcement: Equivalent or distinct processes?

It has been suggested that delay (or temporal) discounting, the process by which delayed reinforcers lose value, and probability (or odds) discounting, the process by which uncertain reinforcers lose value, reflect the same underlying process (Green and Myerson, 1996, Mischel, 1966, Mazur, 1989, Mazur, 1995, Mazur, 1997, Rachlin et al., 1987, Rachlin et al., 1986, Rachlin et al., 1991, Rotter, 1954, Sozou, 1998, Stevenson, 1986). For example, choosing an uncertain reinforcer five times but only

Systemic pharmacological studies

Given the importance of impulsive choice in disorders such as addiction (Bickel et al., 1999, Evenden, 1999a, Heyman, 1996, Mitchell, 1999, Poulos et al., 1995) and attention-deficit/hyperactivity disorder (ADHD) (Sagvolden et al., 1998, Sagvolden and Sergeant, 1998), a number of groups have studied the effects on impulsive choice of manipulating neurochemical and neuroanatomical systems implicated in these disorders. I will review pharmacological and neurochemical studies first. To date, more

Neuroanatomically specific studies

In recent years, a number of studies have examined the effects of focal excitotoxic or neurochemical lesions on choice and learning involving delayed or uncertain rewards, in additional to correlational studies using functional imaging, microdialysis, and electrophysiology. These studies centre on interconnected structures forming part of the limbic corticostriatal loop (Fig. 2).

Conclusions

A number of limbic corticostriatal structures, together with the major forebrain neuromodulatory systems, play a role in learning and choice involving delayed and probabilistic rewards. The contribution of these structures is best understood for delayed reward (Fig. 2), although recent functional imaging and lesion studies have examined the neuroanatomical basis of choice involving uncertain reward.

To summarize, many structures have been implicated in the processing of delayed and/or

Acknowledgements

This work was supported by the UK Medical Research Council (MRC) and the Wellcome Trust within the University of Cambridge Behavioural and Clinical Neuroscience Institute. I thank two anonymous referees for their helpful comments.

References (373)

  • G. Bush et al.

    Anterior cingulate cortex dysfunction in attention-deficit/hyperactivity disorder revealed by fMRI and the Counting Stroop

    Biological Psychiatry

    (1999)
  • R.N. Cardinal et al.

    Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex

    Neuroscience and Biobehavioral Reviews

    (2002)
  • M.P. Carey et al.

    Differential distribution, affinity and plasticity of dopamine D-1 and D-2 receptors in the target sites of the mesolimbic system in an animal model of ADHD

    Behavioural Brain Research

    (1998)
  • D. Charrier et al.

    Effects of psychotropic drugs on rat responding in an operant paradigm involving choice between delayed reinforcers

    Pharmacology, Biochemistry and Behavior

    (1996)
  • L. Clark et al.

    The contributions of lesion laterality and lesion volume to decision-making impairment following frontal lobe damage

    Neuropsychologia

    (2003)
  • B.J. Cole et al.

    Effects of 6-hydroxydopamine lesions of the nucleus accumbens septi on performance of a 5-choice serial reaction time task in rats: Implications for theories of selective attention and arousal

    Behavioural Brain Research

    (1989)
  • K. Coleman-Mesches et al.

    Unilateral amygdala inactivation after training attenuates memory for reduced reward

    Behavioural Brain Research

    (1996)
  • S.J. Cooper et al.

    5-Hydroxyindoleacetic acid in cerebrospinal fluid and prediction of suicidal behaviour in schizophrenia

    Lancet

    (1992)
  • E. Corruble et al.

    Understanding impulsivity in severe depression? A psychometrical contribution

    Progress in Neuro-Psychopharmacology and Biological Psychiatry

    (2003)
  • M.S. Cousins et al.

    Nucleus accumbens dopamine depletions alter relative response allocation in a T-maze cost/benefit task

    Behavioural Brain Research

    (1996)
  • J. Crean et al.

    Effect of tryptophan depletion on impulsive behavior in men with or without a family history of alcoholism

    Behavioural Brain Research

    (2002)
  • D. Cremniter et al.

    CSF 5-HIAA levels are lower in impulsive as compared to nonimpulsive violent suicide attempters and control subjects

    Biological Psychiatry

    (1999)
  • J.W. Dalley et al.

    Deficits in impulse control associated with tonically-elevated serotonergic function in rat prefrontal cortex

    Neuropsychopharmacology

    (2002)
  • H. de Wit et al.

    Acute administration of d-amphetamine decreases impulsivity in healthy volunteers

    Neuropsychopharmacology

    (2002)
  • P.L. Delgado et al.

    Neuroendocrine and behavioral effects of dietary tryptophan restriction in healthy subjects

    Life Sciences

    (1989)
  • A. Dickinson

    Instrumental conditioning

  • D.M. Eagle et al.

    Lesions of the medial prefrontal cortex or nucleus accumbens core do not impair inhibitory control in rats performing a stop-signal reaction time task

    Behavioural Brain Research

    (2003)
  • M. Ernst et al.

    Choice selection and reward anticipation: An fMRI study

    Neuropsychologia

    (2004)
  • G Ainslie

    Specious reward: A behavioral theory of impulsiveness and impulse control

    Psychological Bulletin

    (1975)
  • G. Ainslie

    Breakdown of will

    (2001)
  • C. Amiez et al.

    Anterior cingulate error-related activity is modulated by predicted reward

    European Journal of Neuroscience

    (2005)
  • Amiez, C., Joseph, J. P., & Procyk, E. (2005b). Reward encoding in the monkey anterior cingulate cortex [Advance Access...
  • S.G. Anagnostaras et al.

    Temporally graded retrograde amnesia of contextual fear after hippocampal damage in rats: Within-subjects examination

    Journal of Neuroscience

    (1999)
  • I.M. Anderson et al.

    The effect of acute tryptophan depletion on probabilistic choice

    Journal of Psychopharmacology

    (2003)
  • APA

    Diagnostic and statistical manual of mental disorders

    (2000)
  • A. Apter et al.

    Anxiety, impulsivity and depressed mood in relation to suicidal and violent behavior

    Acta Psychiatrica Scandinavica

    (1993)
  • A.R. Aron et al.

    Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning

    Journal of Neurophysiology

    (2004)
  • M. Åsberg

    Neurotransmitters and suicidal behavior. The evidence from cerebrospinal fluid studies

    Annals of the New York Academy of Sciences

    (1997)
  • M. Åsberg et al.

    5-HIAA in the cerebrospinal fluid: A biochemical suicide predictor

    Archives of General Psychiatry

    (1976)
  • G. Aston-Jones et al.

    The locus coeruleus, A5 and A7 noradrenergic cell groups

  • A.G. Barto

    Adaptive critics and the basal ganglia

  • A.M. Basso et al.

    Feeding induced by GABA(A) receptor stimulation within the nucleus accumbens shell: Regional mapping and characterization of macronutrient and taste preference

    Behavioral Neuroscience

    (1999)
  • C. Baunez et al.

    Effects of STN lesions on simple vs choice reaction time tasks in the rat: Preserved motor readiness, but impaired response selection

    European Journal of Neuroscience

    (2001)
  • C. Baunez et al.

    Bilateral lesions of the subthalamic nucleus induce multiple deficits in an attentional task in rats

    European Journal of Neuroscience

    (1997)
  • A. Bechara et al.

    Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making

    Journal of Neuroscience

    (1999)
  • A. Bechara et al.

    Deciding advantageously before knowing the advantageous strategy

    Science

    (1997)
  • A. Bechara et al.

    Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex

    Cerebral Cortex

    (1996)
  • H.W. Berendse et al.

    The connections of the medial part of the subthalamic nucleus in the rat: Evidence for a parallel organization

  • W.K. Bickel et al.

    Impulsivity and cigarette smoking: Delay discounting in current, never, and ex-smokers

    Psychopharmacology

    (1999)
  • Cited by (263)

    • The role of dopamine pharmacotherapy and addiction-like behaviors in Parkinson's disease

      2020, Progress in Neuro-Psychopharmacology and Biological Psychiatry
    View all citing articles on Scopus
    View full text