QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

HOME  |   SEARCH  |   ARCHIVE  |   SUBSCRIBE  |   CONTACT  |   HELP

This Article Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Montague, P. R. Articles by Sejnowski, T. J. Search for Related Content PubMed PubMed Citation Articles by Montague, P. R. Articles by Sejnowski, T. J.

 Previous Article  |  Next Article 

Journal of Neuroscience, Vol 16, 1936-1947, Copyright © 1996 by Society for Neuroscience

ARTICLE

A framework for mesencephalic dopamine systems based on predictive Hebbian learning

PR Montague, P Dayan and TJ Sejnowski
Division of Neuroscience, Baylor College of Medicine, Houston, Texas 77030, USA.

We develop a theoretical framework that shows how mesencephalic dopamine systems could distribute to their targets a signal that represents information about future expectations. In particular, we show how activity in the cerebral cortex can make predictions about future receipt of reward and how fluctuations in the activity levels of neurons in diffuse dopamine systems above and below baseline levels would represent errors in these predictions that are delivered to cortical and subcortical targets. We present a model for how such errors could be constructed in a real brain that is consistent with physiological results for a subset of dopaminergic neurons located in the ventral tegmental area and surrounding dopaminergic neurons. The theory also makes testable predictions about human choice behavior on a simple decision-making task. Furthermore, we show that, through a simple influence on synaptic plasticity, fluctuations in dopamine release can act to change the predictions in an appropriate manner.

This article has been cited by other articles:

				C. Fang and D. Levinthal Near-Term Liability of Exploitation: Exploration and Exploitation in Multistage Problems Organization Science, May 1, 2009; 20(3): 538 - 551. [Abstract] [PDF]

				C. A. Mejias-Aponte, C. Drouin, and G. Aston-Jones Adrenergic and Noradrenergic Innervation of the Midbrain Ventral Tegmental Area and Retrorubral Field: Prominent Inputs from Medullary Homeostatic Centers J. Neurosci., March 18, 2009; 29(11): 3613 - 3626. [Abstract] [Full Text] [PDF]

				K. A. Zaghloul, J. A. Blanco, C. T. Weidemann, K. McGill, J. L. Jaggi, G. H. Baltuch, and M. J. Kahana Human Substantia Nigra Neurons Encode Unexpected Financial Rewards Science, March 13, 2009; 323(5920): 1496 - 1499. [Abstract] [Full Text] [PDF]

				R. Cools, M. J. Frank, S. E. Gibbs, A. Miyakawa, W. Jagust, and M. D'Esposito Striatal Dopamine Predicts Outcome-Specific Reversal Learning and Its Sensitivity to Dopaminergic Drug Administration J. Neurosci., February 4, 2009; 29(5): 1538 - 1543. [Abstract] [Full Text] [PDF]

				J. Glascher, A. N. Hampton, and J. P. O'Doherty Determining a Role for Ventromedial Prefrontal Cortex in Encoding Action-Based Value Signals During Reward-Related Decision Making Cereb Cortex, February 1, 2009; 19(2): 483 - 495. [Abstract] [Full Text] [PDF]

				T. Wachter, O. V. Lungu, T. Liu, D. T. Willingham, and J. Ashe Differential Effect of Reward and Punishment on Procedural Learning J. Neurosci., January 14, 2009; 29(2): 436 - 443. [Abstract] [Full Text] [PDF]

				B. Knutson and S. M Greer Anticipatory affect: neural correlates and consequences for choice Phil Trans R Soc B, December 12, 2008; 363(1511): 3771 - 3786. [Abstract] [Full Text] [PDF]

				M. R Delgado, J. Li, D. Schiller, and E. A Phelps The role of the striatum in aversive learning and aversive prediction errors Phil Trans R Soc B, December 12, 2008; 363(1511): 3787 - 3800. [Abstract] [Full Text] [PDF]

				N. Shea, K. Krug, and P. N. Tobler Conceptual representations in goal-directed decision making Cogn Affect Behav Neurosci, December 1, 2008; 8(4): 418 - 428. [Abstract] [PDF]

				P. Dayan and N. D. Daw Decision theory, reinforcement learning, and the brain Cogn Affect Behav Neurosci, December 1, 2008; 8(4): 429 - 453. [Abstract] [PDF]

				A. A. Moustafa, M. X. Cohen, S. J. Sherman, and M. J. Frank A Role for Dopamine in Temporal Decision Making and Reward Maximization in Parkinsonism J. Neurosci., November 19, 2008; 28(47): 12294 - 12304. [Abstract] [Full Text] [PDF]

				D. Schiller, I. Levy, Y. Niv, J. E. LeDoux, and E. A. Phelps From Fear to Safety and Back: Reversal of Fear in the Human Brain J. Neurosci., November 5, 2008; 28(45): 11517 - 11525. [Abstract] [Full Text] [PDF]

				M. J. Frank Schizophrenia: A Computational Reinforcement Learning Perspective Schizophr Bull, November 1, 2008; 34(6): 1008 - 1011. [Full Text] [PDF]

				W.-X. Pan, R. Schmidt, J. R. Wickens, and B. I. Hyland Tripartite Mechanism of Extinction Suggested by Dopamine Neuron Activity and Temporal Difference Model J. Neurosci., September 24, 2008; 28(39): 9619 - 9631. [Abstract] [Full Text] [PDF]

				P. Kumar, G. Waiter, T. Ahearn, M. Milders, I. Reid, and J. D. Steele Abnormal temporal difference reward-learning signals in major depression Brain, August 1, 2008; 131(8): 2084 - 2093. [Abstract] [Full Text] [PDF]

				S. Kobayashi and W. Schultz Influence of Reward Delays on Responses of Dopamine Neurons J. Neurosci., July 30, 2008; 28(31): 7837 - 7846. [Abstract] [Full Text] [PDF]

				M. X. COHEN Neurocomputational mechanisms of reinforcement-guided learning in humans: A review Cogn Affect Behav Neurosci, June 1, 2008; 8(2): 113 - 125. [Abstract] [PDF]

				K. Nakamura, M. Matsumoto, and O. Hikosaka Reward-Dependent Modulation of Neuronal Activity in the Primate Dorsal Raphe Nucleus J. Neurosci., May 14, 2008; 28(20): 5331 - 5343. [Abstract] [Full Text] [PDF]

				K. Preuschoff, S. R. Quartz, and P. Bossaerts Human Insula Activation Reflects Risk Prediction Errors As Well As Risk J. Neurosci., March 12, 2008; 28(11): 2745 - 2752. [Abstract] [Full Text] [PDF]

				V. Pawlak and J. N. D. Kerr Dopamine Receptor Activation Is Required for Corticostriatal Spike-Timing-Dependent Plasticity J. Neurosci., March 5, 2008; 28(10): 2435 - 2446. [Abstract] [Full Text] [PDF]

				K. D'Ardenne, S. M. McClure, L. E. Nystrom, and J. D. Cohen BOLD Responses Reflecting Dopaminergic Signals in the Human Ventral Tegmental Area Science, February 29, 2008; 319(5867): 1264 - 1267. [Abstract] [Full Text] [PDF]

				Y. B. Kim, N. Huh, H. Lee, E. H. Baeg, D. Lee, and M. W. Jung Encoding of Action History in the Rat Ventral Striatum J Neurophysiol, December 1, 2007; 98(6): 3548 - 3556. [Abstract] [Full Text] [PDF]

				T. Schonberg, N. D. Daw, D. Joel, and J. P. O'Doherty Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making J. Neurosci., November 21, 2007; 27(47): 12860 - 12867. [Abstract] [Full Text] [PDF]

				Y. Niv and M. Rivlin-Etzion Parkinson's Disease: Fighting the Will? J. Neurosci., October 31, 2007; 27(44): 11777 - 11779. [Full Text] [PDF]

				E. M. Izhikevich Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling Cereb Cortex, October 1, 2007; 17(10): 2443 - 2452. [Abstract] [Full Text] [PDF]

				T. E Hazy, M. J Frank, and R. C O'Reilly Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system Phil Trans R Soc B, September 29, 2007; 362(1485): 1601 - 1613. [Abstract] [Full Text] [PDF]

				W. H. Alexander Shifting Attention Using a Temporal Difference Prediction Error and High-Dimensional Input Adaptive Behavior, June 1, 2007; 15(2): 121 - 133. [Abstract] [PDF]

				J. D Cohen, S. M McClure, and A. J Yu Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration Phil Trans R Soc B, May 29, 2007; 362(1481): 933 - 942. [Abstract] [Full Text] [PDF]

				T. Lohrenz, K. McCabe, C. F. Camerer, and P. R. Montague Neural signature of fictive learning signals in a sequential investment task PNAS, May 29, 2007; 104(22): 9493 - 9498. [Abstract] [Full Text] [PDF]

				B. Seymour, N. Daw, P. Dayan, T. Singer, and R. Dolan Differential Encoding of Losses and Gains in the Human Striatum J. Neurosci., May 2, 2007; 27(18): 4826 - 4831. [Abstract] [Full Text] [PDF]

				M. X Cohen Individual differences and the neural representations of reward expectation and reward prediction error Soc Cogn Affect Neurosci, March 1, 2007; 2(1): 20 - 30. [Abstract] [Full Text] [PDF]

				S. E. Hyman Addiction: A Disease of Learning and Memory Focus, January 1, 2007; 5(2): 220 - 228. [Abstract] [Full Text] [PDF]

				J.-P. Changeux The Ferrier Lecture 1998 The molecular biology of consciousness investigated with genetically modified mice Phil Trans R Soc B, December 29, 2006; 361(1476): 2239 - 2259. [Abstract] [Full Text] [PDF]

				C. M. O'Carroll, S. J. Martin, J. Sandin, B. Frenguelli, and R. G.M. Morris Dopaminergic modulation of the persistence of one-trial hippocampus-dependent memory Learn. Mem., November 1, 2006; 13(6): 760 - 769. [Abstract] [Full Text] [PDF]

				Y. Loewenstein and H. S. Seung Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity PNAS, October 10, 2006; 103(41): 15224 - 15229. [Abstract] [Full Text] [PDF]

				O. Hikosaka, K. Nakamura, and H. Nakahara Basal Ganglia Orient Eyes to Reward J Neurophysiol, February 1, 2006; 95(2): 567 - 584. [Abstract] [Full Text] [PDF]

				M. Haruno and M. Kawato Different Neural Correlates of Reward Expectation and Reward Expectation Error in the Putamen and Caudate Nucleus During Stimulus-Action-Reward Association Learning J Neurophysiol, February 1, 2006; 95(2): 948 - 959. [Abstract] [Full Text] [PDF]

				B. S. Gutkin, S. Dehaene, and J.-P. Changeux A neurocomputational hypothesis for nicotine addiction PNAS, January 24, 2006; 103(4): 1106 - 1111. [Abstract] [Full Text] [PDF]

				R. A. Koene and M. E. Hasselmo An Integrate-and-fire Model of Prefrontal Cortex Neuronal Activity during Performance of Goal-directed Decision Making Cereb Cortex, December 1, 2005; 15(12): 1964 - 1981. [Abstract] [Full Text] [PDF]

				A. C. Kreitzer and R. C. Malenka Dopamine Modulation of State-Dependent Endocannabinoid Release and Long-Term Depression in the Striatum J. Neurosci., November 9, 2005; 25(45): 10537 - 10545. [Abstract] [Full Text] [PDF]

				K. Watanabe and O. Hikosaka Immediate Changes in Anticipatory Activity of Caudate Neurons Associated With Reversal of Position-Reward Contingency J Neurophysiol, September 1, 2005; 94(3): 1879 - 1887. [Abstract] [Full Text] [PDF]

				M. Giustizieri, G. Bernardi, N. B. Mercuri, and N. Berretta Distinct Mechanisms of Presynaptic Inhibition at GABAergic Synapses of the Rat Substantia Nigra Pars Compacta J Neurophysiol, September 1, 2005; 94(3): 1992 - 2003. [Abstract] [Full Text] [PDF]

				S. E. Hyman Addiction: A Disease of Learning and Memory Am J Psychiatry, August 1, 2005; 162(8): 1414 - 1422. [Abstract] [Full Text] [PDF]

				W.-X. Pan, R. Schmidt, J. R. Wickens, and B. I. Hyland Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network J. Neurosci., June 29, 2005; 25(26): 6235 - 6242. [Abstract] [Full Text] [PDF]

				M. Khamassi, L. Lacheze, B. Girard, A. Berthoz, and A. Guillot Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats Adaptive Behavior, June 1, 2005; 13(2): 131 - 148. [Abstract] [PDF]

				N. P. Rougier, D. C. Noelle, T. S. Braver, J. D. Cohen, and R. C. O'Reilly Prefrontal cortex and flexible cognitive control: Rules without symbols PNAS, May 17, 2005; 102(20): 7338 - 7343. [Abstract] [Full Text] [PDF]

				J. Riba, A. Rodriguez-Fornells, A. Morte, T. F. Munte, and M. J. Barbanoj Noradrenergic Stimulation Enhances Human Action Monitoring J. Neurosci., April 27, 2005; 25(17): 4370 - 4374. [Abstract] [Full Text] [PDF]

				E. Dommett, V. Coizet, C. D. Blaha, J. Martindale, V. Lefebvre, N. Walton, J. E. W. Mayhew, P. G. Overton, and P. Redgrave How Visual Stimuli Activate Dopaminergic Neurons at Short Latency Science, March 4, 2005; 307(5714): 1476 - 1479. [Abstract] [Full Text] [PDF]

				J. L. Krichmar, D. A. Nitz, J. A. Gally, and G. M. Edelman Characterizing functional hippocampal pathways in a brain-based device as it solves a spatial memory task PNAS, February 8, 2005; 102(6): 2111 - 2116. [Abstract] [Full Text] [PDF]

				A. D. Redish Addiction as a Computational Process Gone Awry Science, December 10, 2004; 306(5703): 1944 - 1947. [Abstract] [Full Text] [PDF]

				A. K. Seth, J. L. McKinstry, G. M. Edelman, and J. L. Krichmar Visual Binding Through Reentrant Connectivity and Dynamic Synchronization in a Brain-based Device Cereb Cortex, November 1, 2004; 14(11): 1185 - 1199. [Abstract] [Full Text] [PDF]

				F. G. Ashby and B. J. Spiering The Neurobiology of Category Learning Behav Cogn Neurosci Rev, June 1, 2004; 3(2): 101 - 113. [Abstract] [PDF]

				S. M. McClure, M. K. York, and P. R. Montague The Neural Substrates of Reward Processing in Humans: The Modern Role of fMRI Neuroscientist, June 1, 2004; 10(3): 260 - 268. [Abstract] [PDF]

				N. Schmitzer-Torbert and A. D. Redish Neuronal Activity in the Rodent Dorsal Striatum in Sequential Navigation: Separation of Spatial and Reward Responses on the Multiple T Task J Neurophysiol, May 1, 2004; 91(5): 2259 - 2272. [Abstract] [Full Text] [PDF]

				K. Kitano and T. Fukai Temporal Characteristics of the Predictive Synchronous Firing Modeled by Spike-Timing-Dependent Plasticity Learn. Mem., May 1, 2004; 11(3): 267 - 276. [Abstract] [Full Text] [PDF]

				J. O'Doherty, P. Dayan, J. Schultz, R. Deichmann, K. Friston, and R. J. Dolan Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning Science, April 16, 2004; 304(5669): 452 - 454. [Abstract] [Full Text] [PDF]

				M. Haruno, T. Kuroda, K. Doya, K. Toyama, M. Kimura, K. Samejima, H. Imamizu, and M. Kawato A Neural Correlate of Reward-Based Behavioral Learning in Caudate Nucleus: A Functional Magnetic Resonance Imaging Study of a Stochastic Decision Task J. Neurosci., February 18, 2004; 24(7): 1660 - 1665. [Abstract] [Full Text] [PDF]

				P. R. Montague, S. M. McClure, P. R. Baldwin, P. E. M. Phillips, E. A. Budygin, G. D. Stuber, M. R. Kilpatrick, and R. M. Wightman Dynamic Gain Control of Dopamine Delivery in Freely Moving Animals J. Neurosci., February 18, 2004; 24(7): 1754 - 1759. [Abstract] [Full Text] [PDF]

				R. Kawagoe, Y. Takikawa, and O. Hikosaka Reward-Predicting Activity of Dopamine and Caudate Neurons--A Possible Mechanism of Motivational Control of Saccadic Eye Movement J Neurophysiol, February 1, 2004; 91(2): 1013 - 1024. [Abstract] [Full Text] [PDF]

				T. Satoh, S. Nakai, T. Sato, and M. Kimura Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons J. Neurosci., October 29, 2003; 23(30): 9913 - 9923. [Abstract] [Full Text] [PDF]

				A. J. Gruber, S. A. Solla, D. J. Surmeier, and J. C. Houk Modulation of Striatal Single Units by Expected Reward: A Spiny Neuron Model Displaying Dopamine-Induced Bistability J Neurophysiol, August 1, 2003; 90(2): 1095 - 1114. [Abstract] [Full Text] [PDF]

				H. R. Dinse, P. Ragert, B. Pleger, P. Schwenkreis, and M. Tegenthoff Pharmacological Modulation of Perceptual Learning and Associated Cortical Reorganization Science, July 4, 2003; 301(5629): 91 - 94. [Abstract] [Full Text] [PDF]

				D. Durstewitz Self-Organizing Neural Integrator Predicts Interval Times through Climbing Activity J. Neurosci., June 15, 2003; 23(12): 5342 - 5353. [Abstract] [Full Text] [PDF]

				C. D. Fiorillo, P. N. Tobler, and W. Schultz Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons Science, March 21, 2003; 299(5614): 1898 - 1902. [Abstract] [Full Text] [PDF]

				S. Fenu, V. Bassareo, and G. Di Chiara A Role for Dopamine D1 Receptors of the Nucleus Accumbens Shell in Conditioned Taste Aversion Learning J. Neurosci., September 1, 2001; 21(17): 6897 - 6904. [Abstract] [Full Text] [PDF]

				W. Schultz Book Review: Reward Signaling by Dopamine Neurons Neuroscientist, August 1, 2001; 7(4): 293 - 302. [Abstract] [PDF]

				S. Ravel, P. Sardo, E. Legallet, and P. Apicella Reward Unpredictability inside and outside of a Task Context as a Determinant of the Responses of Tonically Active Neurons in the Monkey Striatum J. Neurosci., August 1, 2001; 21(15): 5730 - 5739. [Abstract] [Full Text] [PDF]

				G. S. Berns, S. M. McClure, G. Pagnoni, and P. R. Montague Predictability Modulates Human Brain Response to Reward J. Neurosci., April 15, 2001; 21(8): 2793 - 2798. [Abstract] [Full Text] [PDF]

				T. W. Troyer and A. J. Doupe An Associational Model of Birdsong Sensorimotor Learning II. Temporal Hierarchies and the Learning of Song Sequence J Neurophysiol, September 1, 2000; 84(3): 1224 - 1239. [Abstract] [Full Text] [PDF]

				M. J. Thomas, R. C. Malenka, and A. Bonci Modulation of Long-Term Depression by Dopamine in the Mesolimbic System J. Neurosci., August 1, 2000; 20(15): 5581 - 5586. [Abstract] [Full Text] [PDF]

				O. Sporns, N. Almassy, and G. M. Edelman Plasticity in Value Systems and its Role in Adaptive Behavior Adaptive Behavior, March 1, 2000; 8(2): 129 - 148. [Abstract] [PDF]

				W. Schultz The Reward Signal of Midbrain Dopamine Neurons Physiology, December 1, 1999; 14(6): 249 - 255. [Abstract] [Full Text] [PDF]

				J. Brown, D. Bullock, and S. Grossberg How the Basal Ganglia Use Parallel Excitatory and Inhibitory Learning Pathways to Selectively Respond to Unexpected Rewarding Cues J. Neurosci., December 1, 1999; 19(23): 10502 - 10511. [Abstract] [Full Text] [PDF]

				J.-M. Fellous Neuromodulatory Basis of Emotion Neuroscientist, September 1, 1999; 5(5): 283 - 294. [Abstract] [PDF]

				S. S. Kumar and D. S. Faber Plasticity of First-Order Sensory Synapses: Interactions between Homosynaptic Long-Term Potentiation and Heterosynaptically Evoked Dopaminergic Potentiation J. Neurosci., March 1, 1999; 19(5): 1620 - 1635. [Abstract] [Full Text] [PDF]

				M. Usher, J. D. Cohen, D. Servan-Schreiber, J. Rajkowski, and G. Aston-Jones The Role of Locus Coeruleus in the Regulation of Cognitive Performance Science, January 22, 1999; 283(5401): 549 - 554. [Abstract] [Full Text]

				P. L. Cerra and R. Bingham The adaptive nature of the human neurocognitive architecture: An alternative model PNAS, September 15, 1998; 95(19): 11290 - 11294. [Abstract] [Full Text] [PDF]

				W. Schultz Predictive Reward Signal of Dopamine Neurons J Neurophysiol, July 1, 1998; 80(1): 1 - 27. [Abstract] [Full Text] [PDF]

				D. Sulzer, M. P. Joyce, L. Lin, D. Geldwert, S. N. Haber, T. Hattori, and S. Rayport Dopamine Neurons Make Glutamatergic Synapses In Vitro J. Neurosci., June 15, 1998; 18(12): 4588 - 4602. [Abstract] [Full Text] [PDF]

				N. A. Otmakhova and J. E. Lisman D1/D5 Dopamine Receptors Inhibit Depotentiation at CA1 Synapses via cAMP-Dependent Mechanism J. Neurosci., February 15, 1998; 18(4): 1270 - 1279. [Abstract] [Full Text] [PDF]

				F Hellstern, R Malaka, and M Hammer Backward inhibitory learning in honeybees: a behavioral analysis of reinforcement processing. Learn. Mem., January 1, 1998; 4(5): 429 - 444. [Abstract] [PDF]

				S. Dehaene and J.-P. Changeux A hierarchical neuronal network for planning behavior PNAS, November 25, 1997; 94(24): 13293 - 13298. [Abstract] [Full Text] [PDF]

				G. S. Berns, J. D. Cohen, and M. A. Mintun Brain Regions Responsive to Novelty in the Absence of Awareness Science, May 23, 1997; 276(5316): 1272 - 1275. [Abstract] [Full Text]

				W. Schultz, P. Dayan, and P. R. Montague A Neural Substrate of Prediction and Reward Science, March 14, 1997; 275(5306): 1593 - 1599. [Abstract] [Full Text]

				N. A. Otmakhova and J. E. Lisman D1/D5 Dopamine Receptor Activation Increases the Magnitude of Early Long-Term Potentiation at CA1 Hippocampal Synapses J. Neurosci., December 1, 1996; 16(23): 7478 - 7486. [Abstract] [Full Text] [PDF]

				J.-P. Changeux, A. Bessis, J.-P. Bourgeois, P.-J. Corringer, A. Devillers-Thiery, J.-L. Eisele, M. Kerszberg, C. Lena, N. Le Novere, M. Picciotto, et al. Nicotinic Receptors and Brain Plasticity Cold Spring Harb Symp Quant Biol, January 1, 1996; 61(0): 343 - 362. [Abstract] [PDF]

Home  |   Search  |   Archive  |   Subscribe  |   Contact  |   Help