Phasic dopamine signals: from subjective reward value to formal economic utility

https://doi.org/10.1016/j.cobeha.2015.09.006Get rights and content

Highlights

  • Dopamine neurons code subjective value of different rewards according to behavioral choices.

  • Dopamine neuronal and voltammetry responses follow individual risk preferences.

  • Dopamine neuronal and voltammetry responses follow temporal reward discounting.

  • Dopamine voltammetry responses are sensitive to effort cost.

  • Dopamine neuronal responses code formal economic utility.

Although rewards are physical stimuli and objects, their value for survival and reproduction is subjective. The phasic, neurophysiological and voltammetric dopamine reward prediction error response signals subjective reward value. The signal incorporates crucial reward aspects such as amount, probability, type, risk, delay and effort. Differences of dopamine release dynamics with temporal delay and effort in rodents may derive from methodological issues and require further study. Recent designs using concepts and behavioral tools from experimental economics allow to formally characterize the subjective value signal as economic utility and thus to establish a neuronal value function. With these properties, the dopamine response constitutes a utility prediction error signal.

Introduction

The function of reward is derived from the biological needs for nutritional and other substances and reproduction. Thus, rewards have specific value for individual survival and gene propagation. Although rewards have physical aspects that are detected by sensory receptors, there are no specific receptors for the typically polysensory rewards, and their value needs to be inferred from eliciting preferences in behavioral choices. Furthermore, reward value depends on the organism's momentary requirements. Satiation induced by a meal reduces the value of foods but may render liquids such as digestive drinks more attractive. Thus, value is subjective and constructed by the brain; it cannot be estimated entirely from the physical parameters and sensory properties of the rewards. The usual way to estimate subjective value in animals involves behavioral measures, including break points in fixed ratio schedules, preferences in binary choices and psychophysical indifference points against a common reference reward (subjective equivalents). Subjective value estimated in these ways is expressed in physical measures of break points, choice frequency or reference reward amount (e.g. ml of juice or numbers of pellets). By contrast, a more general, and theoretically well defined, measure for subjective value is formal economic utility, which constitutes a mathematical characterization of reward preferences and provides an internal metric of subjective value (sometimes called util) [1]. Individuals have the best chance to survive by preferring rewards with the highest subjective value. Economic theory formalizes this idea with axioms defining the conditions for utility maximization [2].

Maximization of subjective value and utility requires decision mechanisms in which inputs from neuronal value signals compete with each other, and only the option with the highest value gets selected. Neuronal reward signals that serve as appropriate inputs to competitive decision mechanisms should process subjective value or, in their best defined form, economic utility, in a monotonic but usually nonlinear relationship to objective value.

This review describes the neuronal coding of subjective value and formal economic utility in one of the brain's prominent reward systems, the dopamine neurons. We review both the electrophysiological responses of midbrain dopamine neurons and the voltametrically assessed dopamine concentration changes in axonal terminal areas in nucleus accumbens. We also address recent issues concerning voltammetric changes reflecting subjective value in rats.

Section snippets

Concepts and behaviour

How can we maximize subjective value when choosing between apples and oranges? These objects contain important substances for bodily functions, like glucose and water, but their precise contents are difficult to quantify. As different rewards often have no common physical unit, one can assign a ‘common currency’ value to one particular object, called ‘numeraire’ in economic theory. Behavioral preferences serve to estimate the subjective values of all other objects relative to this common

Concepts

Rewards are inherently risky. The terms of risk avoidance and risk seeking refer to individual psychological tendencies of hating or loving risk and characterize the influence of risk on subjective reward value. Risk avoiders value risky outcomes lower than safe outcomes with same objective value, whereas risk seekers do the opposite. Risk is distinct from probability, as it increases up to probabilities of p = 0.5 and then declines again. The most simple and best controlled behavioral risk test

Concepts and behaviour

The subjective value of rewards decays with the delay between a stimulus or action and the reward delivery. This temporal discounting may have its biological origin in the physical decay of many nutrient rewards. Temporal discounting applies to rewards in general, even when they remain physically unchanged and do not decay. Intertemporal choices between a variable early and a set late reward serve to psychophysically assess temporal discounting. The subjective value of the late reward is

Concepts and behaviour

Caloric rewards provide energy for body functions. However, reward acquisition often involves effort, which amounts to energy expenditure. The gain from reward is reduced by the loss. This notion can be extended to all rewards; effort is considered an economic cost that should be subtracted from income value (however, subtracting cost from income does not define formal economic utility as viewed by economists [2, 33, 34]). Monkeys show longer reaction times, more task errors and lower task

Concepts

The quantification of subjective reward value relative to a numeraire employs an objective scale. The £20,000 price of car is an objective money amount, even though my preference for that car over another car reveals my subjective value. By contrast, formal economic utility advances by one crucial step in providing a mathematical function of objective value u (x). Knowing such a function allows to determine the subjective value for goods solely based on their objective amounts, without every

Conflict of interest statement

Nothing declared.

References and recommended reading

Papers of particular interest, published within the period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

Acknowledgements

Our work has been supported by Wellcome Trust (WS: 058365, 093270, 095495), European Research Council (WS: ERC, 293549), NIH Conte Center at Caltech (WS: P50MH094258) and NIH (RMC: DA034021; RMW: DA010900).

References (56)

  • D.L. Robinson et al.

    Monitoring rapid chemical communication in the brain

    Chem Rev

    (2008)
  • S.R. Jones et al.

    Dopamine neuronal transport kinetics and effects of amphetamine

    J Neurochem

    (1999)
  • N.T. Rodeberg et al.

    Construction of training sets for valid calibration of in vivo cyclic voltammetric data by principal component analysis

    Anal Chem

    (2015)
  • J.H. Kagel et al.

    Economic Choice Theory: An Experimental Analysis of Animal Behavior

    (1995)
  • J. von Neumann et al.

    The Theory of Games and Economic Behavior

    (1944)
  • C. Padoa-Schioppa et al.

    Neurons in the orbitofrontal cortex encode economic value

    Nature

    (2006)
  • A. Lak et al.

    Dopamine prediction error responses integrate subjective value from different reward dimensions

    Proc Natl Acad Sci U S A

    (2014)
  • C.D. Fiorillo et al.

    Discrete coding of reward probability and uncertainty by dopamine neurons

    Science

    (2003)
  • G. Morris et al.

    Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons

    Neuron

    (2004)
  • J.J. Day et al.

    Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens

    Nat Neurosci

    (2007)
  • J.A. Sugam et al.

    Phasic nucleus accumbens dopamine encodes risk-based decision-making behavior

    Biol Psychiatry

    (2012)
  • M.P. Saddoris et al.

    Mesolimbic dopamine dynamically tracks, and is causally linked to, discrete aspects of value-based decision making

    Biol Psychiatry

    (2015)
  • J. Mirenowicz et al.

    Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli

    Nature

    (1996)
  • C.D. Fiorillo et al.

    Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli

    J Neurosci

    (2013)
  • C.D. Fiorillo

    Two dimensions of value: dopamine neurons represent reward but not aversiveness

    Science

    (2013)
  • M. Rothschild et al.

    Increasing risk: I. A definition

    J Econ Theory

    (1970)
  • A.N. McCoy et al.

    Risk-sensitive neurons in macaque posterior cingulate cortex

    Nat Neurosci

    (2005)
  • M. O’Neill et al.

    Coding of reward risk distinct from reward value by orbitofrontal neurons

    Neuron

    (2010)
  • Cited by (0)

    View full text