Article Figures & Data
Figures
- Figure 1.
Simulated visually guided saccade task and hypothesized corticobasal ganglia circuit. A, The simulated task. On each trial, a target appears pseudorandomly at the left or right, and a correct response is followed by either large or small reward. Contingency between the target location and the reward amount is fixed for each block (20–28 trials) and alternated across blocks. B, Hypothesized structure and operation of the corticobasal ganglia circuit. Target location is represented by a subset of a major subtype of corticostriatal neurons, CCS cells, that predominantly target striatonigral (direct pathway) dMSNs (Reiner et al., 2010) (left). These CCS cells activate dMSNs, which then inhibit the SNr and thereby disinhibit the SC to initiate a saccade. Meanwhile, the CCS cells also activate, via unidirectional projections (Morishima and Kawaguchi, 2006), another major subtype of corticostriatal neurons, CPn/PT cells, which predominantly target striatopallidal (indirect pathway) iMSNs (Reiner et al., 2010). These CPn/PT cells sustain their activity via strong facilitatory recurrent excitation (Morishima et al., 2011), whereas the activity of CCS cells declines because of their weaker and depressive recurrent excitation. When reward is obtained (right), dopamine neurons in the SNc receive excitatory inputs representing the value (amount) of the obtained reward from the neurons in the PPN. The SNc dopamine neurons also receive inhibitory inputs from the collaterals of SNr neurons, which are disinhibited by the iMSNs downstream of the still active CPn/PT cells. Given that the CPn/PT cells represent the location of the saccadic target and that the iMSNs represent the reward value predicted from that location, the SNr → SNc inhibition represents the reversed-sign predicted reward value. Thus, together with the input from the PPN, the dopamine neurons compute reward prediction error. The resulting phasic dopamine response induces proportional plastic changes of the synaptic strength between the CCS cells and dMSNs so that the activity of dMSNs can represent updated reward value predictions. At the synapses between the CPn/PT cells and iMSNs, sustained inputs from the CPn/PT cells cause adenosine accumulation and glutamate spillover. Phasic increase in dopamine then stimulates the signaling cascade downstream of the adenosine A2A receptors, leading to LTP, whereas phasic decrease in dopamine causes LTD through the signaling cascade downstream of mGluR5. In consequence, the activity of iMSNs can also represent updated reward value predictions. Notably, molecular mechanisms of plasticity induction, as well as structures between the striatum (Str) and the dopamine neurons, including the SNr, were not explicitly modeled, and our model can be also compatible with recent findings that the dopamine neurons are regulated by the LHb neurons, which are then driven by GPb (Fig. 7A,B). Also notably, although not explicitly illustrated here, we implicitly assumed that the striatum and the SNr are topographically organized and only a portion of neurons having the corresponding saccadic response fields, rather than an entire population, is activated in the task. C, Model equations used for the simulations shown in Figures 2, 4, and 5. xCCS and xCPn indicate the activity of the CCS cells and the CPn/PT cells that represent the left-target location (value of 1 is arbitrary; for details, see Materials and Methods), xdMSN and xiMSN indicate the population activity of dMSNs and iMSNs, respectively, xPPN indicates the activity of the PPN neurons, and xDA indicates the response of dopamine neurons compared with its baseline activity. w(n) indicates the strength of the connections between the CCS cells and dMSNs and those between the CPn/PT cells and iMSNs in the nth left-target trial [note that we assumed that CCS–dMSN and CPn/PT–iMSN connections are updated in the same way and so described their strength by the single variable w(n)]. f1 and f2 represent the input–output functions of dMSNs and iMSNs, respectively: these functions are assumed to be a threshold–linear function under the presence of tonic dopamine (Fig. 3). α represents the learning rate. γ represents the (relative) strength/efficacy of the direct pathway over the indirect pathway, and it corresponds to the time discount factor defined in the TD reinforcement learning algorithms (Fig. 6C). The gray parts in the equations will be modified in the elaborated model used for the simulations shown in Figure 6.
- Figure 2.
Activity of neurons and reaction times predicted from the model. A, Stimulus-induced response of striatal (Str) dMSNs downstream of the cortical CCS cells representing the left-target location, aligned at the switch of blocks from left-large reward to left-small reward and from left-small to left-large. B, Response of iMSNs at the timing of reward reception. It exhibits the same pattern as the stimulus-induced response of dMSNs because the activity of the CCS cells (upstream of the dMSNs) is presumably transmitted to the CPn/PT cells (upstream of the iMSNs) and sustained there (Fig. 1B, right). C, Response of SNc dopamine neurons at the timing of reward reception, which is presumably determined by the excitatory inputs from the PPN neurons representing the obtained reward value (large or small) and the net inhibitory inputs from the iMSNs (through the indirect pathway and the SNr) representing the predicted reward value; it thus represents reward prediction error. Notably, the pattern of dopaminergic neuronal response also matches the mathematical differential (in an approximate sense) of the response pattern of the MSNs (A, B), or, in turn, the pattern of the MSNs matches the mathematical integral of the dopaminergic neuronal pattern, because the corticostriatal (i.e., CCS–dMSN and CPn/PT–iMSN) connection strengths are assumed to be changed in proportion to the dopaminergic neuronal response. D, Experimentally observed dopaminergic neuronal response at the timing of reward (Takikawa et al., 2004) during a slightly different task, in which one of four (rather than 2) possible target locations was associated with reward in each block (thus, there were twice more unrewarded → unrewarded switches than rewarded → unrewarded switches for each location, and this possibly explains the smaller amplitude of the change of the activity at the beginning of the unrewarded blocks than the change at the beginning of the rewarded blocks). This panel was taken from Takikawa et al. (2004) with modification. E, Predicted saccadic reaction time, which was assumed to be quasi-inversely related to the stimulus-induced response of the dMSNs.
- Figure 3.
Presumed effects of dopamine receptor antagonists on the responsiveness of MSNs. A, The black line represents the relationship between the strength of inputs from CCS cells to dMSNs (horizontal axis) and the response of dMSNs (vertical axis) in the presence of tonic dopamine. Response of dMSNs to strong cortical (CCS) inputs, which would be mediated significantly by the NMDA receptor current that is suggested to be enhanced by D1R activation, was assumed to be attenuated by the application of D1 antagonist (arrow and gray line). B, The black line represents the relationship between the strength of the inputs from CPn/PT cells to iMSNs (horizontal axis) and the response of iMSNs (vertical axis) in the presence of tonic dopamine. Response of iMSNs to weak cortical (CPn/PT) inputs, which would normally be downregulated by D2R activation through the suppression of the AMPA and/or NMDA receptor-mediated excitation, was assumed to be enhanced by the application of D2 antagonist (arrow and gray line). Response of iMSNs to strong cortical (CPn/PT) inputs, which would normally be predominated by NMDA components, was assumed not to be enhanced by D2 antagonist, considering that strong cortical inputs would cause generation of a high amount of adenosine, which presumably blocks the suppressive effect of D2R activation on the NMDA receptor-mediated excitation. Notably, the actual relationship between the input strength and the output firing rate in the case without antagonists may entail saturating nonlinearity, but we assumed the threshold–linear function for simplicity. Str, Striatum.
- Figure 4.
Predicted effects of D1 antagonist on the neuronal activity and reaction time. A–D, The black and red lines indicate the simulated neuronal response (A–C) and the saccadic reaction time (D) in the conditions without and with D1 antagonist, respectively; these two are overlapped in B and C. Notations are the same as those in Figure 2A–C and E. The stimulus-induced response of dMSNs in large-reward blocks is reduced by D1 antagonist (A) because of the presumed suppressive effect of D1 antagonist when dMSNs receive strong inputs (Fig. 3A), whereas the activity of iMSNs is not affected (B) because iMSNs do not express D1Rs. The activity of dopamine neurons at the timing of reward reception is also not affected by D1 antagonist (C), because it is presumably determined by the inputs from the iMSNs-indirect pathway and the PPN; the activity of dMSNs is assumed to already decline at this timing (Fig. 1B, right) and thus have little effect (however, other dMSNs may have activity representing expectation of rewards in the following trials, and that possibility is examined in the elaborated model; Fig. 6). Given that the reaction time was assumed to be quasi-inversely related to the stimulus-induced response of dMSNs, D1 antagonist slows a reaction to stimulus in large-reward blocks (D). The inset shows the average reaction times for large- and small-reward blocks. E, Experimental results for the effect of D1 antagonist on the reaction time (Nakamura and Hikosaka, 2006). The black and red lines indicate the conditions without and with D1 antagonist, respectively, in the same manner as in D; this panel was taken from Hong and Hikosaka (2011). Str, Striatum.
- Figure 5.
Predicted effects of D2 antagonist. A–D, The black and blue lines indicate the conditions without and with D2 antagonist, respectively. Although D2 antagonist does not have direct effect on dMSNs, it causes an increase in the activity of iMSNs in small-reward blocks (B, blue open triangles) because of the presumed enhancement of their response to weak cortical inputs (Fig. 3B). Because the dopaminergic neuronal response to reward is presumably a subtraction of the iMSN activity from the PPN inputs representing the obtained reward value, it entails the same pattern of the effect of D2 antagonist but upside-down (C, blue open triangles). This decrease in the dopaminergic neuronal response causes a decrease in the corticostriatal (i.e., CCS–dMSN and CPn/PT– iMSN) connection strength through the presumed plasticity mechanism, resulting in an accumulative decrease in the input to, and the activity of, the dMSNs (A, blue filled triangles). The same accumulative decrease in the input also occurs in the iMSNs, gradually canceling out the direct facilitating effect of D2 antagonist, explaining why the increase in iMSN activity seen in the early part of small-reward blocks (B, blue open triangles) disappears later. Because the reaction time was assumed to be quasi-inversely related to the stimulus-induced response of dMSNs, application of D2 antagonist manifests as an increase in the reaction time in small-reward blocks (D). The inset shows the average reaction times for large- and small-reward blocks. E, Experimental results for the effect of D2 antagonist (Nakamura and Hikosaka, 2006); this panel was taken from Hong and Hikosaka (2011). Str, Striatum.
- Figure 6.
Consideration on reward expectation over multiple trials. A–C, Elaborated model for the saccade task. A, Presumed state transitions. B, It is assumed that, while the subject experiences these state transitions, at each state, a subset of CCS cells actively represents that state (current state) and a subset of CPn/PT cells actively represents the previous state (red and white circles in the figure indicate active and inactive subsets, respectively). C, Receiving inputs from the active CCS cells and CPn/PT cells, dMSNs and iMSNs presumably represent the predicted value of the current and previous states, respectively, under the condition in which the input–output functions f1 and f2 of MSNs operate in their suprathreshold linear regimens and are not affected by dopamine receptor antagonists. The dopamine neurons are assumed to receive net positive and negative effects from the dMSNs and iMSNs, respectively, and also reward-representing inputs from the PPN when reward is obtained, and thereby compute TD reward prediction error, which is then used to plastically modify the corticostriatal (i.e., CCS–dMSNs and CPn/PT–iMSNs) connection strengths. In our model, the relative strength/efficacy of the direct pathway over the indirect pathway (γ) represents the time discount factor appeared in the definition of the TD reward prediction error in reinforcement learning algorithms. D, E, Population activity of dMSNs and iMSNs during two successive trials and intertrial intervals between them, for two different values of γ (D, γ = 0.75; E, γ = 0.9). The black, red, and blue lines indicate the cases without antagonists, with D1 antagonist, and with D2 antagonist, respectively. The left and right columns show the average for a duration after a large-reward trial (intertrial interval + next trial) and those after a small-reward trial, respectively. The vertical solid lines (with light-yellow backgrounds) indicate the timing of target presentation in the large-reward (left column) or small-reward (right column) trials, and the vertical dashed lines (with light-yellow backgrounds) indicate target presentation in the next trials (for which large- and small-reward trials were mixed). Light-orange backgrounds indicate the timings for reward. The initial 10 trials in each block are not included in the average so as to see that the situation after learning of location–reward contingency essentially reaches a steady state. Both left-target and right-target trials were simulated, and the data with the first trials (in the left of the panels) being left-target trials were shown here. F, G, Response of dMSNs at the timing of target presentation (top row), responses of iMSNs and the dopamine neurons at the timing of reward (middle 2 rows), and the reaction time (bottom row) in simulations with the two different values of γ (F, γ = 0.75; G, γ = 0.9). The black lines indicate the cases without dopamine receptor antagonists, whereas the red and blue lines indicate the cases with D1 and D2 antagonist, respectively. Notations are the same as those in Figure 4 and 5. Left-target trials were shown in these panels (the model is left-right symmetric, and “left” and “right” are just labeling).
- Figure 7.
Putative pathways conveying positive and negative reward signals, with the common corticostriatal mechanism for computing the TD of reward predictions. A, Pathway that conveys positive reward signals and computes reward prediction error in the SNc (Morita et al., 2012), which is then transmitted to the LHb with a sign reversal. The arrows, filled squares, and open squares indicate excitation, inhibition, and disinhibition, respectively. B, Pathway that conveys negative reward signals, which are transmitted from the GPb to the LHb, the RMTg (also referred to as the tVTA; Bourdy and Barrot, 2012), and then to the SNc and the VTA with a sign reversal. The model used for the simulations in the present study does not explicitly describe structures between the striatum (Str) and the SNc/VTA (see Materials and Methods), and it could thus simultaneously represent both of these two pathways, except for suggested mutually inhibitory interactions between the SNc and the LHb.
- Figure 8.
Reward prediction error computation in parallel with action selection and/or execution in the corticobasal ganglia circuit. A–D, Four different (extreme) possibilities are described. A, An action plan selected at the outside of the circuit considered here is loaded onto a subset of CCS cells and then sent for execution through the direct pathway–thalamic loop and the CCS → CPn/PT connections. In the meantime, the dopamine neurons compute R(ti) + γQ(A(ti)) − Q(A(ti − 1)), which corresponds with reward prediction error defined in the SARSA algorithm (under the condition in which the input–output functions f1 and f2 of MSNs operate in their suprathreshold linear regimens and are not affected by dopamine receptor antagonists; the same applied to all the cases). B, Candidates of action plans are loaded onto different subsets of CCS cells, and an action plan is selected in the CCS–dMSNs pathway, through feedforward inhibition (and possibly also lateral and feedback inhibition), with the probability depending on its predicted value, implementing a soft-max operation. The selected action plan is then sent for execution without perturbation at the stage of CPn/PT cells, while the dopamine neurons compute the same SARSA error signal as the case of A. C, Candidates of action plans are loaded onto different subsets of CCS cells in the same manner as in B, but the CCS–dMSNs pathway implements the (hard) max operation rather than a soft-max operation assumed in B. Consequently, an action plan that currently has the maximum value is selected at the stage of dMSNs after a brief initial transient phase, and thereby the dopamine neurons compute R(ti) + γ max{Q(Acand(ti))} − Q(A(ti − 1)), which corresponds with reward prediction error defined in Q-learning. Meanwhile, receiving the initial dMSN–direct pathway–thalamic inputs that represent predicted values of each action plan (see Ea), CPn/PT cells begin to compete with each other, and a plan of the action that will eventually be executed is selected in a soft-max manner; this selected (to-be-executed) action plan is not necessarily the same as the one selected by dMSNs (i.e., the one with the maximum value), because the two selection processes presumably proceed rather separately. D, Candidates of action plans are again loaded onto different subsets of CCS cells, but the CCS–dMSN pathway implements neither a soft-max nor the max operation. Instead, values of the action plan candidates are computed in the dMSNs and sent to the dopamine neurons as they are. The dopamine neurons therefore compute R(ti) + γ mean{Q(Acand(ti))} − Q(A(ti − 1)), which corresponds with the TD error for the expected SARSA learning (van Seijen et al., 2009) (or “sum” instead of “mean”; but these two could become equivalent by changing γ). Receiving the thalamic feedback inputs, the CPn/PT cells compete to select an action plan that is to be executed, implementing a soft-max operation. E, Schematic illustration of the presumed operation of the circuit, with a consideration that action plan selection and action execution are serially operating separable (in an approximate sense) processes (a → b). Among two action-plan candidates (“plan of action 1” and “plan of action 2”), the plan of action 2 is selected through the presumed Q-learning-compatible mechanism, and the corresponding subset of CPn/PT cells becomes selectively activated. These activated CPn/PT cells representing the plan of action 2 then activate a subset of CCS cells that represent execution of action 2, via presumably hard-wired intracortical (possibly inter-areal) connections (c). Subsequently, those CCS cells activate CPn/PT cells that also represents execution of action 2, via both the direct intracortical connections and the CCS → dMSNs → SNr → thalamus → CPn/PT loop pathway, and thereby action 2 is initiated (d). Those CPn/PT cells will keep their activity via strong recurrent excitation and send inputs to iMSNs so as to provide the dopamine neurons with a negative-signed signal of the value of the executed action via the indirect pathway (e). Notably, reaction time corresponds to the duration between c and d and is thus modulated by the activity of dMSNs that receive inputs from the CCS cells representing execution of action 2 (c). Ctx, Cortex; Str, striatum.
- Figure 9.
Simulation of a task involving action selection considered by Roesch et al. (2007) and Takahashi et al. (2011): explanation of the neural and behavioral data and the predictions. A, Left, Presumed state transitions during individual task trials (similar to the one proposed in a model developed by Takahashi et al., 2011). S1, S2, … represent subjects' states that are defined by external events (i.e., cue onset, cue offset, reward bolus), their own movements, or internally (i.e., internal state), and A1, A2, … represent option(s) that can be taken at each state (e.g., “plan/prepare for moving to the left,” “move to the left,” or “keep rest”). At the beginning of a trial (at the leftmost in the diagram), the subject is presented with one of three odor cues, two of which (cues 1 and 3) indicated that reward will be given if the animal entered either the left-side or right-side well, respectively (forced-choice trial), whereas the remaining cue (cue 2) indicated that the animal will be rewarded in both of the wells (free-choice trial). In any case, the amount of reward, either small (1 bolus of sucrose solution at S8 or S9) or large (2 boluses, with the second bolus given successively at S10 or S11), was determined according to the predetermined direction-amount contingency that was fixed during a block (at least 60 trials in the experiments; 120 trials in our simulations) and then reversed. Forced-choice trials and free-choice trials were pseudorandomly intermingled. In our model, when the subject enters each state, subset(s) of CCS cells in the OFC are assumed to represent combination(s) of that state and options that can be taken there [e.g., at state S2, a subset represents “S2 − A2” (plan/prepare for moving to the left) and another represents “S2 − A3” (plan/prepare for moving to the right)]. Subsequently, the circuit is assumed to operate as we considered in Figure 8C, and one of the options is selected (and executed if it is a movement); if there is just a single option, it is selected (and executed). In the meantime, the dopamine neurons compute TD reward prediction error (regardless of whether there are multiple options or a single option), according to which the strengths of corticostriatal (i.e., CCS–dMSNs and CPn/PT–iMSNs) connections are plastically modified. Notably, the three time points for S2 (cue onset), S4/S5 (cue offset), and S6/S7 correspond to ti, ti + 1, and ti + 2 in Figure 8E, and thus reaction time is presumably modulated by the activity of dMSNs at cue offset. Right, Assumed soft-max function for choice. B, Dopamine neuronal response sorted by different experimental conditions: the top and bottom rows show the results of the experiments (Roesch et al., 2007) and our simulations, respectively. The vertical black line indicates the timing of cue onset. Left, Dashed and solid lines indicate the average response of dopamine neurons in the first and last 10 forced-choice select-large (green) or select-small (orange) trials in the blocks, respectively. Middle and Right, Green and orange lines indicate the average response of dopamine neurons in forced-choice (middle) or free-choice (right) select-large and select-small trials, respectively. To control for learning and also for the possibility that disadvantageous choices might be more often early in the block, only trials after the ratio of selecting more valuable option exceeds 50% were included, and each free-choice trial was paired with the immediately preceding and following forced-choice trial of the same reward amount [as done in the original work (Roesch et al., 2007)]. C, Population activity of dMSNs (top row) and iMSNs (bottom row) predicted from our model, sorted by the same experimental conditions as considered in B. D, The impact of reward amount on choice behavior in free-choice trials. Line graphs show the ratio (percentage) of choices before and after the switch from big reward to small reward (indicated by the vertical line), and the inset bar graphs show the percentage choice for large versus small reward across all free-choice trials. The left and right panels show the results of the experiments (Roesch et al., 2007) and our simulations, respectively. E, Average reaction time in the forced-choice (left column) and free-choice (right column) trials included in the data shown in the middle and right panels of B, respectively. Top row shows the experimental results (Roesch et al., 2007), and the bottom row shows our simulation results. F, Prediction from our model about the effects of D1 (left) or D2 (right) antagonist on the reaction time in forced-choice trials (top) and free-choice trials (bottom). G, Prediction from our model about the effects of optogenetic stimulation of dMSNs (left) or iMSNs (right) on choice behavior. In a new set of simulations, in one of the two wells (left and right), virtual optogenetic stimulation was applied to either dMSNs or iMSNs coincidently with reward (at S8 or S9), in addition to giving an extra bolus of reward at the subsequent timing in both of the wells; the contingency between the stimulation on/off and the location of the well was fixed for a block and alternated across blocks. The bar graphs show the ratio (percentage) of choices for with versus without optogenetic stimulation across all free-choice trials, either without dopamine receptor antagonist (light gray bars) or with both D1 and D2 antagonists (dark gray bars). The top rows in B and E and the left panels of D were taken from Roesch et al. (2007).
Tables
- Table 1.
Functions of the direct and indirect pathways of the basal ganglia: a popular view versus a new view suggested from our model
Direct pathway Indirect pathway Popular view Operate for Upcoming action Upcoming action How to operate Execute (“go”) Inhibit (“no go”) If stimulated Action execution/facilitation Action inhibition Our model Operate for Upcoming action Ongoing/executed action How to operate Execute (“go”) Terminate If stimulated Action execution/facilitation and positive reward prediction error Action termination and negative reward prediction error
