Dopaminergic Control of Motivation and Reinforcement Learning: A Closed-Circuit Account for Reward-Oriented Behavior

Simulated saccade task.

We first simulated a visually guided saccade task (Fig. 1A) used in an experimental study (Nakamura and Hikosaka, 2006). In the experiment, a visual stimulus (saccadic target) appeared at the left or the right of the screen on each trial, and the subject (monkey) was required to make a saccade toward the target. If the subject made a correct response, a liquid reward was given after 100 ms of fixation at the target. There were two kinds of reward amount, “large” (0.4 ml) or “small” (0.05 or 0 ml), each of which was associated with either left or right target; contingency between the target location and the reward amount was fixed for an individual task block consisting of 20–28 trials, and “left-large” and “left-small” blocks were switched alternately without a preceding cue for a switch. The target location (left and right) was pseudorandomly determined in each trial. We incorporated these features into our simulations, although there are several points that differ from the experimental study as we describe below. We constructed two models: (1) a simpler one, which modeled neuronal activity only at the timings of target presentation and reward reception, and (2) an elaborated one, which modeled intertrial intervals as well. In both of the models, we did not directly model the amount of reward or neural processes for reward sensation/consumption but instead assumed two different levels of reward-representing inputs to the dopamine neurons for the large- and small-reward conditions (see below). Also, we did not model the two different amounts of reward for the small-reward condition (0.05 or 0 ml); in the experiment (Nakamura and Hikosaka, 2006), trials with 0.05 ml and those with 0 ml were not analyzed separately. With the simpler model, we simulated only the trials with a left target for simplicity and in the same manner as the previous modeling study (Hong and Hikosaka, 2011). With the elaborated model, we simulated both left-target and right-target trials, although only the results for the left-target trials were shown in the figures unless otherwise mentioned. In the following, we first describe the details of the simpler model and, thereafter, those about the elaborated one.

Simulated neural circuit for reward-oriented saccade.

A series of experimental studies have revealed that the cortex, basal ganglia, and the dopamine neurons in the substantia nigra pars compacta (SNc) play essential roles in learning and execution of reward-associated saccade tasks (Kawagoe et al., 2004; Takikawa et al., 2004; Hikosaka et al., 2006; Nakamura and Hikosaka, 2006). We constructed a computational model of the corticobasal ganglia–SNc circuit according to the conjecture (Morita et al., 2012) derived from recent anatomical and physiological findings (Fig. 1B). There are two distinct types of corticostriatal neurons, the CCS cells and the CPn/PT cells, and there exist unidirectional projections from the CCS cells to the CPn/PT cells (Morishima and Kawaguchi, 2006) and strong facilitatory recurrent excitation only among the CPn/PT cells (Morishima et al., 2011). The CCS cells and the CPn/PT cells predominantly project to the striatal dMSNs and iMSNs, respectively (Lei et al., 2004; Reiner et al., 2010), which presumably upregulate and downregulate the dopamine neurons in the SNc (Aggarwal et al., 2012; Morita et al., 2012) via the substantia nigra pars reticulata (SNr) (Tepper and Lee, 2007). The SNc dopamine neurons receive significant inputs also from other structures (Watabe-Uchida et al., 2012), including excitation from the pedunculopontine nucleus (PPN) (Mena-Segovia et al., 2004) and inhibition from the striatal striosomes (Gerfen et al., 1985; Paladini et al., 1999; Fujiyama et al., 2011; Watabe-Uchida et al., 2012). The former input could convey information of actually obtained reward, because it has been shown (Okada et al., 2009) that a population of neurons in the PPN represents such information. We did not incorporate the latter, striosomal input to the dopamine neurons or many other known connections. Exploring how these unincorporated connections can be related to the circuit that we modeled is an important future issue.

We assumed that phasic dopamine response induces proportional changes of the strengths of connections between CCS cells and dMSNs and those between CPn/PT cells and iMSNs via plasticity mechanisms; in the case of synapses on the iMSNs, presumably together with adenosine acting on the A_2A receptors and/or glutamate acting on the metabotropic glutamate receptor 5 (mGluR5) that are assumed to be synaptically accumulated during sustained inputs from CPn/PT cells (see below). Phasic dopamine response may also gradually change the level of tonic dopamine (cf. Niv et al., 2007). However, in the saccade task that we simulated (Nakamura and Hikosaka, 2006), as well as the other task that we also simulated as we describe below (Roesch et al., 2007; Takahashi et al., 2011), such changes across blocks would not be very significant, because large- and small-reward trials were intermingled and also because the frequency of phasic release (on reward reception) is relatively low (each task trial takes >4 s) and seems not to vary much compared with self-paced free-operant tasks. We thus did not consider changes in tonic dopamine; nevertheless, we did consider the effects of tonic dopamine on the responsiveness of dMSNs/iMSNs and their blockade by antagonists as we explain below.

Because the experimental results (Kawagoe et al., 2004; Hikosaka et al., 2006; Nakamura and Hikosaka, 2006) suggest that the caudate is especially involved in the learning and execution of reward-associated saccade tasks, “striatum” in our models for the saccade task basically refers to the caudate. Likewise, “cortex” refers to the areas that project to the caudate and exhibit saccade-related activity, e.g., the frontal eye field (FEF), and we assumed that there exist CCS cells and CPn/PT cells in those areas. However, other striatal regions could potentially operate in similar ways given the suggested common architecture across the entire striatum (Pennartz et al., 2011). There have been various types of neurons in the striatum (Hikosaka et al., 2006), and not all of them are incorporated into our model; in particular, neurons showing block-indicating activity before the presentation of the target (Watanabe and Hikosaka, 2005; Hikosaka et al., 2006) were not incorporated.

We assumed the following set of equations (Fig. 1C) describing the time (t)-dependent neuronal activity of cortical CCS cells and CPn/PT cells that represent the left-target location [x_CCS(t) and x_CPn(t)], the activity of striatal dMSNs and iMSNs [x_dMSN(t) and x_iMSN(t)], the activity of the PPN neurons that represent obtained reward [x_PPN(t)], the response of SNc dopamine neurons compared with its baseline activity [x_DA(t)], and the strength of connections between the CCS cells and the dMSNs and those between the CPn/PT cells and the iMSNs [w(n), where n represents the index of left-target trials, i.e., w(n) represents the strength at the nth left-target trial]. Notably, w(n) represents both the CCS–dMSNs and CPn/PT–iMSNs connection strengths, because we assumed that the synaptic strength between CCS cells and dMSNs and the strength between CPn/PT cells and iMSNs are modified in effect similarly (see below). At the timing of target presentation in the nth left-target trial [t = t(n)] in a given block, The CCS cells that represent the left-target location become active in response to a presentation of the left target. Notably, it is arbitrary to set this activity to what value; we set this to 1 for simplicity, but we can equivalently set this to 10, for example, and scale down w(n) and also α (learning rate; see below) to 110 of the original values: The CPn/PT cells that represent the left-target location presumably do not show immediate response to a presentation of the target but rather gradually become activated by the CCS cells via the unidirectional connections (Morishima and Kawaguchi, 2006), and so we assumed that their activity is 0 at time t(n). More precisely, however, other CPn/PT cells could represent a state preceding the target presentation and drive iMSNs. This point was not incorporated into the model considered here for simplicity but was taken into account in our elaborated model described below: w(n) × x_CCS(t(n)) (= w(n)) represents the inputs from the CCS cells to the dMSNs. f₁ is the input–output function of the dMSNs under the presence of tonic dopamine (see below), and x_dMSN(t(n)) presumably represents the reward value (reward amount) predicted from the left-target location under the condition in which f₁ operates in the suprathreshold linear regimen and is not affected by D₁ antagonist: f₂ is the input–output function of the iMSNs under the presence of tonic dopamine (see below), which is assumed to take 0 when input is 0: This indicates that the visual stimulus (saccadic target) itself is neutral (not rewarding): This relationship was derived from the hypothesized upstream circuitry of dopamine neurons (Fig. 1B) (Morita et al., 2012). γ represents the (relative) strength/efficacy of the direct pathway over the indirect pathway (its value does not affect the results presented in Figs. 2, 4, 5). Notably, it has been suggested that narrower dynamic range of negative dopamine response than positive response can significantly affect learning (Potjans et al., 2011), but such asymmetry was not considered in the present study; considering it in our model could be a good theme for future research.

At the timing of reward reception in the nth left-target trial [t = t(n) + τ, in which τ includes a reaction time, period for fixation at target (100 ms was required in the experiment) (Nakamura and Hikosaka, 2006) and time for liquid reward delivery and sensation/consumption], The target-induced activity of the CCS cells is assumed to decline, because the recurrent excitation among CCS cells is relatively weak and entails short-term synaptic depression (Morishima et al., 2011). Therefore, we assumed x_CCS(t(n) + τ) = 0. More precisely, however, other CCS cells could represent a new state at time t(n) + τ and drive dMSNs. This point was not incorporated into the model considered here for simplicity but was taken into account in our elaborated model described below: We assumed this because the CPn/PT cells presumably become active by the input from the CCS cells via the unidirectional connections (Morishima and Kawaguchi, 2006) and then sustain activity via strong recurrent excitation (Morishima et al., 2011): f₁ (as well as f₂) is assumed to take 0 when input is 0 (see below). This is equal to x_dMSN(t(n)), thus representing the reward value (reward amount) predicted from the left-target location, under the condition in which f₁ and f₂ are in their suprathreshold linear regimens and are not affected by dopamine receptor antagonists (see below): The 5 and 10 values correspond to the small- and large-reward conditions, respectively: This relationship (Morita et al., 2012) indicates that the dopaminergic neuronal response represents reward prediction error or, more specifically, TD error defined in the TD learning (Sutton and Barto, 1998), and the parameter γ, representing the (relative) strength/efficacy of the direct pathway over the indirect pathway, corresponds to the degree of time discount for future rewards, namely, time discount factor.

In reference to empirical findings (Kawagoe et al., 2004) and the results of the previous modeling study (Hong and Hikosaka, 2011), we assumed a simple quasi-inverse relationship between the saccadic reaction time [RT(n)] and the stimulus-induced response of the dMSNs: where C₁ and C₂ are constants and were set to 3000 and 6, respectively, in the simulations shown in Figures 2, 4, and 5.

Notably, as shown in the above, we did not explicitly model structures between the striatum and the dopamine neurons, namely, the SNr, the external segment of the GP (GPe), and the subthalamic nucleus. For one thing, this was for simplicity; however, with such abstraction, our model could be also compatible with recent findings (Matsumoto and Hikosaka, 2007; Hong and Hikosaka, 2008; Hong et al., 2011; Shabel et al., 2012) that the dopamine neurons, in the SNc and the ventral tegmental area (VTA), are regulated by the lateral habenula (LHb) neurons, which are then driven by the border region of GP (GPb). Specifically, we assumed that the same corticostriatal circuit mechanism for computing the TD of values of the current and previous states/actions (Morita et al., 2012) can operate with two different circuits connecting to the dopamine neurons: (1) one via the SNr (see Fig. 7A) and (2) the other via the GPb and the LHb (see Fig. 7B) (see Results, Reward prediction error computation in parallel with action selection and/or execution).

Presumed effects of dopamine receptor antagonists.

Existence of tonic dopamine is considered to affect the relationship between the strength of cortical inputs and the firing rate of MSNs, in different ways for dMSNs and iMSNs, and application of D₁- or D₂-type dopamine receptor antagonist presumably “demodulates” such a tonic dopamine-modulated relationship to a certain extent. For simplicity, we assumed the following threshold–linear function for the input–output functions of dMSNs and iMSNs under the condition with a presumed certain level of tonic dopamine and without antagonists (see Fig. 3A,B, black lines): where I is input to MSNs, and θ represents a threshold level, which was set to 5 in the simulations. The actual relationship between the input strength and the output firing rate under this condition may entail a certain degree of saturating nonlinearity, but we assumed the above threshold–linear function for simplicity.

Dopamine is known to upregulate the activity of dMSNs via the activation of D₁Rs and downregulate the activity of iMSNs via the activation of D₂Rs (Gerfen and Surmeier, 2011), although exact ways of modulation in vivo remain elusive. As for the D₁Rs, it has been suggested that their activation enhances the NMDA current more significantly than the AMPA current (Levine et al., 1996; Flores-Hernández et al., 2002; Moyer et al., 2007). Because the NMDA receptor has voltage dependence (Schiller and Schiller, 2001), the enhancement of the NMDA current would become more prominent when dMSNs receive stronger glutamatergic (cortical) inputs. Therefore, we assumed that the response of dMSNs to strong (but not weak) inputs, which would normally be enhanced by tonic dopamine, is attenuated by D₁ antagonist (see Fig. 3A). In the presence of D₁ antagonist (see Fig. 3A, gray), Conversely, regarding the D₂Rs, it was originally suggested that their activation suppresses the AMPA current more significantly than the NMDA current (Levine et al., 1996; Flores-Hernández et al., 2002; Hernández-Echeagaray et al., 2004; Moyer et al., 2007), but later studies (Azdad et al., 2009; Higley and Sabatini, 2010) have shown that D₂R activation does reduce the NMDA receptor-mediated excitation [and in fact, Higley and Sabatini (2010) has shown that D₂R activation did not affect non-NMDA synaptic currents, apparently contradictory to the results mentioned above]. At the same time, however, it was also shown that, if A_2A adenosine receptors, which are predominantly expressed in iMSNs, are simultaneously activated, such a suppression of the NMDA receptor-mediated excitation by D₂R activation can be fully blocked (Azdad et al., 2009; Higley and Sabatini, 2010). It has been suggested (Cunha, 2001; Schiffmann et al., 2007) that the formation of extracellular adenosine results from the action of ecto-nucleotidases on ATP released with neurotransmitters, including glutamate, and thus the synaptic pool of adenosine reflects neuronal firing. Together, it is conceivable that the activation of D₂Rs reduces the AMPA and/or NMDA receptor-mediated excitation when iMSNs receive weak cortical inputs and there exist a moderate amount of adenosine, whereas D₂R activation could still reduce the AMPA current but would never reduce the NMDA receptor-mediated excitation when iMSNs receive strong cortical inputs and there exist a high amount of adenosine. Based on these considerations, we assumed that the response of iMSNs to weak (but not strong) inputs, which would normally be suppressed by tonic dopamine, is enhanced by D₂ antagonist (see Fig. 3B). In the presence of D₂ antagonist (see Fig. 3B, gray), Notably, in our simulations shown in Figures 2, 4, and 5, I = w(n) [i.e., input to dMSNs at time t(n) and input to iMSNs at time t(n) + τ] was always, except for the initial transient, near the central part of the input range shown in Figure 3, A and B, and therefore both D₁ and D₂ antagonists were able to properly cause their effects. Changing the assumptions about the effects of the antagonists on the input–output functions, or the magnitude of rewards, can significantly alter the main results of this study. Nevertheless, given the observed cortical neural representation of relative (rather than absolute) preference of rewards (Tremblay and Schultz, 1999), matching between the range of reward amounts relevant in a given situation (task) and the dynamic range of value-representing neurons in the brain, including the MSNs, could actually be achieved via certain mechanisms.

Presumed dopamine-dependent corticostriatal plasticity.

We assumed that the reward value (reward amount) predicted from the left-target location is stored in the strength of corticostriatal synapses, in the manner as described above, and it is updated through plasticity mechanisms depending on phasic dopamine response after reward reception as in the following equation: The parameter α represents the learning rate, and it was set to 0.75, which is close to an empirically estimated value (0.7) in a study using a different task (Bayer and Glimcher, 2005). We checked that the central features of our simulation results shown in Figures 2, 4, and 5 were successfully reproduced even if the learning rate was set to 0.6 or 0.9 (data not shown). We assumed the same form of dopamine-dependent changes of corticostriatal inputs as above for both the dMSNs and iMSNs. It might appear odd to make such an assumption, given the distinct properties of the D₁Rs and D₂Rs that are separately expressed in dMSNs and iMSNs (Gerfen and Surmeier, 2011). However, according to our hypothesis (Fig. 1B), the synapses on dMSNs and those on iMSNs are in fact under quite different conditions at time t(n) + τ. Specifically, whereas the synapses on iMSNs continuously receive inputs from the CPn/PT cells that sustain activity, those on dMSNs no longer receive inputs from the CCS cells representing the same target stimulus because those CCS cells presumably have already become inactive. It has been suggested (Shen et al., 2008) that, different from the case of dMSNs, induction of long-term potentiation (LTP) or long-term depression (LTD) in iMSNs requires activation of the A_2A adenosine receptors or mGluR5, respectively. These two receptor types are suggested to form oligomers with D₂Rs (Cabello et al., 2009), and they are presumably activated by adenosine, which is generated around synapses from ATP released with glutamate (Cunha, 2001; Schiffmann et al., 2007) and glutamate that is spilled over from synapses (Mitrano et al., 2010), respectively. We conjectured that the sustained inputs from CPn/PT cells cause adenosine accumulation and glutamate spillover at the synapses on iMSNs, and such adenosine and glutamate can activate the A_2A receptors and mGluR5, respectively (Fig. 1B, right, orange circle). Moreover, because it has been suggested that effective generation of a response downstream of the A_2A receptors needs a stimulation of D₂Rs (Azdad et al., 2009), we conjectured that phasic increase in dopamine stimulates the A_2A receptor-signaling cascade leading to LTP, whereas phasic decrease in dopamine results in LTD presumably through the mGluR5-signaling cascade. Based on these conjectures, we assumed that the above equation effectively holds for both the synapses on dMSNs and those on iMSNs. Notably, however, it has been shown that, at least under a certain condition, A_2A receptor agonist induced LTP in the presence of D₂ antagonist (Shen et al., 2008, their Fig. 1F). Possibly, afferent stimulation applied in that study caused phasic dopamine release, and its effect on plasticity was not completely blocked by D₂ antagonist (although the responsiveness of iMSNs was likely to be significantly affected by D₂ antagonist). However, exploring how the assumption of our model can be well reconciled with these experimental results remains as an important future issue.

Plasticity induction depending on phasic dopamine response has been demonstrated in vivo (Reynolds et al., 2001). However, the synapses stimulated by contralateral cortical stimulation in that study would be mainly between CCS cells and dMSNs, because CCS cells but not CPn/PT cells target contralateral striatum (Reiner et al., 2010). For the synapses on dMSNs, a precise intracellular mechanism has also been proposed (Nakano et al., 2010). However, whether phasic decrease in dopamine induces LTD as assumed in the above was not tested previously by Reynolds et al. (2001) or Shen et al. (2008); in the latter study, effect of D₁ antagonist was examined, but it can be different from effects of phasic dopamine decrease. On the other hand, involvement of phasic dopamine response in plasticity induction in the synapses on iMSNs remains unclear. The abovementioned study by Shen et al. (2008) has shown that tonic activation of D₂Rs (by bath application of D₂ agonist in vitro) is unnecessary for LTP induction and would rather induce LTD in those synapses. However, again, afferent stimulation applied in that study possibly caused phasic dopamine release, and its effect was possibly masked by bath application of D₂ agonist [possible occurrence of such a masking in vivo has been discussed previously (Klein et al., 2007)]. Potential difference between bacterial artificial chromosome transgenic mice and wild-type animals (Bagetta et al., 2011; Kramer et al., 2011; Chan et al., 2012; Nelson et al., 2012) may also need to be carefully considered.

Overall, our assumption on plasticity could be in line with, or at least not inconsistent with, several experimental results as we explained above, but there remain divergences and limitations that need to be clarified in the future. An important limitation of our model, which would especially affect plasticity, is that it modeled neural activity as a continuous variable and did not model individual spike timings. In fact, a major finding by the in vitro study mentioned above for plasticity (Shen et al., 2008) was that the direction of induced plasticity (i.e., LTP or LTD) critically depends on the precise timings of presynaptic and postsynaptic neuronal firings (as known as spike-timing-dependent plasticity). Our model, lacking spikes, cannot be directly compared with their experiments. It is expected that more elaborated models using spiking neuron models will be developed in the future, e.g., combining the existing spiking neuron model for reinforcement learning (Potjans et al., 2011) with the circuit architecture that we proposed in this study. Also notably, not only the temporal resolution but also the spatial resolution of the model would need to be improved, given recent suggestions that plasticity might actually be regulated not by somato-axonic firings but instead by dendritic local spikes (Poirazi and Mel, 2001; Golding et al., 2002; Morita, 2009; Legenstein and Maass, 2011) in pyramidal cells and also possibly in the MSNs as the author of the abovementioned plasticity study himself recently pointed out (Surmeier et al., 2011).

Numerical simulations.

We set the initial value of w(n) to 0 and simulated successive 501 task blocks in three conditions: (1) without dopamine receptor antagonists; (2) with D₁ antagonist; and (3) with D₂ antagonist. The simulations were conducted by using MATLAB (MathWorks), and the presented data are the averages of the last 500 blocks for each condition (the initial block was not included because there was initial transient). For the simulations, we wrote codes (driven equations) for 110 scaled-down values of the variables (except for x_CCS and x_CPn), and then the results were 10 times scaled up when they were illustrated (this is just for practical reasons, and it should be mathematically equivalent to writing codes for the variables without scaling).

Elaborated model for the saccade task.

In the simple model described so far, we assumed that the activity of dMSNs at the timing of reward, which represents the predicted value of the “end-of-trial” state, was 0. However, given an empirical suggestion (Enomoto et al., 2011) that the response of dopamine neurons can reflect multiple rewards over multiple trials, such an assumption may not be very appropriate. Therefore, to examine effects of reward prediction (expectation) over multiple trials, we constructed an elaborated model for the same saccade task (Nakamura and Hikosaka, 2006) (see Fig. 6A–C). At each time step (t_i), the subject is assumed to be in one of the states shown in Figure 6A. A subset of CCS cells is assumed to represent that state, S(t_i) (see Fig. 6B), and the input from these CCS cells to dMSNs, denoted by I(S(t_i)), is assumed to represent the predicted value of state S(t_i) [denoted by V(S(t_i))], shifted by the amount of the threshold of the input–output function, i.e., V(S(t_i)) = I(S(t_i)) − θ. The activity of the dMSNs is assumed to be determined by imposing the input–output function f₁ (the same function defined above) on I(S(t_i)), i.e., f₁(I(S(t_i))), which is equal to V(S(t_i)) under the condition in which f₁ operates in the suprathreshold linear regimen and is not affected by D₁ antagonist. Meanwhile, a subset of CPn/PT cells is assumed to represent the state of the subject at the previous time step, i.e., S(t_{i − 1}) (see Fig. 6B). The input from these CPn/PT cells to iMSNs is assumed to encode I(S(t_{i − 1}))), and the activity of the iMSNs is assumed to be determined by imposing the input–output function f₂ on it, i.e., f₂(I(S(t_{i − 1}))), which is equal to V(S(t_{i − 1})) under the condition in which f₂ operates in the suprathreshold linear regimen and is not affected by D₂ antagonist. We assume that individual subsets of CCS cells or CPn/PT cells project to different subsets of dMSNs or iMSNs, respectively, and define x_dMSN and x_iMSN as variables representing the population activity of all the subsets of dMSNs and iMSNs; because single subsets of CCS cells and CPn/PT cells are assumed to be active at each time step as described above, this results in x_dMSN = f₁(I(S(t_i))) and x_iMSN = f₂(I(S(t_{i − 1}))). If reward is obtained at a given state, the activity of the PPN neurons, x_PPN, is assumed to represent the reward value Rew, Rew = 10 (large-reward case) or Rew = 5 (small-reward case); otherwise, x_PPN is assumed to be 0. Given the activity of dMSNs, iMSNs, and the PPN neurons, dopaminergic neuronal response (deviation from the baseline activity), x_DA, is assumed to be determined by where γ represents the strength/efficacy of the direct pathway (relative to the indirect pathway) (see Fig. 6C). This indicates that the dopamine response represents Under the condition in which the functions f₁ and f₂ operate in their suprathreshold linear regimens and are not affected by dopamine receptor antagonists, this is equivalent to the following: which is equal to the TD reward prediction error for state value (Sutton and Barto, 1998), and γ corresponds to the time discount factor. Such dopaminergic response presumably induces plasticity in the corticostriatal (CCS–dMSNs and CPn/PT–iMSNs) connections, and it is assumed to be represented by a modification of I(S(t_i-1)): where α is a constant representing the learning rate, and it was set to 0.75, the same as in the original simple model described above. Notably, this update (synaptic modification) is assumed to occur at every time step, not limited to the time point at which reward is obtained (as in the original model). The saccadic reaction time was assumed to be in the same quasi-inverse relationship with the activity of dMSNs at target presentation (i.e., at S₂ or S₃) as assumed for the simple model described above. The effects of D₁ and D₂ antagonists were also assumed to be the same as assumed in the simple model (see Fig. 3). The number of trials in individual blocks was set similarly to the case of the simple model described above. Both left-target and right-target trials were simulated, and Figure 6 shows the data for left-target trials [as for Fig. 6D,E, the first trials (shown in the left of the panels) were left-target trials and the second trials were left- or right-target trials (mixed)]; the model is left–right symmetric, and “left” and “right” are just labeling. Values of I were initialized to 0, and 521 task blocks were simulated in the absence of antagonists or with either D₁ and D₂ antagonist (the initial 10 blocks were simulated without antagonist in any cases) for each value of γ: γ = 0.75 and 0.9 (see Fig. 6) and 0.3 and 0.5 (data not shown). Average for the last 500 blocks are shown in Figure 6. As in the case of the original model, for the simulations, we wrote codes (driven equations) for 110 scaled-down values of the variables, and then the results were 10 times scaled up when they were illustrated.

As mentioned above, we ran simulations with different values of γ (strength/efficacy of the direct pathway relative to the indirect pathway). We described its purpose and results in detail below (see Results). Here we make another remark related to this point. As mentioned previously, there are direct projections from the striatal striosomes to the SNc dopamine neurons (Gerfen et al., 1985; Paladini et al., 1999; Fujiyama et al., 2011; Watabe-Uchida et al., 2012), but they are not incorporated into our model. At present, it is still somewhat ambiguous whether those projections are physiologically strong (Chuhma et al., 2011), and also the ratio with which the striosomal neurons express the D₁Rs or D₂Rs and as well as the ratio with which those cells receive inputs from CCS or CPn/PT cells are not clear (for the latest views on this issue, see Reiner et al., 2010; Crittenden and Graybiel, 2011). Our model would need to be revised when these things are clarified in the future. Notably, it has been shown previously (Fujiyama et al., 2011) that single striosomal neurons can project to both the SNr and SNc. If these neurons target GABAergic neurons (rather than dendrites of dopamine neurons) in the SNr and dopamine neurons in the SNc and also if they receive inputs predominantly from the CCS cells, existence of such co-projections could reduce the effective weight of the CCS–dMSNs–SNr–SNc pathway (i.e., CCS-direct pathway influence on the SNc) assumed in our model. In contrast, if the predominant upstream of the striosomal cells is the CPn/PT cells, existence of the co-projections to the SNr and the SNc would affect the presumed CPn/PT-indirect pathway influence on the SNc in our model. It has been shown previously (Watabe-Uchida et al., 2012) that not only the striosomal inputs but also dopamine neurons receive direct inputs from a number of structures, including the GP (rodent homolog of GPe). Any connections that were not incorporated into our model can potentially affect the effective value of the parameter γ (at the minimum, it is of course likely that those connections have certain functions other than modulating this parameter). As described in Results and shown in Figure 6, the elaborated model can well explain the observed distinct effects of the antagonists on reaction times if γ = 0.75 (actually, better than the original simple model; see Results) and could still explain them to a certain degree if γ = 0.9. We further confirmed that those effects could also be mostly reproduced if γ is set to 0.5 or 0.3 (data not shown). Therefore, at least in this sense, our model appears to have certain robustness against the unincorporated connections including the striosomal direct projections to the SNc.

Notably, the discrete time description of our model is certainly a very rough approximation, and construction of more detailed model with continuous time is desired in the future. Nevertheless, some sort of temporal discreteness might in fact exist in the actual operation of the dopamine system, i.e., particular time points can be “marked” in a sense that dopamine release is specifically enhanced through, for example, slowly oscillating neural activity, in particular, the recently reported 4 Hz synchronized oscillation in the prefrontal cortex, VTA, and hippocampus (Fujisawa and Buzsáki, 2011), possibly coupled with cholinergic induction of dopamine release (Threlfell et al., 2012). A single time step (duration between states) in our model is assumed to be approximately up to a few (to several) hundred milliseconds, which seems broadly consistent with this possibility. This timescale could also be in line with a different line of experimental observation. Specifically, it has been shown that dopamine neurons sometimes respond to events that should have zero reward prediction error with a biphasic “excitation-then-inhibition” pattern (Kakade and Dayan, 2002) whose duration looks to be approximately a few hundred milliseconds. The biphasic response could reflect such a circuit architecture as the one assumed in our model, i.e., fast excitation via the CCS-direct pathway and delayed inhibition via the CPn/PT-indirect pathway, and the time interval from excitation to inhibition could represent a canonical time step of the system.

Model of a task involving action selection.

The saccade task we modeled in the above consists of forced-choice trials only. To consider how our proposed circuitry operates in a situation in which subjects can voluntarily select an action based on their learned evaluations, we constructed an extended model of the corticobasal ganglia circuit incorporating a mechanism for action selection and simulated (a half of) the task involving action selection used in recent studies (Roesch et al., 2007; Takahashi et al., 2011); the latter study (Takahashi et al., 2011) has developed a computational model, and we extended our model in reference to their experimental results as well as the model. In the task, at the beginning of each trial, the subject (rat) was presented with one of three odor cues, two of which indicated that reward will be given if the animal entered either the left or the right well, respectively (i.e., forced choice), whereas the remaining cue indicated that the animal will be rewarded in both of the directions (i.e., free choice). In any case, the amount of reward, either small (one bolus of sucrose solution) or large (two boluses given sequentially), was determined according to the predetermined direction-amount contingency that was fixed during a block and then reversed. The three types of cues were presented according to a pseudorandom sequence so that the free-choice cue was presented in 7 of 20 trials and the two forced-choice cues were presented in equal numbers, and the same cue was not presented on more than three consecutive trials.

Takahashi et al. (2011) examined the effects of a lesion of the orbitofrontal cortex (OFC) on the dopamine neuronal activity and subjects' behavior. Comparing the results with a computational model developed within the work and also combining electrical stimulation, they have revealed that, in intact subjects, the OFC contains information about the “model” of the task structure, i.e., a diagram for transitions between different “states,” and influences the VTA dopamine neurons so that they can compute reward prediction error according to the model. Based on these results and according to a demonstration of the existence of CCS cells and CPn/PT cells in the OFC (Hirai et al., 2012), we assumed that the CCS cells and CPn/PT cells in the OFC upregulate and downregulate VTA dopamine neurons, respectively, through the pathways shown in Figure 7B. At each time step (t_i), the subject is assumed to be in one of the states shown in Figure 9A, which is similar to (although not exactly the same as) the states proposed by Takahashi et al. (2011). There are either multiple options (at state S₂) or a single option (at the other states) that can be taken.

In the case with a single option (i.e., states other than S₂), a subset of CCS cells is assumed to represent the combination of that state and the available choice option [A(t_i)], and the input from these CCS cells to dMSNs, denoted by I(A(t_i)), is assumed to represent the predicted value of option A(t_i) [denoted by Q(A(t_i))], shifted by the amount of the threshold of the input–output function, i.e., Q(A(t_i)) = I(A(t_i)) − θ. The activity of the dMSNs is assumed to be determined by imposing the input–output function f₁ on I(A(t_i)), i.e., f₁(I(A(t_i))), which is equal to Q(A(t_i)) under the condition in which f₁ operates in the suprathreshold linear regimen and is not affected by D₁ antagonist. Meanwhile, a subset of CPn/PT cells is assumed to represent the combination of the state of the subject at the previous time step (t_{i − 1}) and a choice option that has been taken at that state [A(t_{i − 1})]. The input from these CPn/PT cells to iMSNs is assumed to encode I(A(t_{i − 1})), and the activity of the iMSNs is assumed to be determined by imposing the input–output function f₂ on it, i.e., f₂(I(A(t_{i − 1}))), which is equal to Q(A(t_{i − 1})) under the condition in which f₂ operates in the suprathreshold linear regimen and is not affected by D₂ antagonist. We assume that individual subsets of CCS cells or CPn/PT cells project to different subsets of dMSNs or iMSNs, respectively, and define x_dMSN and x_iMSN as variables representing the population activity of all the subsets of dMSNs and iMSNs; because single subsets of CCS cells and CPn/PT cells are assumed to be active at each time step as described above, this results in x_dMSN = f₁(I(A(t_i))) and x_iMSN = f₂(I(A(t_{i − 1}))). If reward is obtained at a given state, the activity of the PPN neurons, x_PPN, is assumed to represent the reward value Rew (set to 10); otherwise, x_PPN is assumed to be 0. Specifically, x_PPN = Rew = 10 at S₈ or S₉ in the case of small reward, and x_PPN = Rew = 10 at (S₈ and S₁₀) or (S₉ and S₁₁) in the case of big reward.

In the state in which multiple options are available (i.e., in state S₂), we assumed that the “max” operation is implemented through the corticostriatal (CCS–dMSN) feedforward inhibition (Parthasarathy and Graybiel, 1997; Gittis et al., 2010) and possibly also through lateral or feedback inhibition, i.e., x_dMSN = f₁(max{I(A(t_i))}) so that the activity of dMSNs reflects the predicted value of an option that currently has the maximum predicted value [max{Q(A(t_i))}] under the condition in which f₁ operates in the suprathreshold linear regimen and is not affected by D₁ antagonist (see Fig. 8C). This assumption is based on a previous finding (Roesch et al., 2007) that the VTA dopamine neurons appear to represent TD reward prediction error for Q-learning (for a detailed explanation, see below and Results). The activity of iMSNs was assumed to be the same as the case with a single option described above.

Given the activity of dMSNs, iMSNs, and the PPN neurons, dopaminergic neuronal response (deviation from the baseline activity), x_DA, is assumed to be determined by where γ represents the strength/efficacy of the direct pathway (relative to the indirect pathway), and it was set to 0.75 in the simulations. This indicates that the dopamine response represents in the case in which there is a single-choice option, and in the case in which there are multiple options. Under the condition in which the functions f₁ and f₂ operate in their suprathreshold linear regimens and are not affected by dopamine receptor antagonists, these are, respectively, equivalent to and which equal to the TD reward prediction error for action value defined in Q-learning (Sutton and Barto, 1998), except for decay of plastic changes in the corticostriatal connection strengths that we also assumed as we describe below.

In the case in which there are multiple (two) choice options, i.e., at state S₂ (the two options are A₂ and A₃), one of the options were assumed to be selected according to a soft-max operation (see Fig. 9A, right graph) through competitive neural dynamics among the CPn/PT cells receiving the effects of dMSNs (Fig. 8C). Specifically, we assumed that the probabilities that options A₂ and A₃ are selected, Prob(A₂) and Prob(A₃), are given by and respectively, where β is a parameter determining the balance between exploration and exploitation, and it was set to 0.5 in the simulations (for a detailed explanation on the presumed circuit operation, see Results).

These are equal to under the condition in which f₁ operates in the suprathreshold linear regime and is not affected by D1 antagonist.

Dopamine response is assumed to induce plasticity in the connections between CCS cells and dMSNs and those between CPn/PT cells and iMSNs at every time step, resulting in the following update rule: A(t_{i − 1}) represents an option that has actually been taken at time t_{i − 1}, which is not necessarily the same as the one that has been once selected at the stage of dMSNs (i.e., the one with the maximum predicted value) because of the assignments of the hard-max and soft-max operations to the stages of dMSNs and CPn/PT cells, respectively, in our presumed implementation of Q-learning (see Fig. 8C). We propose that the subset of dMSNs corresponding to A(t_{i − 1}), synapses on which should be modified according to x_DA, could be marked as the target of plasticity by receiving inputs from CPn/PT cells representing A(t_{i − 1}) via CPn/PT–dMSN connections, which are minor but known to still exist (Lei et al., 2004; Reiner et al., 2010). Indeed, these minor inputs may cause only dendritic spikes/plateaus, which have been suggested to promote plasticity (Golding et al., 2002; Surmeier et al., 2011) as we mentioned above, but not somatic spikes so that they do not affect the computation of reward prediction error in the dopamine neurons or the action selection process. Notably, in the cases of our proposed implementations of SARSA (state–action–reward–state–action) (see Fig. 8A,B), mismatch between the maximally activated dMSNs and the eventually selected option never occurs. α is the learning rate and is set to 0.6. The reaction time was assumed to be quasi-inversely related to the activity of dMSNs at cue offset (at S₄ or S₅), in a manner similar to the assumption made in the models for the saccade task above: with different parameters: C₁ and C₂ were set to 2300 and 10, respectively. The effects of D₁ and D₂ antagonists were assumed to be the same as assumed in the models for the saccade task (see Fig. 3).

We explored parameters of the model with which the performance of the model becomes comparable with the reported experimental data (Roesch et al., 2007; Takahashi et al., 2011). In the course of this exploration, we realized that an additional assumption seems to be necessary to be added to the model to reproduce important features of the neural activity data. In the experiment, the dopamine neuronal response to reward clearly remained and was in fact stronger than the response to a cue, even in the last 10 trials of a block (Roesch et al., 2007, their Fig. 4d). This indicates that, for the dopamine neurons (not necessarily for the animals), reward remained to be rather “surprising,” not fully predictable, through an entire task block. At the same time, however, the dopamine neuronal response to a cue clearly increased or decreased depending on whether the cue was associated with large or small reward, respectively, indicating that the dopamine neuronal responses certainly reflect the progression of learning. To reproduce both of these features simultaneously, some sort of time-dependent decay of learned values seems to be required to be incorporated into the model, while the learning rate and the time discount factor, which are represented by the (relative) strength of the direct pathway over the indirect pathway in our model (Morita et al., 2012), should be kept reasonably high. We achieved this by assuming time-dependent decay of plastic changes of the corticostriatal connection strengths. Specifically, we made an additional assumption that changes in the corticostriatal connection strengths, or more precisely, deviations of I(A₁), etc., from their initial (baseline) value, I₀, are subject to time-dependent decay at every time step: (we assumed 16 time steps within a single task trial, and thus the above corresponds to the fact that changes in I(A_i) decay to 90% per trial). I₀ was set to 4.5, which was below the threshold of the input–output functions of the MSNs in the case without antagonists (θ = 5 as described above). We explored parameters with which the model qualitatively reproduced main features of the experimental results (Roesch et al., 2007; Takahashi et al., 2011) and obtained the values described in the above.

In a separate set of simulations, in one of the two wells (left and right), virtual optogenetic stimulation was applied to either dMSNs or iMSNs coincidently with a reward bolus (at S₈ or S₉), in addition to giving an extra bolus of reward at the subsequent timing in both of the wells. Specifically, in addition to setting Rew to 10 at S₁₀ and S₁₁, which represents the extra bolus, the value of x_dMSN or x_iMSN at S₈ or S₉ was increased by 10, representing the optogenetic stimulation (10 was added directly to x_dMSN rather than to the input term, on which f₁ or f₂ was imposed, because currents induced by optogenetic stimulation would bypass the AMPA and NMDA channels and so would not be directly affected by the antagonists in the same way as the synaptic currents as we assumed in Fig. 3). The contingency between the stimulation on/off and the location of the well was fixed for a block and alternated across blocks in a similar manner to the contingency between the presence/absence of the extra bolus of reward and the location of the well in the original experiments.

The number of trials in each block was set to 120 [in the experiments by Roesch et al. (2007), it is described that at least 60 trials were collected per block]. The three types of cues were presented pseudorandomly, in a manner similar to the one in the experiment (Roesch et al., 2007) described above: the free-choice cue was presented on 7 of 20 trials, and the two forced-choice cues were presented in equal numbers, on average in each session, and the same cue was not presented on more than three consecutive trials. Values of I were initialized to I₀ = 4.5, and 2021 task blocks were simulated under seven conditions: (1) big versus small reward, without antagonists; (2) big versus small reward, with D₁ antagonist; (3) big versus small reward, with D₂ antagonist; (4) dMSN stimulation, without antagonists; (5) iMSN stimulation, without antagonists; (6) dMSN stimulation, with D₁ and D₂ antagonists; and (7) iMSN stimulation, with D₁ and D₂ antagonists. The averages for the last 2000 blocks are shown in Figure 9. As in the cases of the abovementioned models, for the simulations, we wrote codes (driven equations) for 110 scaled-down values of the variables, and then the results were 10 times scaled up when they were illustrated.