Abstract
The prospect of immediate reward elicits goal-oriented behavior. However, animals often have to perform actions that do not immediately lead to reward in the pursuit of a long-term goal. Here we identify neural activity in monkey caudate nucleus that specifically correlates with rewarded and unrewarded eye movements. The monkey performed a visually guided saccade task in which only one position was associated with positive reinforcement. To advance in the experimental session, however, the monkey had no choice but to complete a saccade to an unrewarded position as well as to a rewarded position. Some caudate saccadic neurons showed enhanced activity around the time of the saccade in rewarded trials (rewarded-saccade neurons). Another subset of neurons discharged selectively around the execution of the saccade in unrewarded trials (unrewarded-saccade neurons). In both rewarded and unrewarded trials, stronger activity of these neurons was associated with reduced saccade latency. These results suggest that both rewarded and unrewarded saccades are facilitated by caudate saccadic activity. The neuronal activity of unrewarded-saccade neurons might reflect the required execution of unrewarded eye movements on the way to future reward.
Introduction
The expectation of immediate reward evoked by sensory stimuli is a central element of the motivational state underlying approach behavior (Bindra, 1968). Previous studies in nonhuman primates showed that the prospect of immediate reward facilitates the operant behavior required to obtain it (Schultz et al., 1992; Bowman et al., 1996; Shidara et al., 1998; Tremblay and Schultz, 2000; M. Watanabe et al., 2001, 2002; Lauwereyns et al., 2002; Shidara and Richmond 2002; Takikawa et al., 2002; Watanabe et al., 2003). The manual and oculomotor response times become shorter if the action is associated with immediate reward. The facilitation of an action with the expectation of immediate reward attests to the positive motivational state of the animal (Shidara et al., 1998; Lauwereyns et al., 2002; Shidara and Richmond, 2002). By contrast, an action that is not immediately followed by reward tends to be suppressed and/or delayed. The low priority assigned to an unrewarded action reflects the negative motivational state of the animal (Watanabe et al., 2003). In natural situations, however, animals often have to perform actions that do not lead to immediate reward in the pursuit of a long-term goal (Miller, 2000; Glimcher, 2001; Schall, 2001). For instance, when pressed to find food, animals may enter unknown or dangerous territory to increase their search area. Thus, being able to execute an action without immediate reward is a critical condition for success in situations in which reward resources are limited and sparse.
To investigate the neural mechanisms for the control of action with and without immediate reward, we used the biased saccade task (BST; Lauwereyns et al., 2002). In the BST, the monkey has to make an immediate saccade to a peripheral visual target in every trial, but is rewarded for a saccade to only one of two possible positions. The BST has been successfully used for investigations of neural mechanisms for response bias because the asymmetric reward schedule produces a conspicuous difference in saccade latency and anticipatory activity (which occurs before a visual target presentation) in the monkey caudate nucleus (Lauwereyns et al., 2002).
Besides the asymmetric reward schedule, the BST has yet another important feature; the same trial is repeated for an uncompleted response. Hence, to go on with the experimental session, the monkey has to make a saccade even when no reward will be delivered in the current trial. This feature allows us to examine how neuronal activities are related to the production of a rewarded action and that of an unrewarded, but required, action. The primate caudate nucleus plays a central role in controlling saccadic eye movements (Hikosaka et al., 1989, 2000) and also in cognitive-motivational functions (Rolls, 1994; Hikosaka et al., 2000). Here we focused on exploring how caudate saccadic (as opposed to visual, reward, and anticipatory) neurons change their activity on the basis of reward expectation and how neural activity is related to saccade latency.
Parts of this paper have been published previously (K. Watanabe et al., 2002).
Materials and Methods
Subjects and surgery. We used two adult male Japanese monkeys (Monkey A and Monkey B, Macaca fuscata; 6.0-7.5 kg). A head-holding device, a chamber for unit recording, and a scleral search coil were implanted under general anesthesia. A recording chamber (anteroposterior: 42 mm; lateral: 30 mm; depth: 10 mm) was placed over the frontoparietal cortices, tilted laterally by 35° in the coronal plane, and was aimed at the caudate nucleus based on magnetic resonance imaging. The monkeys were kept in individual primate cages in an air-conditioned room, where dry pellets were always available. During periods of training and experiments, the monkeys' access to water in the cage was controlled and monitored. All surgical and experimental procedures conformed to the National Institutes of Health (NIH) Principles of Laboratory Animal Care (NIH publication number 86-23, revised 1985) and were approved by the Juntendo University Animal Care and Use Committee.
Behavioral task. The monkey sat in a primate chair inside a sound-attenuated room with his head fixed. Visual stimuli were small red spots, 0.2° in diameter, back-projected onto a tangent screen by LED projectors. After the monkey directed and maintained his gaze at a central fixation spot for 1.5 sec, the fixation spot disappeared, and another target spot immediately appeared at 20° to the left or to the right. The monkey had to make a saccade within 500 msec to within 3° from the target position. Trials with a target to the left or to the right were randomly interleaved. An auditory feedback (800 Hz rectangular waveform) was provided for each completed trial. If the monkey broke fixation or did not make a saccade within 500 msec (uncompleted trials), the same trial was repeated. There was a random intertrial interval of between 3 and 4 sec. In half of the completed trials the monkey was rewarded with a drop of water (asymmetrical reward schedule). During blocks of 20 completed trials, reward was mapped consistently onto one target position. The position-reward association remained constant within a block but was reversed frequently (6-16 times) and automatically, without any indication to the monkey, and without any pause between blocks. Note that the position-reward contingency was fixed, but the target position was unpredictable. Because the same trial was repeated after an uncompleted trial, the monkey had to properly make a saccade even when no reward would be provided in a particular trial. Both monkeys were trained in the BST for >6 months.
Electrophysiological recording. Eye position was measured with a standard magnetic search-coil technique (Judge et al., 1980), digitized at 500 Hz, and stored with event times for off-line analysis. During recording sessions, action potentials of single neurons were recorded with tungsten electrodes, which were advanced perpendicularly to the cortical surface using an oil-driven micromanipulator. The action potentials were amplified, filtered (500 Hz to 2 kHz), and processed by a window discriminator. We selected extracellular spike activity of presumed GABAergic (Fisher et al., 1986) medium-spiny (Kawaguchi et al., 1990) projection neurons, which show very low spontaneous activity (Hikosaka et al., 1989), but not of presumed interneurons, which show irregular tonic activity (Aosaki et al., 1994). Because the aim of the present study was to examine the effects of motivational context on saccade-related activity in caudate neurons, we searched selectively for neurons that showed saccade-related activity while the monkey performed the task. When we encountered a presumed caudate projection neuron, we judged whether it displayed saccadic activity by visual inspection. If it did, we proceeded with recording the neuron in as many trials as possible. If it did not, we resumed the search.
Data analysis. An eye movement was considered a candidate of a saccade if its instantaneous velocity rose above 30°/sec and acceleration exceeded 90°/sec2 after the visual target onset. The end of the eye movement was determined to be when the velocity became lower than 40°/sec. The eye movement was then accepted as a saccade if the velocity was higher than 45°/sec for at least 10 msec and the total duration was longer than 30 msec.
Neurons were defined as saccade related if they showed statistically reliable higher saccadic activity (activity in the period between 50 msec before and 50 msec after saccade onset), compared with the period from -150 to -51 msec before saccade onset and with the period from +51 to +150 msec after saccade onset (two-tailed t test; p < 0.05). The saccade-related activity was further confirmed with a memory-guided saccade task and/or the absence of activity in aborted trials. First, we characterized the relation between saccade latency and saccadic activity by using median-split analysis (Miller and Low, 2001). For each neuron, trials were divided into high-activity and low-activity trials at median discharge rate within each of four trial categories (contra-ipsi × rewarded-unrewarded). Mean saccade latencies in high- and low-activity trials were calculated within each type of neuron and for each location-reward relation and compared with paired t tests. To support observations obtained by the median-split analysis, we further performed linear regression analyses. For the regression analysis, neuronal discharge rate and saccade latency from all neurons and all trials were normalized within condition after Box-Cox transformation.
Results
We recorded the single-unit activity of 150 neurons from the left caudate nucleus of Monkey A and both caudate nuclei of Monkey B and concentrated on 71 neurons that selectively increased their activity around the moment of saccade onset. The saccadic activity of 59 (83%) of these neurons showed significant main effects of saccade direction and reward expectation, and/or interaction between these factors (p < 0.05; two-way unbalanced ANOVA; contra-ipsi × rewarded-unrewarded). Of these 59 neurons, the majority (45 of 59 or 76%) were tuned to contraversive saccades, and we focused on these contraversive neurons in the following analysis.
Figure 1a shows an example of neurons that showed higher activity for contraversive saccades in rewarded trials (black) than in unrewarded trials (gray) (rewarded-saccade neurons). This neuron was found in the left caudate nucleus of Monkey B and was selectively active when the monkey made a right (i.e., contraversive) saccade. The saccade-related activity was enhanced when the right position was associated with reward. Mean saccade latencies were shorter for rewarded than those for unrewarded trials (rewarded = black vertical bar; unrewarded = gray vertical bar). Twenty-six of the 45 contraversive-saccadic neurons (58%; 10 from Monkey A and 16 from Monkey B) were classified as rewarded-saccade neurons. The remaining 19 (42%; 4 from Monkey A and 15 from Monkey B) showed higher activity in unrewarded trials (unrewarded-saccade neurons). A representative unrewarded-saccade neuron is shown in Figure 1b. This neuron was found in the right caudate nucleus of Monkey B and was selectively active when the monkey made a left (i.e., contraversive) saccade. The activity for leftward saccades was enhanced when the left position was not rewarded.
Effect of position-reward mapping and reward gain on activity of caudate saccadic neurons. a, Example of rewarded-saccade neuron. b, Example of unrewarded-saccade neuron. Top and middle, Rasters of spikes aligned with saccade onset. Each raster represents one trial; trials were classified on the basis of position-reward mapping (boxes at the center) and saccade direction (contraversive and ipsiversive). Black, Rewarded; gray, unrewarded. The gray lines in rewarded trials indicate water delivery. Bottom, Average discharge rates aligned with saccade onset. The vertical lines represent the averaged times of the target presentation relative to saccade onsets (black, rewarded; gray, unrewarded). Broken lines show mean discharge rate in aborted trials (based on 2-6 trials). Because no saccade occurred on those uncompleted trials, the activity was aligned with average latencies for rewarded and unrewarded trials. No trial was aborted for rightward saccades while recording RRG1406.
Although the monkeys were highly trained, trials were occasionally aborted because of fixation break or incorrect saccade (rewarded trials, 1.3%; unrewarded trials, 3.6%). On those aborted trials, most of these saccade-related neurons did not show an increase of discharge rate (Fig. 1, broken lines), confirming the activity was saccade-related. Also, we reanalyzed the data excluding trials that were repeated because of previously uncompleted trials and found the same pattern of results. Even after excluding repeated trials, all the neurons were classified to the same categories, and there was little change in the statistical results.
Consistent with our previous study (Lauwereyns et al., 2002), we found that the average latency of saccades was significantly shorter in rewarded trials than in unrewarded trials (p < 0.001; two-tailed t test) (Fig. 2a,b). Mean saccade latency was also slightly longer in contraversive than ipsiversive trials. This result was attributable to the fact that Monkey A had idiosyncratically longer latencies for rightward saccades than for leftward saccades and we recorded only from the left caudate in Monkey A. This result was of no theoretical interest and did not affect the comparison of rewarded versus unrewarded trials. An intuitive explanation for the effect of reward expectation on saccade latency would be that rewarded-saccade neurons generally facilitate saccades, whereas unrewarded-saccade neurons generally inhibit saccades. We first used the median-split analysis to characterize the observed saccadic activity and to test specifically these hypotheses.
a, Left, Relationship between saccadic activity of caudate neurons (horizontal coordinate) and saccade latency (vertical coordinate), revealed by median-split analysis for rewarded-saccade neurons. For each trial category (combination of saccade-direction and reward; contra-ipsi × rewarded-unrewarded), a line is drawn between mean saccade latency of low-activity trials and that of high-activity trials. Within trial category, mean saccade latencies were examined by paired t test:*p < 0.05; **p < 0.025; ***p<0.01. Black and gray lines, respectively, indicate rewarded and unrewarded trials. Thick, solid lines designate contraversive saccades. Thin, broken lines designate ipsiversive saccades. Error bars = 1 SE. The black lines are located further to the right than the gray lines, confirming the activity enhancement in rewarded trials. The negative slopes of the lines signify that higher saccadic activity is associated with earlier saccade. b, Right, Median-split analysis for unrewarded-saccade neurons. That the gray lines are located further to the right than the black lines verifies that the activity is enhanced in unrewarded trials. Nevertheless, higher saccadic activity tended to produce earlier eye movements within trial category.
The “facilitatory” hypothesis for rewarded-saccade neurons would predict earlier saccades with higher neuronal activity (i.e., negative slope) even within trial category. As shown in Figure 2a, this prediction was confirmed for contraversive saccades (both rewarded and unrewarded trials; two-tailed t test; p < 0.05 and p < 0.01, respectively), but not for ipsiversive saccades. Higher activity of the rewarded-saccade neurons correlated with earlier saccades to a contralateral rewarded position.
On the other hand, the “inhibitory” hypothesis for unrewarded-saccade neurons would predict later saccades with higher neuronal activity (i.e., positive slope). This was obvious for the activity-latency relation between trial categories (Fig. 2b). However, within trial category, this turned out to be incorrect; higher activity of unrewarded-saccade neurons was associated with earlier saccades to a contraversive, unrewarded position (p < 0.025; two-tailed t test). Even though the unrewarded-saccade neurons became more active when no reward was expected and therefore when the saccade latency was generally long, the saccades to unrewarded positions were quicker with higher activity.
These results were robust across subjects. ANOVAs with monkey identity as a main factor showed that, although main effects of subject identity were significant (rewarded-saccade, p < 0.05; unrewarded-saccade, p < 0.01), the factor of subject identity did not interact with saccade latency (rewarded-saccade, p = 0.051; unrewarded-saccade, p = 0.667) nor with the combination of saccade latency and reward condition (rewarded-saccade, p = 0.161; unrewarded-saccade, p = 0.2).
Results of the linear regression analysis generally confirmed those of the median-split analysis. For rewarded-saccade neurons (Fig. 3) there were statistically significant negative relations between neural activity and saccade latency when the monkeys made contraversive saccades (both in rewarded and unrewarded trials). Also for unrewarded-saccade neurons (Fig. 4) there was a significant negative relation between neural activity and saccade latency when the monkeys made contraversive saccades in unrewarded trials.
Confirmation of negative slopes in linear-regression for rewarded-saccade neurons. For illustrative purpose, the normalized discharge rates were binned (20 bins; fixed width, 0.25), and the means of normalized saccade latency in each bin were presented (white circle) with error bars indicating 1 SE. Some individual bins are not shown because there were not enough data points in those bins.
Confirmation of negative slopes in linear regression for unrewarded-saccade neurons.
Discussion
The present study has examined the relation between neural activity and saccade latency by focusing on the caudate saccadic activity. Similar modulations were found in visual neurons in the caudate nucleus (Kawagoe et al., 1998) and in prefrontal cortex (Kobayashi et al., 2002); visual responses of neurons in these areas are enhanced or depressed if the visual stimulus indicates immediate reward. In a previous study (Itoh et al., 2003) it was shown that visual activity of reward-enhanced neurons is negatively correlated with saccade latency, whereas visual activity of reward-depressed neurons is not. The present results demonstrated that the bivalency (i.e., both positive and negative expectations of reward gain) modulates neural activity of saccadic (i.e., primarily motor-related) neurons, and the saccadic neural activity is correlated with saccade latency.
Rewarded-saccade neurons showed higher activity in rewarded trials, and the neural activity was negatively correlated with saccade latency within trial categories. Because saccade latency in rewarded trials was shorter than in unrewarded trials, the results of rewarded-saccade neurons suggest that the saccadic activity generally is correlated with a quicker (contraversive) saccade. The relation between neural activity and saccade latency seems more multifaceted for the unrewarded-saccade neurons. Saccadic activity was overall higher in unrewarded trials than in rewarded trials. Because saccade latency was longer in unrewarded trials, it would be natural to assume that higher activity of unrewarded-saccade neurons leads to longer saccade latency even within trial categories. However, it turned out that stronger activity of unrewarded-saccade neurons was associated with reduced saccade latency.
What is the behavioral significance of unrewarded-saccade neurons? One possibility is that it reflects some cognitive and/or motivational state under a motivational conflict. The prospect of immediate reward sets up a motivational response bias toward a spatial position associated with immediate reward (Lauwereyns et al., 2002; Watanabe et al., 2003). This bias would facilitate a saccade to the rewarded position, but it hinders a saccade to the unrewarded position. Yet, in the BST, the animal has to make a saccade even though no immediate reward is expected. The unrewarded-saccade neurons showed higher activity in such a situation. Then, the neuronal activity of unrewarded-saccade neurons might correlate with the extra effort or attention that is required to counteract the motivational response bias. Such a counteractive process should occur after the registration of a visual target at an unrewarded position, whereas the response bias toward a rewarded position may exist throughout an entire block of trials (Watanabe et al., 2003). Consistent with this account, the negative correlation between saccade latency and neural activity of unrewarded-saccade neurons was significant only for contraversive unrewarded trials.
The present results imply that both the prospect of immediate reward and the pursuit of a long-term goal are transformed and attached to eye movement signals in the caudate nucleus in primates. Obviously, this hypothesis warrants further investigation because the relations between neural activity and saccade latency within trial categories represent a relatively minor variation in neural activity and saccade latency, compared with the modulation by reward itself.
Relation to other studies
Our BST is somewhat similar to the paradigm used in several other studies (Bowman et al., 1996; Shidara et al., 1998; Tremblay and Schultz, 2000; Shidara and Richmond, 2002). For example, in Tremblay and Schultz (2000), rewarded-movement and unrewarded-movement trials involved reaching from the same starting position toward the same response lever. Reaction times were significantly shorter for rewarded as compared with unrewarded-movement trials. Unrewarded movements did not lead to immediate reward but were followed by a conditional auditory reinforcer and a subsequent rewarded trial. Other researchers focused on how the expectation of reward affects the neural activity of ventral striatum (Shidara et al., 1998) and anterior cingulate cortex (Shidara and Richmond, 2002). They trained monkeys to respond successfully in several consecutive trials to gain a reward at the end of the sequence and found a set of neurons that fired selectively in unrewarded trials (i.e., trials in which the monkey had to respond successfully while expecting an opportunity to obtain reward in a future trial). Thus, similar to the present study, these researchers came across neural activities that could be associated with the execution of an action that is not immediately followed by reward but necessary for pursuing future reward. However, the present study differs from these previous studies in two important ways.
First, the present study has examined explicitly the relation between neuronal activity and behavioral parameters. Here we investigated saccade-related neuronal activity from single neurons and saccadic latency between and within conditions. The between-condition comparison showed that the saccade-related activity in the caudate nucleus is sensitive to the expected gain of reward. However, to study the functional significance of neuronal activity of these neurons, it is essential to examine neuronal activity in relation to behavioral parameters within each condition (e.g., contraversive saccades in unrewarded trials). For example, higher activity of unrewarded-saccade neurons could be related to a general increase in saccade latency or it could be related to some cognitive and/or motivational functions under the motivational conflict. The within-condition comparison revealed that the activity of unrewarded-saccade neurons was not simply related to motor control because it was dependent on the reward value of the saccade, suggesting a motivational function. In this way, we were able to explore the functional contribution of the caudate nucleus to goal-oriented eye movements; the caudate saccadic neurons seem to facilitate eye movements in both rewarded and unrewarded trials. Thus, the between-condition and within-condition analysis proves to be a valuable tool to examine the relation between neuronal activity and behavioral parameters.
Second, in the previous studies, the reward gain in a particular trial was clearly known. The visual cue (Shidara et al., 1998; Shidara and Richmond, 2002) and the feedback (Tremblay and Schultz, 2000) indicated how many responses would be required to receive the reward after an unrewarded trial. In other words, the probability of receiving a reward in a particular future trial was explicitly defined. In the present study, there was little information about the reward availability after an unrewarded trial. Thus, in a given unrewarded trial the subject had no choice but to make the required eye movement without any guarantee that a reward would be given in the next trial. Fortunately, the primate brain is capable of producing a required eye movement under such challenging circumstances, and the activity in the caudate nucleus may be a sign of such processes.
Footnotes
This work was supported by the Japanese Society for the Promotion of Science and the United States National Institutes of Health.
Correspondence should be addressed to Okihide Hikosaka, Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health, Building 49, Room 2A50, 49 Convent Drive, Bethesda, MD 20892-4435. E-mail: oh{at}lsr.nei.nih.gov.
Copyright © 2003 Society for Neuroscience 0270-6474/03/2310052-06$15.00/0
