Abstract
Novelty modulates sensory and reward processes, but it remains unknown how these effects interact, i.e., how the visual effects of novelty are related to its motivational effects. A widespread hypothesis, based on findings that novelty activates reward-related structures, is that all the effects of novelty are explained in terms of reward. According to this idea, a novel stimulus is by default assigned high reward value and hence high salience, but this salience rapidly decreases if the stimulus signals a negative outcome. Here we show that, contrary to this idea, novelty affects visual salience in the monkey lateral intraparietal area (LIP) in ways that are independent of expected reward. Monkeys viewed peripheral visual cues that were novel or familiar (received few or many exposures) and predicted whether the trial will have a positive or a negative outcome—i.e., end in a reward or a lack of reward. We used a saccade-based assay to detect whether the cues automatically attracted or repelled attention from their visual field location. We show that salience—measured in saccades and LIP responses—was enhanced by both novelty and positive reward associations, but these factors were dissociable and habituated on different timescales. The monkeys rapidly recognized that a novel stimulus signaled a negative outcome (and withheld anticipatory licking within the first few presentations), but the salience of that stimulus remained high for multiple subsequent presentations. Therefore, novelty can provide an intrinsic bonus for attention that extends beyond the first presentation and is independent of physical rewards.
Introduction
One of the most remarkable abilities of the mammalian brain is its vast capacity to learn and acquire new information. An important ingredient of this ability is a novelty bias, whereby animals show preference over novel rather than familiar items (Brockmole and Henderson, 2005; Wittmann et al., 2008; Yang et al., 2009; Park et al., 2010; Liao et al., 2011). In humans, monkeys, and rats, novelty has widespread effects on sensory processing and motivation (Rainer and Miller, 2002; Düzel et al., 2010; Roozendaal and McGaugh, 2011), showing that understanding its neural coding is essential for understanding the control of learning and exploration.
A central question in novelty research is how animals become motivated to explore novel items given that, by definition, they do not yet know the value of these items. One explanation, proposed in studies of reinforcement learning, is that the effects of novelty are explained in terms of reward. According to this idea, a novel stimulus is by default afforded high reward value but this salience is modified through learning, declining rapidly if the stimulus signals a negative outcome (Kakade and Dayan, 2002; Laurent, 2008). Consistent with this hypothesis, novelty activates the reward circuitry, including midbrain dopamine (DA) neurons in animals (Ljungberg et al., 1992; Horvitz, 2000) and dopamine-recipient structures in humans (Bunzeck and Düzel, 2006; Wittmann et al., 2007, 2008; Bunzeck et al., 2009, 2010, 2011, 2012; Düzel et al., 2010; Guitart-Masip et al., 2010). An alternative possibility, however, is that novelty has some reward-independent effects (Krebs et al., 2009). Consistent with this possibility, novelty enhances visual responses in mid- and high-level temporal areas (Li et al., 1993; Rainer and Miller, 2002; Yanike et al., 2004; Woloszyn and Sheinberg, 2012), but it is unknown how this enhancement depends on expected reward. With this background in mind, we sought to examine the neural mechanisms by which novelty enhances salience and attention, and how these effects relate to those of reward.
We focused on a dorsal stream area involved in eye movements and spatial attention: the monkey lateral intraparietal area (LIP). LIP cells are particularly appropriate for addressing this question because they have spatially selective responses indicating the locus of attention and forthcoming saccades (Bisley and Goldberg, 2010; Gottlieb and Balan, 2010). In addition, the cells integrate information about multiple factors including physical conspicuity, task relevance, and reward associations, suggesting that they will shed light on novelty × reward interactions. We show that salience—as evidenced in the monkeys' saccades and LIP responses—was enhanced by both novelty and reward associations, but these factors had dissociable effects. The monkeys rapidly recognized when a novel stimulus signaled a negative outcome and extinguished their anticipatory licking in response to the stimulus within the first few presentations. Surprisingly however, the salience of that stimulus remained high on a longer time scale. Whereas a highly familiar no-reward cue suppressed attention and LIP responses at its visual field location, this repulsion was not seen for novel stimuli over the first tens of presentations. This suggests that novelty provides a temporally extended bonus for salience and attention that is independent of, and can partly contravene, the impact of physical rewards.
Materials and Methods
General methods.
Data were collected from two adult male rhesus monkeys (Macaca mulatta) using standard behavioral and neurophysiological techniques as described previously (Oristaglio et al., 2006). All methods were approved by the Animal Care and Use Committees of Columbia University and New York State Psychiatric Institute as complying with the guidelines within the National Institutes of Health Guide for the Care and Use of Laboratory Animals. Single electrodes were advanced into the intraparietal sulcus using a Kopf Microdrive (David Kopf Instruments), and the data were recorded using the APM digital processing module for neural signal recording (FHC), and MATLAB (MathWorks) for offline data analysis. Visual stimuli were presented on a Sony GDM-FW9000 Trinitron monitor (30.8 × 48.2 cm viewing area) located 57 cm in front of the monkey. The precise timing of stimulus presentation was measured using a diode fixed to the top left corner of the monitor to detect the onset of a refresh cycle. Licking was measured by means of an infrared beam that was projected between the monkey's mouth and the reward spout and produced a transistor–transistor logic pulse each time it was interrupted by protrusions of the monkey's tongue. Eye position was recorded using an eye-coil system and digitized at 500 Hz.
Behavioral task.
During the task, two placeholders were continuously present, positioned so they fell in and opposite the receptive field (RF) of the recorded cells when the monkey achieved central fixation (Fig. 1). After the monkey achieved fixation, a reward cue was presented for 300 ms at a randomly selected placeholder location. The disappearance of the cue was followed by a 600 ms delay period during which monkeys had to maintain fixation and then by the removal of the fixation point and brightening of a placeholder at a randomly selected location. Monkeys had to make a saccade to this target within 100–700 ms of target onset to complete the trial; error trials were aborted without a reward and immediately repeated until correctly completed. On rewarded trials, a reward of constant size (250 ms solenoid open time) was delivered at 350 ms after the end of a correct saccade. On unrewarded trials, there was no juice reward, but an additional 600 ms postsaccade delay was applied to equate the total trial length across cue conditions.
The reward cues were abstract wireframe figures of distinct shape and color, approximately equated for size and luminance. Stimuli were scaled with retinal eccentricity to range from 1.5 to 3.0° in height and from 1.0 to 2.0° in width. The fixation point was a 0.5 × 0.5° square, and fixation was enforced within 2.5° of the fixation point and 3° of the saccade target. The fixation and saccade windows were constant across trials, so that accuracy requirements did not differ according to reward condition.
For neural recordings, electrode tracks were aimed to the lateral bank of the intraparietal sulcus based on stereotactic coordinates and structural MRI. Neurons were tested on the task if they had spatially tuned visual, delay, or presaccadic activity on a standard memory-guided saccade task (Oristaglio et al., 2006).
Statistical analysis and trial selection.
Incomplete trials (in which the monkey did not make a saccade to the target) were removed and not considered further. Although the analyses shown in the text include all correctly completed trials, we reanalyzed the data in two ways to validate our conclusion. First, we repeated the analyses by excluding trials that followed several successive failures (as sometimes happened for a negative cue). Second, we repeated the analyses by excluding the first two to four cue presentations, to control for nonstationarities during the session. These analyses produced equivalent behavioral and neural results. Therefore, the results shown in the text include included all completed trials to maximize statistical power.
All statistical analyses were preceded by tests of normality and symmetry (p < 0.05). If the data met the criteria of normality and symmetry, a paired-sample t test was used. If only the symmetry criterion was met, a Wilcoxon's signed-rank test was used. If neither criterion was met, a Mann–Whitney U test was computed. To evaluate population latencies for regressions, the nonparametric Fisher's signed-rank test was used.
Analysis of behavioral data.
Licking behavior was analyzed based on cumulative licking time during the delay period. Saccades were analyzed using a velocity-based algorithm (Nyström and Holmqvist, 2010). Reaction time (RT) was calculated from the earliest sample of continuous acceleration at every time point. Saccade accuracy was calculated from the last sample of continuous deceleration as (180 − d)/180, where d is the absolute angular distance in degrees (modulo 180), between the vectors representing the target and the saccade endpoint relative to fixation position. Corrective glissades were not considered, because their interpretation was confounded by the varying eccentricity of targets among the sessions.
Analysis of neural data.
Neural data were analyzed in several different ways. In the regression analyses, raw firing rates >50 ms windows sliding at 1 ms were computed separately for each neuron, over the relevant trial conditions.
Visual response latencies were computed based on the method of Bisley et al. (2004). The response of each neuron was partitioned into non-overlapping 2 ms bins. A normal distribution was fit to the 100 bins before cue onset (200 ms) and then compared with bins after cue onset. Each neuron was considered to have a visual response at the first of two consecutive bins that deviated from the distribution at p < 0.001.
Normalized firing rates were computed for each neuron as FR/(V − B), where V is the mean neural response over all conditions in a 300 ms window starting at the visual onset of each neuron, and B is the mean response in the 150 ms before cue onset. Analysis was always conducted on unsmoothed rates; however, for display purposes only, these normalized rates were convolved with the right half of a Gaussian kernel of 20 ms SD; this kernel assigned maximal probability to the time of spike occurrence and smeared the signal only forward in time, thus preventing an underestimate of the true latency. For the cross-trial analysis (see Fig. 6D), neural firing rates were fitted to a four-parameter linear regression computed in 50 ms bins, which included as regressors the reward, novelty, and location of the cue and the location of the target on the previous trial. Only pairs of completed trials were used. An additional analysis that included incomplete trials and a performance regressor showed equivalent results.
Results
Task
Two monkeys performed a task in which each trial began with a 50% previous probability of receiving a reward, and, after an initial period of central fixation, a peripheral cue provided full information about the outcome of the trial (Fig. 1). The cues were abstract colored patterns that flashed for 300 ms at one of two possible peripheral locations, which were randomly selected and marked by placeholders inside and opposite the RF of a neuron. Half of the cue patterns in each session signaled a positive outcome (a reward) and half signaled a negative outcome (a lack of reward). Cue novelty was manipulated orthogonally to reward associations. Within each reward class, half of the cues were highly familiar, having been trained with fixed reward associations for >10,000 trials before recordings began, and were shown in each session (Fam+ and Fam− cues). The remaining half was novel patterns that were shown in only one session for an average of 32.5 ± 0.19 presentations (minimum of 16; Nov+ and Nov−). Each cue class was represented by two visual patterns to rule out sensory effects, resulting in eight patterns that were randomly interleaved within a session.
Task design. The schematic shows the stages of a trial, including fixation, cue, delay, target, and outcome periods. Two placeholders (gray line patterns) remained stably on the screen throughout a trial block. A trial began when the central fixation point appeared and the monkey fixated it for a 300–500 ms period (Fixation). A cue was then presented for 300 ms at a randomly selected placeholder location (for simplicity, only 1 cue location is illustrated). The cue could fall into one of four categories depending on whether it was familiar (Fam) or novel (Nov) and signaled a positive (+) or a negative (−) outcome. In this and all subsequent figures, we use blue and red to indicate, respectively, rewarded and unrewarded trials and dark and light hues to indicate, respectively, novel and familiar cues. The cue presentation was followed by a fixed 600 ms delay period during which the placeholder display was reinstated and the monkeys had to maintain fixation (Delay). This was followed by the target period, when the fixation point was removed and one of the placeholders simultaneously brightened, instructing the monkeys to make a saccade to this target location (arrow). The target location was independently randomized and could fall either at the same (congruent) or opposite (incongruent) location relative to the cue (only trials with cues on the right are shown in this example). A correct saccade received the outcome predicted by the cue, a reward on Nov+ and Fam+ trials, but no reward on Nov− and Fam− trials.
Our goal was to examine whether the visual cues gain intrinsic salience, i.e., bias attention automatically, by virtue of their novelty or reward associations, even when they are not relevant to a required action. To this end, we structured the task so that the cue merely provided reward information; after viewing the cue, the monkeys had to make a saccade to a different target whose location was statistically independent from that of the cue. As shown in Figure 1, after the 300 ms cue presentation, the monkeys maintained fixation for a 600 ms delay period. At the end of this delay, one of the placeholders brightened, indicating the saccade goal, and the monkeys had to make a saccade to this target as quickly and accurately as possible (Fig. 1, target/saccade periods). Importantly, although the cue and target locations overlapped, they were independently randomized, so that the saccade goal was equally likely to be at the same or at the opposite location relative to the cue. This arrangement allowed us to determine whether the cues biased spatial attention by comparing saccades directed toward cue-congruent versus cue-incongruent locations. If saccades were facilitated for congruent versus incongruent locations, this would indicate that the cues automatically attracted attention, whereas if the saccades were impaired for cue-congruent locations, this would indicate that the cues automatically repelled attention from their location (Fecteau et al., 2004).
After the monkeys' saccade, the trial ended with the outcome that had been predicted by the cue. For positive trials (Nov+ and Fam+ cues), a correctly completed saccade was followed by a reward at a fixed interval of 350 ms after the saccade end. On Nov− and Fam− trials, no reward was given; however, any saccade error resulted in the immediate repetition of the trial, so that the monkeys had to complete these trials to progress in the task. Therefore, the optimal strategy in the task was to make a saccade to the target regardless of cue novelty or reward associations; the monkeys could not use the cue information to improve on the preordained reward outcome.
Novelty and reward produce motivational and spatial effects
Before examining the spatial biases produced by the cues, we tested whether the monkeys understood the significance of the cue by examining two nonspatial measures: (1) anticipatory licking; and (2) overall saccade RTs (pooled over congruent and incongruent configurations). Anticipatory licking was much more vigorous on positive versus negative trials for both familiar and novel items, showing that the monkeys quickly learned the reward significance of the novel cues (we discuss this result in more detail below). Consistent with this conclusion, saccade RTs were significantly shorter on rewarded versus unrewarded trials for both novel and familiar cues, suggesting that reward enhanced motivation (Table 1 and associated statistics in the figure legends). In addition, saccade RTs were shorter after novel relative to familiar cues, suggesting that novelty produced an arousing effect (Table 1 and associated statistics). The novelty effects were seen for each reward class tested separately, showing that they were not the result of a novelty × reward interaction. In sum, both cue novelty and reward produced global increases in arousal and motivation.
Nonspatial effects of novelty and reward
We next tested whether the cues biased attention in a spatial manner by comparing, for each cue class, saccades that were spatially congruent relative to incongruent with the cue location. Figure 2, A and B, illustrates this analysis after subtracting the mean RT and accuracy within each cue class to remove global effects; analysis of the raw data showed equivalent results, and raw values for accuracy and RT for each individual monkey are given in Table 2.
The cues produce spatial biases that depend on novelty and reward associations Saccade RT (A) and accuracy (Acc; B) for the four cue types, segregated according to the spatial congruence between the saccade goal and the cue location. The four lines indicate the different cue classes with color conventions as in Figure 1. The symbols show mean ± SEs across all correctly completed trials collected during neural recordings (6640 trials from monkey 1 and 13,163 trials from monkey 2). Values are normalized by subtracting the mean for each cue type to remove nonspatial effects of novelty and expected reward.
Congruence effects
The cues produced automatic biases in spatial attention that depended on their novelty and reward associations. Positive cues exerted a mild attentional attraction, shown by a slightly but significantly increase in saccade accuracy on congruent relative to incongruent configurations (without an effect on RT; Fig. 2A,B, left column; Table 2). In contrast, negative cues strongly impaired saccades toward their location (Fig. 2A,B, right). This repulsion was familiarity dependent and was much stronger for Fam− relative to Nov− cues. For Fam− cues, RTs were significantly higher and accuracy was lower at congruent relative to incongruent locations in both the full dataset and each individual monkey (Fig. 2A,B, pink traces; Table 2). For Nov− cues, there were no significant congruence effects in the full dataset (Fig. 2A,B, dark red traces; Table 2). As shown in Table 2, this result was attributable to a mixture of mild attraction in monkey 1 and mild repulsion in monkey 2, an individual variation that was correlated with the LIP response as we discuss below (see Fig. 7D). Therefore, although the cues were uninformative for the subsequent saccades, they exerted automatic spatial biases that depended on their novelty and reward associations. Cues that were novel or had positive associations produced no spatial bias or a mild attraction, whereas familiar negative cues produced strong repulsion from their visual field location.
Motivational and spatial effects are learned on different timescales
The finding that Nov− cues did not produce spatial repulsion (Fig. 2A,B) is surprising given our initial observations that the monkeys did show differential global effects related to these cues (i.e., lower anticipatory licking and higher overall RT for Nov− vs Nov+ cues). This suggests that novelty and reward may affect different processes on different timescales: for a novel negative cue, motivation may decline rapidly but salience may remain high for multiple presentations. To explicitly examine this idea, we examined the evolution of anticipatory licking and saccade congruence effects (differences in RT and accuracy between congruent and incongruent configurations) as a function of cue exposure during a trial block (Fig. 3A–C).
Motivational learning is fast and oculomotor learning is slow. Learning of cue–reward associations as a function of the number of cue exposures during a session, as measured in anticipatory licking (A) and the saccade congruence effects (B, C). Congruence effects are defined as the difference between RT (B) and accuracy (C) on congruent (C) and incongruent (IC) trials and plotted so that negative numbers indicate a disadvantage at the congruent location (i.e., IC − C for RT and C − IC for accuracy). Each symbol shows the mean ± SEM across 118 values (in which each value is the average for one pattern per session per cue class, i.e., 2 patterns for 59 sessions). The asterisks indicate the results of statistical comparisons between the Nov− and Nov+ cues (top row) and between the Nov− and Fam− cues (bottom row). Comparisons were made by z-scoring the values across the two cue classes, calculating the difference between the z-scores in each class, and bootstrapping the difference (resampling with replacement 10,000 times). Differences were classified as significant (asterisks) if the 95% confidence interval of the bootstrapped distribution did not include 0 (p < 0.05). D, Learning indices for anticipatory licking and saccade congruence effects, defined as the normalized change (difference in z-scores) between exposures 1–2 and exposures 15–16 to a Nov− cue (for details, see Results). The bars show average z-score differences, and the error bars show bootstrapped 95% confidence intervals. Acc, Accuracy.
As expected, based on their long training with the familiar cues, the monkeys showed clear differences between Fam+ and Fam− patterns in both their anticipatory licking (Fig. 3A, pale blue vs pink traces) and their saccade congruence effects (Fig. 3B,C). In contrast, for the novel cues, the monkeys showed fast learning in the licking response but very little learning in the congruence response. As shown in Figure 3A, the monkeys started out by licking vigorously for all novel patterns, but the response to a Nov− cue quickly extinguished and reached low levels comparable with those for Fam− cues after the eighth presentation (dark red trace and bottom row of stars). In contrast to the rapid habituation of the licking response, spatial congruence indices for the Nov− cues remained close to 0 and significantly higher than those for Fam− cues throughout the course of the session for both RT and accuracy (Fig. 3B,C). We limited this main analysis to the first 16 exposures to ensure that all the sessions contributed to each time point; however, we obtained equivalent results when we examined the results up to 32 exposures, the average number of presentations per pattern (although the reliability of this latter analysis is limited by the diminishing number of data points).
To directly compare the rate of learning in the two measures, we estimated the habituation of the Nov− response by computing the differences between the first two and the last two Nov− presentations (presentations 1–2 vs 15–16). We first z-scored the values in the pooled (early–late) distributions (separately for the licking and congruence measures). We then computed the differences between the z-scores in the early and late presentations and used bootstrapping (resampling with replacement 10,000 times) to estimate the reliability of the difference. As shown in Figure 3D, the licking response showed a large and significant change, on the order of 0.5–1 SDs for each monkey and in the combined data (because we plot the difference between the early and late presentations, a decline in licking registers as a positive difference). In contrast, the changes in the congruence effect were close to 0 and significantly smaller than those in the licking response (no overlap in the 95% confidence intervals). Note that these differences in learning cannot be attributable to the different scales that measure the licking and congruence effects, because by z-scoring we expressed these effects on a standardized scale (as the change relative to the mean relative and SD of each distribution). In addition, by comparing the first two and last two Nov− presentations (rather than, e.g., the first and second half of trials), we maximize our chances of detecting a change if one were truly there. Finally, by using an analysis based only on the Nov− cues, we avoid artifacts related to idiosyncratic features of other cue classes, i.e., exclude the possibility that the effects are attributable to overtraining for the Fam− cues. In summary, learning for the novel cues proceeds at different rates for different behavioral measures, with a rapid change in motivational reactions and a much slower decline in the salience response.
LIP neurons show converging but distinct effects of novelty and reward
To see how oculomotor cells encode novelty and reward, we recorded the activity of 59 LIP neurons (21 in monkey 1, 38 in monkey 2). We selected cells that had spatially tuned visual responses and sustained activity on a memory-guided saccade task (see Materials and Methods), because these cells were implicated in salience computations for attention and eye movement control (Bisley and Goldberg, 2010).
LIP cells showed enhancement by both novelty and expected reward. When the cues appeared in their RF (Fig. 4, top row), LIP neurons had visual transient and sustained postcue responses that were stronger for positive relative to negative cues (see below and Peck et al., 2009). In addition, responses were stronger for novel relative to familiar patterns. A novelty effect was seen for both reward classes (100–300 ms after cue onset, Mann–Whitney U test, p < 0.001 for both positive and negative cues), showing that it was not the result of a novelty × reward interaction. The novelty enhancement was specific for cues within the RF (Fig. 4, top vs bottom rows). If the cues appeared outside the RF, novelty had an inconsistent influence, with no effect for positive cues (Fig. 4, bottom panels; 150–300 ms after cue onset, Mann–Whitney U test, both monkeys, p = 0.35; monkey 1, p = 0.82; monkey 2, p = 0.41) and reduced firing for negative cues (both monkeys, p < 10−5; monkey 1, p < 10−4; monkey 2, p = 0.03). Therefore, the cells did not reflect a global novelty-related increase in arousal or motivation but an enhancement in the spatial encoding of a novel cue.
LIP responses show spatially specific novelty enhancement that persists during a recording session. Normalized population activity (mean ± SEM, n = 59 cells) for cues appearing within the RF (top row) or at the opposite location (bottom row). The thick horizontal bar shows the 300 ms cue presentation.
To examine the convergence of novelty and reward in individual cells, we fitted the trial-by-trial response of each cell to an RF cue with a bivariate linear regression:
where FR is the number of spikes recorded in a sliding window aligned on cue onset (50 ms width, 1 ms step), and β0–β2 are fitted regression coefficients. Novelty and Value were coded as dummy variables of 0 and 1, such that positive coefficients indicate enhancement for novel or positive cues.
The neural population (Fig. 5A) showed enhancement by both novelty and expected reward, and nearly every individual cell showed the effect of both factors (Fig. 5B). The results were equivalent after including an interaction term, showing that the effects could not be attributable to a spurious novelty × reward interaction (e.g., a novelty effect only for the negative or only for the positive cues). The individual cell results are based on an effect criterion of two consecutive bins with a coefficient significant at p < 0.05 (Fig. 5B, black dots) but remained robust for other criteria [e.g., five consecutive bins, or using a nonregression latency analysis; note that these criteria are not subject to multiple comparison error because they involve an “and” rather than an “or” rule (i.e., significance must be achieved in all bins to meet the criterion); thus, a criterion involving five bins is more conservative than one requiring two bins]. Across the population, the novelty and reward effects became significant at, respectively, 70 and 76 ms (Fig. 5A, the first of two consecutive bins showing p < 0.001 by the Fisher's signed-rank test), and in individual cells, the median latencies were 105 ms for novelty versus 113 ms for reward (Fig. 5B; Mann–Whitney U test, p = 0.003 for both monkeys; p = 0.08 for monkey 1, p = 0.008 for monkey 2). Although the reward effects arose somewhat later, they were more sustained during the delay [Fig. 5A, inset; average duration (based on the number of significant bins) was 199.4 ± 15.8 ms for novelty and 293.5 ± 24.3 ms for reward; p = 0.0001 for both monkeys, p = 0.04 for monkey 1, and p = 0.0001 for monkey 2]. Note that these effects could not be attributable to spurious sensitivity to the visual patterns, because they were sustained during the postcue delay and were robust over a large set of patterns (256 novel and eight familiar patterns over the two monkeys). Thus, novelty and reward have separable effects even as they impinge on a common population of spatially tuned cells.
Sensitivity to novelty and expected reward. A, The average regression coefficients estimating sensitivity to novelty (orange) and reward (green). Coefficients were computed for each cell in a sliding window during the cue and delay periods (50 ms window, at 1 ms step), and traces show the mean ± SEM of the time-resolved coefficients across all 59 cells. The value for each time bin is plotted in the middle of the 50 ms window (i.e., 0–50 ms is plotted at 25 ms). The inset compares the duration of the novelty and reward effects and shows that the duration of the reward effect is significantly longer (p < 0.001). Effect duration was defined as the percentage of the time between cue onset and target onset that each neuron was sensitive to value and novelty, respectively. Each point represents one cell, and the diagonal line is the equality line. B, Color maps showing the time course of the novelty regressor (top) and the reward regressor (bottom) for each individual cell. Cells are sorted independently in each panel according to the latency of the effect (black dots).
Neurons show slow habituation of responses to novel negative cues
The population responses shown in Figure 4 suggest that LIP neurons had suppressed delay period responses after a Fam− cue but did not show this suppression for a Nov− cue. Thus, the cells may show a slow habituation of the Nov− response and encode the spatial congruence effects in the monkeys' saccades. To examine these ideas, we analyzed the time course of habituation of the Nov− response and the correspondence between the spatial encoding in LIP and the congruence bias in the monkeys' saccades.
To examine the time course of neural learning, we plotted the LIP visual and delay period responses as a function of the number of cue presentations (Fig. 6A,B). For the familiar cues, visual and delay responses were much lower for Fam− than for Fam+ patterns throughout a session (Fig. 6A,B, pink vs pale blue traces). However, the responses to the Nov− cues remained significantly higher than those to Fam− cues at many points in the session (bottom row of asterisks) and showed only a mild decline during the course of a session. To measure the extent of learning, we used a difference analysis similar to that for the behavioral response. As shown in Figure 6C, the z-scored differences between the first two and last two Nov− presentations revealed no significant change in the visual response. Although some learning was seen in the delay response, this was not statistically different from that for the visual response and was only on the order of 0.3 SDs.
LIP neurons show slow adjustments in salience but fast cross-trial reward effects. Normalized responses as a function of cue exposures, during the visual epoch (A, 150–300 ms after cue onset) and late delay period (B, 750–900 ms after cue onset) for cues shown in the RF. Conventions are the same as in Figure 3A–C. Normalized firing rates were computed for each neuron as FR/(V − B), where V is the mean neural response over all conditions in a 300 ms window starting at the visual onset of each neuron, and B is the mean response in the 150 ms before cue onset. Symbols show mean ± SEM of the normalized visual response (with 118 points per exposure, corresponding to 2 patterns for 59 sessions for each cue class). The top row of asterisks represents a significant difference (bootstrap analysis, p < 0.05) between Nov− and Nov+ cues, and the bottom row shows significant differences between the Nov− and Fam− cues. C, Learning indices for the visual and delay responses, defined as the normalized change (difference in z-scores) between exposures 1–2 and exposures 15–16 to a Nov− cue (for details, see Results). D, Regressors indicating the effect of prior trial rewards, calculated in a sliding window aligned on cue onset. The bars show average z-score differences, and the error bars show bootstrapped 95% confidence intervals.
The slow habituation of the Nov− response did not indicate a general inability of LIP cells to learn on faster timescales. This was shown by the fact that the cells did have significant sensitivity to rewards delivered in the previous trial (Fig. 6D). A regression analysis showed that firing rates before cue onset were enhanced by the delivery of a reward on the previous trial, an effect that was independent of other factors (i.e., previous trial cue position, cue novelty, or target position; see Materials and Methods) and was significant in each individual monkey (Wilcoxon's test, regression coefficient relative to 0, 150–0 ms before cue onset; monkey 1, p = 0.01; monkey 2, p = 0.0001; both, p < 10−6). The previous trial reward effect vanished after cue presentation (150–300 ms, all p > 0.5), suggesting that it was erased by current trial events. Thus, the slow habituation of the Nov− response does not reflect a general lack of fast learning in LIP cells but a slow habituation of the Nov− salience response.
LIP cells reflect the slow extinction of orienting for a negative cue
Having shown that the Nov− responses are slowly habituating, we next examined how the spatial selectivity of the neurons for the cue and target locations correlated with the congruence biases in the monkeys' saccades. To measure encoding of the cue and target locations, we fitted firing rates with two equations:
In both equations, FR is the number of spikes of an individual cell in a 50 ms window sliding by 1 ms aligned on cue onset, and β1 and β2 are fitted coefficients estimating the sensitivity for, respectively, cue and target location. For Equation 2, we used all trials for a given cue class regardless of the target location. For Equation 3, we further separated the trials for each cue class into those with congruent and incongruent target configurations. In each case, we coded CueLoc (TarLoc) as dummy variables of 1 if the cue (target) appeared in the RF and 0 otherwise. Thus, a positive β1 (β2) coefficient indicated positive spatial bias, with stronger firing if the cue (target) was in the RF.
As shown in Figure 7, A and B (main panels), the cells had cue location coefficients that were high and positive during the visual epoch and remained significant (positive or negative) during the delay. Therefore, LIP cells maintained a spatial memory of the cue location, although this location was irrelevant to the task. Because of this memory trace, the cells had a preexisting spatial bias favoring cue-congruent or cue-incongruent locations even before the onset of the target itself. This is shown in the insets for Figure 7, A and B, which plot the spatial encoding aligned on target presentation, cue-congruent and cue-incongruent configurations. For the Nov+ and Fam+ cues, the sustained postcue response translated into a bias toward congruent and away from incongruent target locations (positive congruent coefficients and negative incongruent coefficients). A two-way ANOVA showed a main effect of congruence in the pretarget response that was highly significant in the combined data (two-way ANOVA, p = 10−8) and in each individual monkey (Fig. 7C, top row; monkey 1, p = 10−5; monkey 2, p = 10−5; effects of novelty or congruence × novelty interaction, all p > 0.1). In contrast, for the negative cues, the cells showed a congruence × novelty interaction (Fig. 7C, bottom row, p = 10−6 for each monkey; main effects of congruence and novelty, all p > 0.05). For the Fam− cues, the sustained cue-evoked suppression produced a strong bias away from congruent locations (Fig. 7B, bottom inset), whereas the Nov− cues produced mild spatial attraction similar to the positive cues. Thus, the pretarget spatial biases in LIP mirrored the congruence effects in the monkeys' saccades (Fig. 2).
LIP neurons show cue-evoked spatial biases that correlate with saccade congruence effects. A, The main panels show regressors indicating selectivity for the cue location for Nov+ and Fam+ cues. The insets show selectivity for the target location aligned on target onset, separately for congruent (C, colored) and incongruent (IC, gray) target locations. All traces show mean ± SEM of the regression coefficients across 59 cells. B, Same as A for negative cues. C, Average and SEM of spatial coefficients for the target location, during the pretarget epoch split by monkey. D, The congruence effect in the neural response (ordinate, the difference between the pretarget bias for congruent and incongruent trials) plotted against the congruence effect in RT (left) and accuracy (Acc; right) separately by monkey. The numbers in the top right corner show the Spearman's rank correlation across all classes, along with the associated significance level.
As seen in Figure 7C, the attractive neural bias toward a Nov− cue was slightly stronger in monkey 1 relative to monkey 2 (slightly larger difference between congruent and incongruent configurations). This small neural difference correlated with the across-monkeys difference in congruence effects (Fig. 7D) in terms of RT (r = −0.24, p = 0.007) and accuracy (r = 0.22, p = 0.016). Specifically, monkey 2 showed a mild saccadic repulsion from the Nov− cues, showing that it had some learning of the negative salience of these cues (Table 2). In contrast, monkey 1 showed saccadic attraction toward a Nov− cue that was significant in both accuracy and RT (Fig. 7D, dark red circle; Table 2), suggesting that this monkey had a stronger novelty effect.
A final noteworthy aspect of this analysis is that the cue-evoked saccade biases were better encoded by the pretarget response than they were in the response to the target itself. During the post-target epoch, the neurons primarily encoded the target location (as shown by a strong increase in the location coefficient for all types of cues; Fig. 7, A and B, insets) and no longer showed strong congruence effects. A trend toward an incongruent bias for negative cues was significant in the pooled data (p = 0.02) but not for individual monkeys (p = 0.17 for monkey 1, p = 0.07 for monkey 2), and there was no correlation between the neural and saccadic congruence effects (RT, r = −0.14, p = 0.11; accuracy, r = 0.07, p = 0.41). This result seems counterintuitive, because saccade metrics are better correlated with a neural response that occurs earlier in time rather than with the target response that immediately precedes the saccade. However, it may be explained by the fact that LIP encodes relatively early stages of planning and target selection, and its saccade responses may reflect feedback from downstream motor areas (Ganguli et al., 2008; Suzuki and Gottlieb, 2013) that may be influenced by additional factors and relatively dissociated from the final saccade.
Novelty is encoded for task-irrelevant items
Although the cues in the standard task were not informative for the monkeys' saccades, they did convey task-relevant reward information, and it was possible that the novelty effects were gated by this relevance. To evaluate this possibility, we tested a subset of cells (n = 37) in a separate set of “probe” trials in which previously trained novel or familiar patterns were presented as salient distractors that were no longer informative about the reward of the trial.
The probe test was conducted in a separate block of trials that followed data collection on the main task (i.e., after an average of 32 presentations of a novel pattern). As shown in Figure 8A, probe trials were very similar to those in the main task, with two exceptions. The cue and target appeared at a single location opposite the RF, drawing the monkeys' attention as much as possible away from the RF. In addition, simultaneous with target onset, an irrelevant visual probe was flashed for 80 ms at the RF location. The probe was drawn from the set of novel and familiar cues that had been shown in the preceding block, but its identity was randomized independently of the initial cue, so that it conveyed no reward information. Analysis of anticipatory licking showed that, as intended, the monkeys predicted reward based only on the first cue and not on the probes. Licking was measured during the 350 ms delay between the saccade and the reward onset, well after probe presentation; nevertheless it was not affected by the reward associations of the probe and depended solely on those of the initial cues (two-way ANOVA, p = 10−7 for cue effect; p = 0.89 for probe and cue × probe interaction).
Novelty encoding for task-irrelevant stimuli. A, Stages of the probe task. A single placeholder remains stably opposite the RF, and the initial reward cue and target appear at this location, similar to the main task. Simultaneous with the offset of the fixation point and target onset, a task-irrelevant probe cue is flashed in the RF. The probes have reward associations by virtue of training in the preceding trial block but are irrelevant in this condition, because the reward is controlled by the first (valid) cue. B, Normalized neural responses to the probes (mean ± SE across cells). The thick horizontal line denotes probe duration. C, Time course of the regressors describing sensitivity to novelty and reward on probe trials. Note that the regression fits included an interaction term, although for simplicity, this term is omitted from the figure. Conventions are the same as in Figure 5.
Although the probes were irrelevant for predicting rewards, they were visually salient and elicited an LIP visual response. This response was influenced by novelty and expected reward in a manner analogous to that on standard trials (Fig. 8B,C). The novelty effect appeared earlier on probe versus standard trials (median latencies of 87 vs 105 ms for the standard trials), most likely because of the different sensorimotor conditions (i.e., absence of an RF placeholder and simultaneous preparation of a null-direction saccade) that are known to affect LIP cells (Bisley et al., 2004). Nevertheless, effects of novelty and reward were found in the vast majority of cells (novelty, 35 of 37 cells and value, 30 of 37 cells; Fig. 8C). Consistent with this result, saccade RTs were significantly longer in the presence of novel relative to familiar probes (213 ± 1.2 vs 204 ± 1 ms, Mann–Whitney U test, p < 10−7), as would be expected if novel stimuli were more salient and interfered more strongly with the oppositely directed saccade. Thus, novelty-induced salience persists in the bottom-up visual response even for stimuli that are entirely irrelevant to the task.
Discussion
Behavioral studies in humans showed that novelty attracts attention and gaze (Brockmole and Henderson, 2005; Yang et al., 2009), and single-neuron studies in monkeys revealed effects of novelty and familiarity in feature-selective ventral visual (Li et al., 1993; Barense et al., 2012; Peterson et al., 2012; Woloszyn and Sheinberg, 2012). We extend these findings by showing that novelty affects salience and saccade-related activity in a dorsal stream area and that these effects are distinct from those of expected reward. We discuss first the reward effects, followed by a discussion of the novelty modulations.
A key finding we report is that the reward associations of a visual cue automatically modify the salience and saccade biases that are afforded to that cue. Cues that bring “good” news automatically enhance visual responses and attract attention toward their location, whereas cues that bring “bad” news automatically inhibit visual responses and repel attention away from their location (Peck et al., 2009). In other words, the brain seems to modify its processing of sources of information according to the “message” it receives even when there is no opportunity for an active choice. This latter feature distinguishes our results from previous studies in which animals were allowed to choose and, by biasing their choices toward the better option, thus increasing their actual rates of reward (Sugrue et al., 2004; Leathers and Olson, 2012). In contrast, in our task, the saccade target was predefined and the reward of the trial was preordained, i.e., the cues merely conveyed information. Therefore, our results suggest that the same biases that have evolved to allow active choices modulate the allocation of cognitive resources for processing information.
Although these results seem unsurprising in view of the literature on economic choice, they are in fact remarkable from the point of view of attention, because biases based on stimulus–reward associations are not necessarily optimal for attention allocation. First, theoretical considerations show that attention would optimally depend on the reliability (informativeness) of a cue rather than its specific message, because a bad news message may be as important as one bringing good news (Dayan et al., 2000). Second, in natural environments, animals must often learn and sample information before knowing what reward to expect (Gottlieb et al., 2013). Therefore, both considerations suggest that attending and sampling information require drives that depend not only on reward associations but also on the informational properties of sensory cues (Dayan et al., 2000; Pearce and Mackintosh, 2010; Gottlieb, 2012; Gottlieb et al., 2013).
The novelty effects we report in this study, which enhanced salience independently of expected reward, may be part of these intrinsic drives. As shown by the saccade and LIP response patterns, novelty had a “protective effect,” i.e., it prevented the attentional suppression by a no-reward cue even after the monkeys understood the negative cue–reward associations. Comparisons between the rates of learning in the salience and anticipatory licking response suggest that novelty habituates at different rates for the visual and reward system. For the reward system, novelty quickly dissipates if a stimulus predicts a negative outcome, consistent with previous results (Kakade and Dayan, 2002; Laurent, 2008; Düzel et al., 2010). However, for the visual system, novelty acts on a longer timescale and can transiently enhance salience independently of reward expectations. Such a sustained novelty response may support learning that is partially independent of reward. For instance, even after one has learned that a cue predicts a negative outcome, one may want to continue attending to the cue to, for example, rule out remaining uncertainty about the reward or allow perceptual learning of its visual attributes (Sasaki et al., 2010; Grossberg, 1982). Therefore, a slowly habituating novelty response may be one mechanism by which the brain enables learning and information sampling to proceed in an intrinsically motivated (reward-independent) manner.
Before we can accept this conclusion, we must consider several alternative interpretations. One possible argument is that the strongest effect in our task was the suppression of the Fam− cues, and the novelty enhancement we report may have been a mere artifact of this suppression. However, this idea is refuted by the fact that novelty produced higher motivation and visual responses not only for negative but also for positive cues (Figs. 4, 5; Table 1), showing that it had bona fide, reward-independent effects. In addition, our comparison of the learning rates for the licking and salience response was based solely on Nov− cues and could not have been an artifact of the Fam− patterns. Therefore, neither the novelty enhancement nor the distinct learning time course could be explained away as an interaction between extensive training and expected reward.
A second possible interpretation is that the Fam− suppression is explained by inhibition of return (IOR), a well known mechanism for suppressing saccades to recently attended locations (Fecteau et al., 2004; Bays and Husain, 2012). However, whereas IOR is triggered by spatially nonpredictive rewarded cues (Fecteau et al., 2004), in our data, positive cues produced only saccadic attraction (Fig. 2). In addition, IOR develops after a single saccade without the need for practice (Bays and Husain, 2012), whereas in our task prolonged training was required to evoke suppression for the negative cues.
A third and final question is whether the sustained novelty-based salience we find may reflect a residual belief, or “hope,” on the part of the monkeys that the Nov− cues might bring a reward. Clearly, the monkeys' rapid decrease in licking and slowing of the saccadic response argue against an immediate reward expectation and is consistent with the fact that, in choice paradigms, monkeys quickly learn to avoid a negative outcome (Sugrue et al., 2004; Leathers and Olson, 2012; Yasuda et al., 2012). However, it can be argued that the brain must have harbored some motivation to enhance the salience of the Nov− cue. However, we propose that this form of motivation is better described as an intrinsic motivation related to novelty or uncertainty, and, as we show here, it is distinct from the type of motivation that depends on physical rewards.
The slow time course of salience learning that we report is consistent with a recent report of slow reward learning in the substantia nigra pars reticulata (SNr) (Yasuda et al., 2012) and with the recent proposal that the basal ganglia serves as a large-capacity store for overlearned, long-term reward associations (Hikosaka et al., 2013). Our results suggest that LIP may communicate with, but may not be a part of, this long-term memory system. Although the SNr cells did not encode rapid changes in preference in a decision task, LIP neurons reflected both slow and faster trial-to-trial learning mechanisms (Fig. 6), suggesting that they encode learning on multiple timescales (Sugrue et al., 2004; Lee et al., 2012). Second, the neural and saccadic suppression for the Fam− cues was not fully developed on the first exposure but emerged rapidly over the first few presentations (Figs. 3B,C, 6). This suggests that reward associations are not stored in the oculomotor system but may be “uploaded” to it from a different area in a context-dependent manner.
Footnotes
The work was supported by National Institutes of Health Training Grants EY013933-10 and T32 MH015174-35 (N.C.F.) and National Eye Institute Grants R01 EY014697-01 and R24 EY015634 (J.G.). We thank Latoya Palmer for expert administrative assistance, Girma Asfaw for superior veterinary care, and the members of the Mahoney Center who read and had helpful suggestions on this manuscript.
The authors declare no competing financial interests.
- Correspondence should be addressed to Dr. Jacqueline Gottlieb, Department of Neuroscience, Columbia University, 1051 Riverside Drive, Kolb Research Annex, Room 569, New York, NY 10032. jg2141{at}columbia.edu
