Abstract
When searching a visual scene for a target, we tend not to look at items or locations we have already searched. It is thought that this behavior is driven by an inhibitory tagging mechanism that inhibits responses on priority maps to the relevant items. We hypothesized that this inhibitory tagging signal should be represented as an elevated response in neurons that keep track of stimuli that have been fixated. We recorded from 231 neurons in the frontal eye field (FEF) of 2 male animals performing a visual foraging task, in which they had to find a reward linked to one of five identical targets (Ts) among five distractors. We identified 38 neurons with activity that was significantly greater when the stimulus in the receptive field had been fixated previously in the trial than when it had not been fixated. The response to a fixated object began before the saccade ended, suggesting that this information is remapped. Unlike most FEF neurons, the activity in these cells was not suppressed during active fixation, had minimal motor responses, and did not change through the trial. Yet using traditional classifications from a memory-guided saccade, they were indistinguishable from the rest of the FEF population. We propose that these neurons keep track of any items that have been fixated within the trial and this signal is propagated by remapping. These neurons could be the source of the inhibitory tagging signal to parietal cortex, where a neuronal instantiation of inhibitory tagging is seen.
SIGNIFICANCE STATEMENT When we search a scene for an item, we rarely examine the same location twice. It is thought that this is due to a neural mechanism that keeps track of the items at which we have looked. Here we identified a subset of neurons in the frontal eye field that preferentially responded to items that had been fixated earlier in the trial. These responses were remapped, appearing before the saccade even ended, and were not suppressed during maintained fixation. We propose that these neurons keep track of which items have been examined in search and could be the source of feedback that creates the inhibitory tagging seen in parietal cortex.
- eye movement
- frontal eye field
- inhibition of return
- inhibitory tagging
- putative tracking neurons
- visual search
Introduction
When searching a visual scene for a target, our eyes tend to go to items that are similar to the target or locations where the target is likely to be. We, among others, have suggested that the brain creates priority maps (Fecteau and Munoz, 2006; Serences and Yantis, 2007; Bisley and Goldberg, 2010; Zelinsky and Bisley, 2015; Shomstein and Gottlieb, 2016; Ferrante et al., 2018), also termed saliency maps (Itti and Koch, 2001; Purcell et al., 2012), that encode the relative importance of each location in space in terms of bottom-up salience and a multitude of top-down factors, often driven by the goals of the behavior. Conceptually, the eyes go to the highest point on the map. To make sure that the eyes continue to scan the scene until the target is found, it has been suggested that a peak will be inhibited once the item in that location has been examined (Koch and Ullman, 1985; Itti and Koch, 2001). This has been called inhibition of return, based on the behavioral results of Posner and colleagues (Posner and Cohen, 1984; Posner et al., 1985), who found that reaction times to a previously attended location are slowed. Klein and colleagues (Klein and MacInnes, 1999; Wang and Klein, 2010) have shown behavioral correlates of this sort of inhibitory tagging during visual search, and we have shown a behavioral (Shariat Torbaghan et al., 2012) and neuronal (Mirpour et al., 2009) instantiation of this in the lateral intraparietal area (LIP), where responses to objects that have been fixated are reduced.
The presence of an inhibitory tagging mechanism would suggest that there should be a representation of items that have been fixated, which would drive this inhibition. Hasegawa et al. (2004) found neurons within the frontal eye field (FEF) and in an area anterior to the FEF that preferentially responded following a “don't look” cue in a spatial nonmatch to sample task. We hypothesized that, if these neurons play a role in indicating where an animal should not look, then perhaps they might naturally indicate which stimuli should not be looked at in search because they have already been fixated. As such, we examined the responses of neurons in FEF to attempt to identify neurons that could be playing a role in keeping track of which items in a scene have been fixated. We should note that our experimental procedure does not allow us to differentiate between keeping track of the locations or the objects in the locations. We only refer to the tracking signal relative to items, rather than locations, for the sake of simplification.
If such putative tracking neurons exist in FEF, then we predict that they should have an important property in addition to preferentially responding to items that have been fixated before: they should not be suppressed during maintained fixation. We recently showed that the responses of most FEF neurons are almost at baseline levels when animals maintain fixation on a stimulus that is likely to give them a reward (Mirpour et al., 2018). We suggested that this suppression is a way of flattening the priority map and, thus, keeping the eyes from moving away from the potentially rewarding stimulus. If the activity in putative tracking neurons is there to keep the eyes from looking at objects that have already been fixated before, then the suppression of that signal would be counterproductive. Therefore, we predicted that the elevated activity in response to a previously fixated stimulus should not be suppressed during maintained fixation in putative tracking neurons.
To test these hypotheses, we recorded from neurons in the FEF of two rhesus macaques while they performed a visual foraging task. We found that the activity of a subset of neurons matched our predictions and that the timing of the tracking signal was consistent with the remapping of this activity across saccades, which could explain how it is propagated across eye movements.
Materials and Methods
All experiments were approved by the Chancellor's Animal Research Committee at University of California–Los Angeles as complying with the guidelines established in the Public Health Service Guide for the Care and Use of Laboratory Animals. Neuronal and behavioral data were collected from 2 male rhesus macaques (8–12 kg). The animals were implanted with head posts, scleral coils, and recording cylinders during sterile surgery under general anesthesia (Mirpour et al., 2009); animals were initially anesthetized with ketamine and xylazine and maintained with isoflurane. Surgery was conducted using aseptic techniques, and analgesics and antibiotics were provided during postoperative recovery. The data in this paper were previously analyzed to look at fixation suppression (Mirpour et al., 2018), so the general methods are similar.
Behavioral tasks.
The animals were trained on a memory-guided saccade task (MGS) and a foraging visual search task (Mirpour et al., 2009, 2018; Mirpour and Bisley, 2012). To begin a trial of the MGS, the animals fixated a central spot for 300–500 ms, after which a peripheral target was flashed for 200 ms. After the target was extinguished, the animal had to remember the location of the target for 600 ms, after which the fixation point was extinguished and the animal had 450 ms to make a saccade to the remembered location of the target. If the animal landed and remained within 2° of the target location, the target reappeared, after which the trial ended and the animal was rewarded.
To begin a trial of the foraging task (Fig. 1), the animals fixated a spot placed to one side of the screen. After a delay of 350–370 ms (uniformly distributed), an array of five potential targets (T) and five distractors (+) were presented, with one over the fixation spot. All stimuli were 1.2° × 0.8°. One of the Ts had a juice reward associated with it (the target), such that, if the animal fixated within 1.5° of it for 500 ms, he would get the reward. After the array appeared, trials were only terminated if the animal did not fixate the target within 8 s. In each session, 10 locations were used and the stimuli were arranged in such a fashion that, when the animal fixated one stimulus (Fig. 1, small circle), the receptive field (RF) of an FEF neuron encompassed no more than one other stimulus (Fig. 1, large oval). On each trial, the locations of the five distractors and the five Ts, including the target, were randomly assigned among the 10 spatial locations.
Behavioral task. In each trial, 5 potential targets (T) and 5 distractors (+) were presented. One T (the target) had a fluid reward linked to it, such that, when the animal fixated it for 500 ms, he obtained the reward. The stimuli were arranged so that, when looking at one stimulus (small black circle), another stimulus was centered in the FEF neuron's RF (black oval).
Electrophysiological recording.
We recorded extracellular single-unit activity from FEF using tungsten microelectrodes inserted through a guide tube into the anterior bank of the arcuate sulcus. To identify FEF, we used MRI images, and we confirmed that each track was in FEF by evoking saccades using low-current microstimulation. Microstimulation was done while animals performed a blink task (Sommer and Wurtz, 2000), with a 70 ms train of biphasic pulses, negative first, 0.2 ms width/pulse phase, delivered at a frequency of 330 Hz. We typically used a current of 50 μA to elicit saccades, and mean ± SD saccadic latency was 50 ± 12 ms. Neurons were recorded if they showed increased activity during the visual, memory, or movement stage of the MGS. Consequently, fixation neurons (Bizzi, 1968; Suzuki and Azuma, 1977; Izawa et al., 2009) were excluded from this study. The size and position of the RF of each neuron were mapped using an automated MGS task covering 9 and then 25 locations (for details, see Mirpour et al., 2010). Neurons were also excluded from the study if their RFs were so large that they would encompass two stimuli in the array. RF centers ranged from 2.8° eccentricity to 15° eccentricity, and RF sizes ranged from 1.25° to 6.5° radius in the horizontal direction and 1.25° to 4° radius in the vertical direction. After mapping, the foraging task was run and neuronal data were recorded.
Experimental design and statistical analysis.
Neuronal data were recorded from 231 FEF neurons (78 from Animal 1 and 153 from Animal 2). We roughly discriminated action potentials online and then accurately sorted spikes offline using the Offline Sorter software (Plexon). The experiments were run using the REX system (Hays et al., 1982), and data were recorded using the Plexon system. Data were analyzed using custom code written in MATLAB (The MathWorks). Eye position signals were sampled using a magnetic search coil system (DNI) at 2 kHz and recorded for analysis at 1 kHz.
When sufficient trials were available from the MGS mapping protocol, we used the definitions from Bruce and Goldberg (1985) to categorize neurons as visual, visuomovement, and movement neurons. We compared the visual response (50–150 ms after target onset) and the movement response (in the 50 ms before saccade onset) with a baseline response (100 ms before target onset) from that trial. We used paired t tests at the p < 0.01 level to indicate significance. Neurons were categorized as visual neurons if only the visual response was significantly higher than the baseline response. Neurons were categorized as visuomovement neurons if both the visual and movement responses were significantly higher than the baseline response. And neurons were categorized as movement neurons if only the movement response was significantly higher than the baseline response. We also characterized the neurons as having delay activity if the response 100–400 ms after target offset was significantly higher than the baseline response.
For the visual foraging task, neuronal data were analyzed from correct trials. To analyze these data, we first separated trials down into fixations in which there was a single object inside the RF. Data were aligned by the beginning of fixation or by the onset of the saccade, using an eye-velocity detection algorithm to detect the saccades. Unless otherwise stated, data were only analyzed from fixations in which the saccade was made away from the RF. Data are presented as spike density functions (Richmond et al., 1987) using an SD of 10 ms. Before averaging across neurons, the spike density functions for each neuron were normalized by dividing the activity by a normalizing factor. The factor was calculated for each neuron from all fixations that occurred after the first saccade, that lasted at least 300 ms and that had a T at the fovea and a distractor in the RF. The window for the calculation was 150 ms long starting 150 after fixation onset. When plotting the spike density functions, we plot the mean and SEM, where the N is the number of neurons. When statistically comparing spike density functions as a function of time, we used paired t tests when data from the same sets of neurons were compared and Wilcoxon rank sum tests when the data were from different populations of neurons. For specific statistical comparisons, we compared spike rates in set windows. For maintained fixation, we used a 350 ms window starting 50 ms after fixation onset. To compare activity when a T or distractor was at the fovea, we used at 100 ms window starting 50 ms after fixation onset and limited the analyses to fixations lasting at least 150 ms. To aid in visualization, we present the square root of the spike rates in the scatter plots.
For the decoding analysis, we implemented a maximum a posteriori (MAP) algorithm similar to Brown et al. (1998; for a theoretical description of the method, see Dayan and Abbott, 2005, chapter 3, p. 103). MAP inference generally produces estimates that are a good representation of the neural population decoding and are not biased by a single neuron contribution. The model output also accurately represents the information content of the neural population, which ranges from the distribution of the prior probabilities, which is effectively chance, to optimal discrimination. For each iteration of the analysis, we divided the data into two sets, for training and decoding, respectively. For each neuron, the parameters (the mean and SD of the firing rate) of the response distributions for each of 3 conditions (unfixated T in RF, fixated T in RF and D in RF) were calculated based on the training data. We then randomly, with repetition, took 50 fixations per condition from the decoding dataset from each neuron. For each fixation, we calculated the probability of the response given each condition and multiplied it by the prior probability of that condition. The prior probability of the condition was the proportion of the trials in which that condition appeared in the training dataset. We then used the maximum log-likelihood of the response conditional probabilities of the population to predict the object in the RF and, for all 50 fixations, to calculate the proportion of correct predictions. This yields a single proportion correct for the population of neurons. For each iteration, we tested the population of 38 putative tracking neurons and 1000 randomly selected populations of 38 neurons, selected without replacement from the entire population of 231 neurons. We repeated this process 1000 times using new training and decoding data for each iteration. This resulted in a set of 1000 predictions from the putative tracking neurons and 1000 sets of 1000 predictions for the randomly selected neurons.
Results
Behavior
We hypothesized that, during visual search, animals keep track of stimuli they have fixated earlier in the trial and that there may be neurons in FEF that keep track of these items. We first looked to see whether there is evidence to suggest that the animals are, indeed, keeping track of stimuli they have fixated earlier in the trial. In both animals, the majority of fixations were of Ts that had not been fixated earlier in the trial (23,697 [65.4%] for Animal 1 and 29,209 [80.6%] for Animal 2). The remaining fixations were approximately split between distractors (5870 [16.2%] for Animal 1 and 3295 [9.1%] for Animal 2) and previously fixated Ts (6680 [18.4%] for Animal 1 and 3018 [8.3%] for Animal 2). These fixations tended to occur when the stimulus sat between the T the animal was fixating and a remaining T that had not yet been fixated, particularly when the distance between them was great.
The frequency of fixations does not, on its own, indicate that the animals kept track of which items they had fixated earlier in the trial. In addition, we need to show that the animals treated stimuli of the same appearance differently depending on whether they had been fixated or not. Specifically, we predict that, if the animals remember which Ts they have fixated, then if they fixate the same stimulus later in the trial, it should be for a reduced duration, as if to indicate that they know they are unlikely to get a reward from the stimulus. To test this, we plotted the percentage frequency distributions of fixation durations when each animal fixated three classes of stimuli: Ts that had not been fixated earlier in the trial, Ts that had been fixated earlier in the trial, and distractors (Fig. 2). Both animals showed a single main mode of fixation duration when foveating a distractor, but clear bimodal distributions when foveating Ts. Importantly, both animals showed substantial differences in their fixation durations to Ts depending on whether they had been fixated earlier in the trial or not.
Distribution of fixation durations for the 2 animals. Bars represent the percentage of fixation frequencies under conditions in which the animals were fixating a T they had not fixated before in the trial, in which they were fixating a T they had fixated earlier in the trial, and in which they were fixating a distractor. Solid lines indicate the best fit bimodal Gaussian distributions fitted to each dataset. Vertical dashed line indicates 500 ms, the fixation duration necessary to get the reward.
To quantify these data, we fit bimodal Gaussians distributions to the data using the Gaussian mixture model fit in MATLAB (Fig. 2, solid lines; for details, see Table 1). The fits were all very good with R2 values (based on comparing the fitted distributions with the actual distributions using the bins shown in Fig. 2) ranging from 0.928 to 0.981 (Table 1). In each animal, the first mode was similar for each stimulus category and fixations in this mode were too brief to get a reward. In both animals, this mode contained a significantly greater percentage of fixations of previously fixated Ts than of Ts that had not been fixated (22.7% vs 18.9%, χ2 = 46.53, p = 9.03 × 10−12 for Animal 1; 24.4% v 7.8%, χ2 = 886.97, p ∼ 0 for Animal 2), suggesting that the animals are more likely to move on quickly from a T they have already fixated. Importantly, we also see a difference in fixation duration when the animals do not move on quickly: the distribution of the second mode was substantially and significantly shifted to the left when the animals fixated a T they had fixated earlier in the trial (p = 1.15 × 10−28 for Animal 1; p = 3.31 × 10−57 for Animal 2, Kolmogorov–Smirnov tests using simulated data based on the same distribution parameters of the second mode and a full range of data). Practically, this results in proportionally more fixations >500 ms when the animals fixate a T they have not fixated earlier in the trial (74.9% and 90.9% in Animals 1 and 2, respectively) compared with when they fixate a T they have fixated earlier in the trial (53.6% and 57.6%). In sum, the animals fixate Ts they have fixated earlier in the trial far less often than Ts they have not yet examined and, when they do refixate a T, they do so for substantially shorter durations. These differences show that each animal's behavior is affected by whether they have fixated a stimulus earlier in the trial, consistent with the hypothesis that information about which stimuli have been fixated is tracked in the brain.
Data from the fitted fixation duration distributions in Figure 2
Neuronal responses to previously fixated stimuli are enhanced in a subset of FEF neurons
We hypothesized that, if neurons in FEF are involved in keeping track of which items have been examined, then they should preferentially respond to Ts that have been previously fixated in the trial and their activity should not be modulated when a potentially rewarding T or a distractor was in their RF. To identify these neurons, we examined the responses from all fixations, after the first saccade, that lasted at least 500 ms and in which there was a stimulus in the RF, aligned by fixation onset (saccade offset). For each neuron, we compared the spike rate in a 350 ms window starting 50 ms after fixation onset (Fig. 3A) when a T that had not been previously fixated was in the RF (not fixated T in RF) and when a T that had been fixated previously in the trial was in the RF (fixated T in RF). Of the 231 neurons, 38 responded significantly more (p < 0.05, t tests) to the fixated T than to the T that had not been fixated (Fig. 3A, closed circles). We will refer to these as putative tracking neurons. We found a similar proportion of putative tracking neurons in the dataset from each animal: 13 of 78 neurons (16.7%) in Animal 1 and 25 of 153 neurons (16.3%) in Animal 2. The length of the analysis window minimally affected this selection of neurons: we found between 36 and 38 neurons were significant when using any window length from 250 to 500 ms in 50 ms increments. Of the remaining neurons (Fig. 3A, open circles), 39 responded significantly more to the T that had not been fixated than to the fixated T (p < 0.05, t tests). The differences we see in response properties between putative tracking neurons and the remaining FEF neurons are not due to confounding behavioral factors in those recording sessions: 21 of the putative tracking neurons were recorded from an electrode on which a second, nontracking neuron was recorded.
Responses of the FEF neurons during ongoing search. A, The responses of 225 neurons in a 350 ms window starting 50 ms after fixation onset when a not fixated T was in the RF are plotted against the response when a previously fixated T was in the RF. Thirty-eight neurons were identified as putative tracking neurons (closed circles) based on having a significantly higher response to the fixated T than to the T that had not been fixated (p < 0.05, t tests). B, The mean normalized responses of the population of 38 putative tracking neurons aligned by fixation onset when a fixated T was in the RF (red trace), when a distractor was in the RF (blue trace), and when a T that had not been fixated was in the RF (green trace). The width of the traces indicates the SEM, where n indicates the number of neurons contributing to the trace. The lines along the x axis indicate times at which the spike-density functions of the fixated T and distractor (blue line) or the not fixated T and the fixated T (green line) were significantly different (p < 0.05, paired t test each millisecond). C, The responses of 231 neurons when a distractor was in the RF are plotted against the responses when a fixated T was in the RF. D, The mean normalized responses of the remaining population of 187 FEF neurons plotted as a function of time. All conventions are the same as in B.
Figure 3B shows the mean population response of these putative tracking neurons as a function of time when a T that had not been fixated before (green trace), a distractor (blue trace), or a fixated T (red trace) was in the RF. The lines along the x axis indicate the times at which the spike density function of the fixated T differs from the not fixated T trace (green line on x axis) and from the distractor trace (blue line on x axis) at the 0.05 level (paired t tests at each millisecond). While the presence of many multiple comparisons would predict some spurious significant points, the likelihood of continuous significance at the 0.05 level over at least 400 ms is highly unlikely to be due to chance.
The finding that the response to the fixated T (Fig. 3B, red trace) is greater than the response to the T that had not been fixated (green trace) is unremarkable given that the neurons contributing to the plot were chosen because they showed a significant difference in a window lasting from 50 to 400 ms. However, it is clear that there is a substantial difference in response to the fixated T (red trace) and the distractor (blue trace) that is also robust for the duration of fixation. Of the 38 putative tracking neurons (Fig. 3C, closed circles), 22 responded significantly more (p < 0.05, t tests) to a fixated T than to a distractor, and all but one lay on that side of the unity line. So as a population, the putative tracking neurons responded significantly more to the fixated T than to the distractor (p = 1.46 × 10−7, Wilcoxon signed rank test). Of the remaining FEF neurons (Fig. 3C, open circles; for illustration, see Fig. 3D), 29 responded significantly more to the distractor than to the fixated T (p < 0.05, t tests), only 14 responded more to the fixated T than to the distractor and, for the population, there was a trend toward a higher response to the distractor, although it did not reach significance (p = 0.0840, Wilcoxon signed rank test). So unlike the general population of FEF neurons, the putative tracking neurons had a much higher response to a stimulus the animal had seen before than to either a distractor or a potentially rewarding T. This is the opposite to what would be predicted in a priority map and to what we have previously seen in LIP (Mirpour et al., 2009).
These responses were consistent throughout the trial. Figure 4 shows the mean normalized responses of the putative tracking neurons (Fig. 4A) and the remaining FEF neurons (Fig. 4B) as a function of the number of fixations the animals had made previously within the trial. The responses of the putative tracking neurons to fixated Ts (Fig. 4A, red data) were similar independent of the number of fixations the animals had made in a trial, whereas the responses to both classes of stimuli increased slightly as the trial progressed in the remaining neurons (Fig. 4B). These qualitative results were confirmed by two-way ANOVA of the data from each set of neurons, with neuronal response as the dependent variable and stimulus category (fixated or not fixated T) and number of previous fixations as the independent variables. For the putative tracking neurons (Fig. 4A), we found a main effect of stimulus category (F(1) = 33.9, p = 1.56 × 10−8), but no effect of number of previous fixations (F(4) = 1.27, p = 0.282) and no significant interaction (F(3) = 2.18, p = 0.071). For the remaining FEF neurons (Fig. 4B), we found a main effect of both stimulus category (F(1) = 21.0, p = 5.02 × 10−6) and number of previous fixations (F(4) = 4.15, p = 0.0024), but no significant interaction (F(4) = 0.096, p = 0.98). These data show that not only do the putative tracking neurons differ in their responses to Ts depending on whether they have been fixated earlier in the trial or not, but they also differ in how their responses are modulated throughout a trial.
Responses as a function of the number of previous fixations. A, The mean ± SEM normalized responses of the 38 putative tracking neurons across a 350 ms window starting 50 ms after fixation onset. Data from fixations in which a fixated T was in the RF (red trace) and in which a T that had not been fixated was in the RF (green trace) are plotted as a function of the number of fixations the animals had made previously in the trial. B, The mean ± SEM normalized responses of the remaining 187 FEF neurons are plotted as a function of the number of fixations the animals had made previously in the trial.
Although the response to the distractor (Fig. 3B, blue trace) was substantially lower than to a fixated T (red trace), it appeared to be slightly higher than the response to a T that had not been fixated (green trace) in the putative tracking neurons. Using the 350 ms window, starting at 50 ms, we compared the mean spike rate when a T that had not been fixated was in the RF to the mean spike rate when a distractor was in the RF for putative tracking neurons and for the remaining FEF neurons. We found that 33 of the 38 putative tracking neurons responded more to the distractor than to the T that had not been fixated, as did the population of these neurons (p = 1.82 × 10−5, Wilcoxon signed rank test), and 19 of 38 showed an individual significant difference (p < 0.05, t tests). Only 2 putative tracking neurons responded significantly more to the T that had not been fixated than to the distractor (p < 0.05, t tests), a proportion that is possibly due to chance. The opposite was seen in the remaining FEF population: the majority of neurons responded more to the T that had not been fixated than to the distractor, with 31 showing this difference significantly (p < 0.05, t tests) and only 2 neurons responding significantly more to the distractor. As a population, the remaining FEF neurons responded more strongly to the T that had not been fixated than to the distractor (p = 0.0015, Wilcoxon signed rank test).
There are two possible explanations for why putative tracking neurons responded to the distractor more than to the T that had not been fixated. One possibility is that because distractors are never linked with a reward, they are also tracked by these neurons, but with a low signal strength. The intuition for such a result is that low-level inhibitory tagging would discourage eye movements to distractors in general. The second possibility is that the elevated response could be because these neurons respond to any stimulus that has been fixated previously in the trial, and we averaged fixations in which an unseen distractor was in the RF with fixations in which a fixated distractor was in the RF. To differentiate between these possibilities, we separated the responses to both Ts and distractors depending on whether they had been fixated before in the trial or not (Fig. 5A). Animals fixate distractors on 9%–16% of fixations, but these fixations tend to be shorter than fixations of potential targets (for distributions, see Fig. 2), indicating that the animals do not expect to get a reward from them. Because of the limited number of distractor fixations, only 33 of the 38 putative tracking neurons contributed to this analysis.
Responses of the 38 putative tracking neurons during ongoing search as a function of whether targets and distractors in the RF were fixated previously in the trial or not. A, The mean normalized responses of the population of 38 putative tracking neurons aligned by fixation onset when a fixated T was in the RF (red trace), when a fixated distractor was in the RF (blue trace), when a T that had not been fixated was in the RF (green trace), and when a distractor that had not been fixated was in the RF (gray/black trace). The width of the traces indicates the SEM. B, The responses of 38 putative tracking neurons in a 350 ms window starting 50 ms after fixation onset when a distractor that had not been fixated was in the RF are plotted against the response when a previously fixated distractor was in the RF. C, The responses of 38 putative tracking neurons when a fixated T was in the RF are plotted against the responses when a fixated distractor was in the RF. D, The responses of 38 putative tracking neurons when a distractor that had not been fixated was in the RF are plotted against the responses when a T that had not been fixated was in the RF. All p values are from Wilcoxon signed rank tests.
We found that putative tracking neurons clearly track distractors that have been fixated before and weakly track all distractors. Figure 5A illustrates the mean firing rate separated by stimulus identity and whether the stimulus had been fixated earlier in the trial. The response to fixated distractors (blue trace) was considerably and significantly (p = 8.61 × 10−4, n = 33; Wilcoxon signed rank test; Fig. 5B) greater than to distractors that had not been fixated (gray/black trace). In the spike density function, the response to the fixated distractor (blue trace) appeared to be similar to the fixated T (red trace), but a close examination of the activity showed a slightly, but significantly (p = 0.0313, n = 33; Wilcoxon signed rank test; Fig. 5C), higher response when a fixated T was in the RF than when a fixated distractor was in the RF. While significant, the small difference in the magnitude of the response seems to support the idea that tracking neurons track what has been seen, with only minimal regard to the stimulus identity. Interestingly, the response to the distractor that had not been fixated (Fig. 5A, gray/black trace) appeared to be greater than the response to the T that had not been fixated (green trace). This difference was seen in 28 of the 33 putative tracking neurons and was highly significant across the population (p = 7.30 × 10−5, n = 38; Wilcoxon signed rank test; Fig. 5D). These data imply that not only do these neurons track all stimuli that have been fixated, but they weakly indicate a class of stimulus that is not linked to a reward.
Putative tracking neurons identify whether a fixated T is in their RF better than any other subset of FEF neurons. To show that the responses to fixated Ts are robust in the putative tracking neuron population and weak in the remaining FEF neurons, we used a MAP inference to decode from populations of 38 neurons, whether a fixated T, a T that had not been fixated earlier in the trial or a distractor was in the RF (for details, see Materials and Methods). Figure 6A shows a typical resulting proportion of correct predictions when a fixated T was in the RF for one iteration of training for the putative tracking neurons (gray arrow) and the 1000 sets of 38 randomly chosen neurons (black histogram). As was the case in most iterations, the majority of the shuffled populations almost never correctly predicted that a fixated T was in the RF, whereas the population of putative tracking neurons was close to 50% correct. The low percentage of correct predictions in the shuffled data occurred in part because the prior probability of having a distractor in the RF was greater than either of the other two possibilities, so when the activity was ambiguous, the model typically picked the stimulus with the greatest prior probability. In the example in Figure 6A, the proportion of correct predictions made by the putative tracking neuron population (gray arrow) lay in the top 93rd percentile of the shuffled data. Figure 6B shows the distribution of where the result from the putative tracking neuron population lay within the shuffled data distribution for all 1000 iterations. This figure shows that, in all iterations, the proportion of correct predictions by the putative tracking neurons was better than a large majority of the shuffled populations: indeed, it was never below the 70th percentile.
Putative tracking neurons decode whether a fixated T is in the RF better than other subsets of FEF neurons. A, The proportion of correct predictions from a single analysis iteration. Gray arrow indicates the results from the population of 38 putative tracking neurons. Black distribution represents the results from 1000 random sets of 38 neurons drawn from the full 231 population. B, The distribution of the percentiles from 1000 iterations. The percentile shows where the putative tracking neuron population lay within the distribution of the 1000 random sets of 38 neurons for each iteration. C, The distribution of the proportion of correct predictions from the putative tracking neurons (gray) across the 1000 iterations. Black distribution represents the range of median proportion of correct predictions taken from each iteration.
In every iteration, the proportion of correct predictions from the putative tracking neuron population were substantially greater than from the shuffled populations. To illustrate this, we have plotted the distribution of proportion of correct responses from the putative tracking neuron population from each iteration in Figure 6C (gray columns). The black columns show the distribution of median proportions from each iteration. Of note is how far apart the mean of the gray distribution (0.494 ± 0.111) is from the mean of the black distribution (0.033 ± 0.014). This substantial difference was also significant (p = 2.52 × 10−165, Wilcoxon signed rank test comparing the median of the shuffled proportion of correct predictions with the proportion of correct predictions from the putative tracking neurons from each iteration). These strong results indicate that putative tracking neurons have a robust preferential response to fixated Ts, which is absent in the remaining FEF neurons.
Putative tracking neurons do not show suppression with maintained fixation or movement bursts
We have already shown that the responses of putative tracking neurons differ compared with the remaining neurons in FEF in at least one additional way: they do not change their responses as a function of how many saccades the animal has made during the trial (Fig. 4A). Here we identify two additional physiological characteristics that are not directly related to our criterion for identifying the neurons, but which seem to be common in putative tracking neurons.
We have previously shown that, across FEF, responses tend to get suppressed when animals maintained fixation on a potential target (Mirpour et al., 2018). As detailed in the Introduction, we predicted that, if the putative tracking neurons play a role in driving inhibitory tagging, then we expect that their responses should not be suppressed during maintained fixation. Comparing Figure 3B and Figure 3D, there is a hint that the putative tracking neurons are not inhibited: unlike the traces in the remaining neurons that rapidly drop around fixation onset (Fig. 3D), the response to the fixated T in the putative tracking neurons (Fig. 3B) remains elevated. However, this could be a result of our inclusion criterion: we classified neurons because they had a greater response to a fixated T than to a T that had not been fixated, so it would stand to reason that the responses do not drop in the same way. To more rigorously test our hypothesis, we compared the responses with a fixated T in conditions in which FEF neurons are active (short fixations with a distractor at the fovea; Fig. 7A,B, blue traces) and in which FEF neurons are usually suppressed (long fixations with a T at the fovea; Fig. 7A,B, green traces) in the remaining FEF neurons (Fig. 7A) and in the putative tracking neurons (Fig. 7B). For these analyses, we only included fixations that were at least 150 ms (vertical dashed lines). For the majority of FEF neurons, we saw a strong suppression of neuronal responses when the animal fixated a potentially rewarding target (Fig. 7A, green trace) compared with when the animal fixated a distractor (Fig. 7A, blue trace). This resulted in a significantly lower response when there was a target at the fovea compared with when a distractor was at the fovea (p = 5.59 × 10−18, Wilcoxon signed rank test; Fig. 7C). However, for the putative tracking neurons, this difference was not obvious: the blue and green traces mostly overlapped in Figure 7B, and there was no obvious drop in the green trace as there was for the other FEF neurons (Fig. 7A,B, compare the green traces). Based on the spike density function, there appeared to be a trend for slightly lower responses when the animals were fixating a T compared with a distractor; however, this difference was not significant (p = 0.0978, Wilcoxon signed rank test; Fig. 7D). Nonetheless, even if this had been significant, it would be a minor effect compared with the strong suppression seen in Figure 7A.
Responses during fixation of a T or a distractor when a fixated T was in the RF. A, B, The mean normalized responses of 142 FEF neurons (A) and 30 putative tracking neurons (B) aligned by fixation onset when a fixated T was in the RF and the animal was fixating a T (green trace) or the animal was fixating a distractor (blue trace). The width of the traces indicates the SEM. Black line along the x axis represents times at which the two traces are significantly different (p < 0.05, paired t tests each millisecond). C, The responses of 142 FEF neurons in a 350 ms window starting 50 ms after fixation onset to a fixated T in the RF when a distractor was at the fovea are plotted against the response when a T was at the fovea. D, The responses of 30 putative tracking neurons to a fixated T in the RF when a distractor was at the fovea are plotted against the response when a T was at the fovea. p values are from Wilcoxon signed rank tests.
Putative tracking neurons did not appear to be a specific subset of neurons when classified into the more common visual, visuomovement, and movement categories of FEF neurons. Within the putative tracking neuron population, 64.0% (16 of 25) were classified as visuomovement neurons, 16.0% (4 of 25) were classified as visual neurons, and 20.0% (5 of 25) were classified as movement neurons. We were unable to classify the remaining 13 neurons because of insufficient data from the MGS. Within the remaining population of FEF neurons, 56.8% (79 of 139) were classified as visuomovement neurons, 24.5% (34 of 139) were classified as visual neurons, and 18.7% (26 of 139) were classified as movement neurons; we were unable to classify 54 of the neurons. The proportions of visual, visuomovement, and movement neurons were not different in the two subsets of neurons (χ2(2) = 0.861, p = 0.650, χ2 test). We additionally examined the proportion of neurons that had delay activity in the MGS. We found that 37% (66 of 180) of the remaining population of FEF neurons and 53% (19 of 36) of the putative tracking neurons had significant delay activity. These proportions were not significantly different (χ2 = 3.26, p = 0.071, χ2 test).
We have hypothesized that the putative tracking neurons may have a different function to the classic FEF view of driving eye movements and covert attention, yet the similarities in proportions of visuomovement and movement neurons seem to contradict that idea. However, the classification of neurons into these classes rests on the presence of a significant response enhancement above a baseline response, not on the magnitude of the response, so it is possible that, while these neurons are responsive in the MGS, they do not play the same role that similarly classified neurons in the rest of the population play. To test this hypothesis, we examined the responses of the neurons when a saccade was made into their RF (Fig. 8). The 179 remaining FEF neurons (dark trace) showed the typical movement-related response, peaking around the time of the saccade at a response magnitude that is far greater than at any other time in the task (note the change in scale on the y axis compared with other figures). The 38 putative tracking neurons, on the other hand, had a slightly elevated response around the time of the saccade. However, they did not show the classic well-timed burst (there are two peaks in the response, neither of which was aligned with the peak in the main population), and the level of responses was significantly (black line on x axis, p < 0.05, Wilcoxon rank sum tests every millisecond) and substantially weaker than the burst in the remaining neurons, despite having the same proportions of visuomovement and movement neurons in each population.
Mean normalized responses of the 38 putative tracking neurons (light gray trace) and 179 FEF neurons (dark gray trace) when a saccade is made into the neuron's RF. Data are aligned by saccade onset. The width of the traces indicates the SEM. Black line along the x axis indicates times at which the two traces are significantly different (p < 0.05, Wilcoxon rank sum tests each millisecond).
Responses to previously fixated stimuli do not vary as a function of how long ago they were fixated
Presumably, there is a limit to how many fixated items can be remembered. We hypothesized that items that have been fixated, but then forgotten, should be represented by reduced responses in the putative tracking neurons. To see whether this occurred and, thus, to identify the limit to the number of items that can be remembered, we plotted the responses of putative tracking neurons when a fixated T was in the RF as a function of the number of eye movements the animal had made since looking at that specific stimulus. Figure 9A shows the mean normalized responses over a 350 ms window starting 50 ms after fixation onset from the 29 putative tracking neurons that had sufficient numbers of fixations in each category. We examined responses when the eye had just come from the stimulus, when the animal made 2 saccades before the stimulus appeared in the RF and when the animal had made 3 or 4 saccades before the stimulus appeared in the RF. The bars and error bars show the mean and SEM for the population, and the mean normalized responses from each neuron are shown as closed circles. While the responses look very similar, there was a trend for a slightly lower mean after 3 or 4 saccades. To see if this trend was significant, we compared the raw spike rates in the condition when only one saccade had been made (“1 back”) and when 3 or 4 saccades had been made (“3 and 4 back”; Fig. 9B). We found that, although the p value approached 0.05 (p = 0.0614, Wilcoxon signed rank test), there was no significant difference in mean response. When we separated the 3 back and 4 back fixations, the results were more ambiguous with much higher p values, but this could be due to the reduced statistical power we incurred when neurons were dropped from the analysis. We should note that this analysis is constrained by the design of our task, which limits the number of purposeful fixations to 5, so it is possible that a clear reduction of activity could be seen if more Ts were to be fixated within a trial.
Responses of 29 putative tracking neurons to a fixated T as a function of when it was fixated previously in the trial. A, The mean ± SEM normalized response (bar graph and error bars) and the individual mean normalized responses of the 29 neurons (closed circles) over a 350 ms window starting 50 ms after fixation onset are plotted as a function of the number of saccades the animal made since fixating the T that is now in the RF. B, The responses to a fixated T in the RF are plotted from fixations in which 3 or 4 saccades were made since fixating the T that is now in the RF against the responses when the animal had made only one saccade since fixating the T. The p value is from a Wilcoxon signed rank test.
Remapping can potentially explain the propagation of the tracking signal across FEF
It is known that a subset of neurons in the FEF predictively respond to a stimulus that will appear in the neurons' RF after a saccade (Umeno and Goldberg, 1997; Sommer and Wurtz, 2006; Joiner et al., 2011). Such remapping has been described as a shift of the representation within a cortical area, analogous to the shift of the image on the retina (Duhamel et al., 1992). We hypothesized that this sort of mechanism could explain how the extraretinal signal representing tracked items could be updated across saccades. To test this hypothesis, we looked to see whether the tracking signal appeared around the time when remapping occurs (i.e., around the time of the eye movement) or whether it appeared well after fixation, which would be more consistent with a fixation-by-fixation top-down input.
We found that the timing of the elevated response to a previously fixated stimulus was consistent with a remapped response in putative tracking neurons. The green line on the x axis of Figure 3B shows the times that the activity represented by the spike density functions were significantly different (p < 0.05, paired t test at each ms) when a fixated T was in the RF compared with when a T that had not been fixated was in the RF. This difference started ∼25 ms before fixation onset, 75 ms before the window we used to create our inclusion criterion. This was not just due to normalizing and smoothing the neuronal data: we compared the mean spike rates between these two conditions in a 25 ms window starting 25 ms before fixation onset (Fig. 10A) and found that 28 of the putative tracking neurons responded more to the upcoming fixated T than to the upcoming T that had not been fixated and 9 of these did so significantly (p < 0.05, t tests). Only one neuron responded significantly more to the upcoming T that had not been fixated than to the upcoming fixated T. Across the population of 38 neurons, the mean response to the upcoming fixated T was significantly higher than the mean response to the upcoming T that had not been fixated (p = 0.0100, n = 38; Wilcoxon signed rank test). These data confirm that, before the saccade even ended, the population of putative tracking neurons indicate whether the stimulus that would enter the RF had been fixated before or not.
Remapped responses of the putative tracking neurons. A, The responses of the 38 putative tracking neurons in a 25 ms window starting 25 ms before fixation onset when a previously fixated T would be brought into the RF are plotted against the response when a T that had not been fixated would be brought into the RF. B, The mean normalized responses of 28 putative tracking neurons when a T that had not been fixated was in the RF (green trace), when the T that had just been fixated was in the RF (yellow trace), and when a T that had been fixated earlier in the trial was in the RF (red trace). The width of the traces indicates the SEM. The lines along the x axis indicate times at which the red and green traces were significantly different (red line), at which the yellow and green traces were significantly different (yellow line), and at which the red and yellow traces were significantly different (black line). Significance was determined as p < 0.05 (paired t test each millisecond).
The finding that putative tracking neurons remap might explain the propagation of this signal across FEF as eye movements are made. We hypothesized that a way for this signal to enter this network would be if the activity from fixation neurons in FEF (Bizzi, 1968; Suzuki and Azuma, 1977; Bon and Lucchetti, 1990; Izawa et al., 2009), which are only active during fixation, were remapped to putative tracking neurons during the saccade that took the object from fixation to the RF. To test this hypothesis, we examined the responses of the 28 putative tracking neurons that had sufficient data in three conditions: when a T that had not been fixated was brought into the RF (Fig. 10B, green trace), when the T that had just been fixated was brought into the RF (Fig. 10B, yellow trace), and when a T that had been fixated earlier in the trial, but not on the last fixation, was brought into the RF (Fig. 10B, red trace). We use the response to the T that had not been fixated as the value to which we can compare the other responses: when they are significantly greater than this activity, then we can claim that the neuron is representing whether the stimulus in the RF has been fixated before. The red line along the x axis shows when the red and green traces were significantly different (p < 0.05, paired t tests every millisecond), the yellow line along the x axis shows when the yellow and green traces were significantly different, and the black line along the x axis shows when the red and yellow traces were significantly different.
There are three important points that can be gleaned from this figure. First, for most of the fixation, the red and yellow traces are superimposed. This means that whether the T had been fixated in the previous fixation or earlier in the trial, the activity in the population appeared to be similar. This is consistent with the data shown in Figures 4A and 9. Second, the enhanced response to a fixated T that had been fixated earlier in the trial (red trace) was significantly greater than the response to a T that had not been fixated starting ∼20 ms before fixation onset. Third, the response to the T that had just been fixated (yellow trace) rose significantly above the response to the T that had not been fixated ∼45 ms after fixation onset. This is substantially later than when the signal was remapped from neurons with peripheral RFs (see black line above the x axis for times the red and yellow traces are significantly different). This suggests that, although the signal indicating that a stimulus has been fixated is remapped, the timing is not consistent with the hypothesis that it is first brought into the network by simply remapping the activity from fixation neurons.
Discussion
Using a visual foraging task, we showed that animals behave as if they have knowledge about which stimuli they had fixated earlier in the trial. We then identified a subset of FEF neurons that preferentially responded to stimuli that had been fixated earlier in the trial than to objects the animal had not yet fixated. We found that the responses of these neurons differed from other FEF neurons along three additional dimensions that were orthogonal to our inclusion criteria: their responses did not vary as a function of the number of saccades the animal had made in the trial, they did not show the typical FEF movement response when a saccade was made into their RFs, and their responses were not suppressed during maintained fixation. We conclude that these neurons have the appropriate properties to track which items in the visual world have been fixated during search and that they could provide the top-down feedback needed to suppress responses to fixated Ts in LIP, and possibly in FEF, to produce the behavioral phenomenon of inhibitory tagging.
One previous study highlighted a subset of ∼10% of neurons in FEF that preferentially responded when animals were cued to not make a saccade to the RF (Hasegawa et al., 2004). It is possible that our putative tracking neurons, which made up ∼16% of the FEF neurons we recorded, are the same population identified by Hasegawa et al. (2004). However, the previous study did not identify any characteristics about the population apart from their response preference to “don't look” locations in the nonmatch to sample task, so we cannot be sure whether these are the same class of neurons. Nonetheless, if these neurons play a role in signaling to priority map areas locations that should be avoided in search, because they indicate which items have been fixated, then one can imagine that they might respond in a spatial nonmatch to sample task. Given that visual search for items or people occurs frequently in everyday life, whereas the suppression of eye movements to forbidden targets is rare, we propose that these neurons are probably more appropriately called tracking neurons rather than “don't look” neurons, as termed in that study.
We found that the elevated response to a stimulus that had been fixated earlier in the trial began, on average, before the saccade even ended. Given this timing, we suggest that the propagation of this extraretinal signal could occur using the remapping mechanism known to be present in FEF (Umeno and Goldberg, 1997; Sommer and Wurtz, 2006; Joiner et al., 2011). This provides a solution to the question of why we should find these neurons in FEF. Given the corollary discharge from the superior colliculus, via the mediodorsal nucleus of the thalamus, is known to drive remapping in FEF (Sommer and Wurtz, 2002, 2006) and the presence of fixation neurons in FEF (Bizzi, 1968; Suzuki and Azuma, 1977; Bruce and Goldberg, 1985; Izawa et al., 2009), it is a logical place to have neurons that track items that have been fixated. We also hypothesized that the remapping of activity directly from fixation neurons (Bizzi, 1968; Suzuki and Azuma, 1977; Bruce and Goldberg, 1985) would be an elegant way to bring this signal into the peripheral FEF representation. However, given that the enhanced tracking response did not emerge until 40 ms after fixation onset when the T that had just been fixated arrived in the RF (Fig. 10B, yellow trace), we suggest that this is not the case. Nonetheless, we speculate that it is likely that there is an indirect mechanism that uses the activity of fixation neurons in FEF (Bizzi, 1968; Suzuki and Azuma, 1977; Bruce and Goldberg, 1985; Izawa et al., 2009) or elsewhere (Bon and Lucchetti, 1990; Munoz and Wurtz, 1992) to initiate the tracking responses we see in these neurons. The alternative would be the addition of a top-down signal that activates these neurons when a stimulus that had just been fixated is brought into their RF.
We have hypothesized that the putative tracking neurons are involved in tracking stimuli that animals should avoid and, thus, may drive an inhibitory tagging signal. This is very different from the more traditional way of thinking about FEF as driving eye movements and guiding covert attention (Schall, 2004; Squire et al., 2013; Nelson et al., 2016; Ramkumar et al., 2016). Yet, when using standard FEF classifications of neuronal cell type, based on the memory-guided saccade, the putative tracking neurons did not appear to differ from the general FEF population. As noted above, the classification of neurons into the three standard categories relies on the presence of a significant response enhancement above baseline rather than the magnitude of the response, resulting in very different movement response profiles when saccades were made into the RF (Fig. 8). We suggest that this may be a serious limitation in the usefulness of these traditional categories.
Based on our data, we cannot prove that these neurons do not represent one end of a continuum of FEF response properties; however, we found that this subset of FEF neurons had three features that were orthogonal to the metric we used to identify the neurons and were, in the population, significantly different from the remaining population of FEF neurons. First, the putative tracking neurons did not change their response as a function of the number of saccades the animal had made, but the remaining FEF neurons did (Fig. 4). Second, the putative tracking neurons did not have a classical saccadic burst when a saccade was made into their RF, whereas the remaining FEF neurons did (Fig. 8). Finally, the responses of the putative tracking neurons did not get suppressed when the animals maintained fixation at a potential target, whereas the population of remaining FEF neurons did (Fig. 7). This last result is additionally important because it is consistent with the hypothesized role of the neurons: if their role is to track stimuli that have been fixated to reduce the probability of fixating the stimulus again, then suppressing this signal during maintained fixation would be counterproductive. Instead, these neurons indicate which items have been fixated throughout the duration of the fixation. The fact that these neurons appeared to be different from the remaining population of FEF neurons in these three unrelated ways suggests that the categorization of putative tracking neurons was not due to spurious factors relating to the animals' behavior in those sessions.
Given these three additional properties, we think it plausible that these neurons represent a separate subpopulation of neurons within FEF, with the acknowledgment that we are likely oversimplifying a far more complex system. If true, we would predict that these neurons would also differ from other FEF neurons based on their projections: we would predict that tracking neurons would feed back to LIP (Andersen et al., 1985; Stanton et al., 1995) to drive the inhibition seen in that area (Mirpour et al., 2009) rather than feeding into more oculomotor areas, such as the superior colliculus or brainstem oculomotor control centers (Komatsu and Suzuki, 1985; Stanton et al., 1988).
Footnotes
This work was supported by National Eye Institute R01 EY019273. We thank members of the University of California–Los Angeles Division of Laboratory Animal Medicine for superb animal care.
The authors declare no competing financial interests.
- Correspondence should be addressed to Koorosh Mirpour at kmirpour{at}mednet.ucla.edu
