Abstract
In natural scenes, multiple visual stimuli compete for selection; however, each saccade displaces the stimulus representations in retinotopicaly organized visual and oculomotor maps. In the present study, we used saccade curvature to investigate whether oculomotor competition across eye movements is represented in retinotopic or spatiotopic coordinates. Participants performed a sequence of saccades and we induced oculomotor competition by briefly presenting a task-irrelevant distractor at different times during the saccade sequence. Despite the intervening saccade, the second saccade curved away from a spatial representation of the distractor that was presented before the first saccade. Furthermore, the degree of saccade curvature increased with the salience of the distractor presented before the first saccade. The results suggest that spatiotopic representations of target–distractor competition are crucial for successful interaction with objects of interest despite the intervening eye movements.
Introduction
To perform an accurate eye movement to the target, the competition between the saccade goal and the distracting stimuli needs to be resolved (Itti and Koch, 2000). Because competing targets and distractors in the oculomotor system are represented in retinotopic coordinates (Port and Wurtz, 2003), it is important to update competing distractor representations with every intervening saccade. Even though current models of visual and oculomotor selection assume that spatiotopic representations are rapidly available across eye movements [inhibition of return (IOR), Posner and Cohen, 1984; visual salience, Itti and Koch, 2000], recent studies have proposed that retinotopic representations dominate in the visual and oculomotor systems after the saccades, whereas spatiotopic representations are formed gradually, if at all (Golomb et al., 2008; Zimmermann et al., 2011; Golomb and Kanwisher, 2012; Zimmermann et al., 2013). Furthermore, recent studies of saccadic IOR have reported conflicting results, demonstrating both the rapid formation of spatiotopic IOR (Pertzov et al., 2010; Hilchey et al., 2012) and a gradual transition from retinotopic to spatiotopic IOR (Mathôt and Theeuwes, 2010). Therefore, it remains unclear whether the oculomotor system represents the competing stimuli in retinotopic or spatiotopic coordinates across eye movements.
In the present study, we used saccade curvature to investigate reference frames of target–distractor competition in the oculomotor system. Saccade curvature has been attributed to competition in the oculomotor map for potential saccade targets occurring within the intermediate layers of superior colliculus (SC; Doyle and Walker, 2001; McPeek et al., 2003; McPeek, 2006; van der Stigchel et al., 2006; Walker and McSorley, 2008). Saccade curvature toward distractor locations has been suggested to arise from failure to suppress competing distractor representations (McPeek, 2006), whereas successful suppression of the competing distractor representations has been suggested to result in curvature away from the distractor location (Godijn et al., 2004; Belopolsky and Theeuwes, 2011). Furthermore, the more salient the distractor, the more competition and thus the more curvature of the saccade path it evokes (van Zoest et al., 2012). In the present study, participants performed a sequence of saccades while we induced oculomotor competition by presenting a task-irrelevant distractor briefly at different times during the saccade sequence (Fig. 1A). If distractor representations are rapidly updated across saccades, then we expected the second saccade to curve away from the spatiotopic location of the distractor presented before the first saccade.
Materials and Methods
Setup
We recorded gaze position with an SR Research EyeLink 1000 eye tracker (sampling rate of 1000 Hz). The experiment was controlled and implemented in MATLAB (MathWorks) using the Psychophysics and Eyelink toolboxes. Stimuli were presented at a viewing distance of 70 cm on 22-inch Samsung Syncmaster 2233RZ monitor in Amsterdam or on 20 inch Sony GDM-F500R monitor in Munich. Both monitors had a vertical refresh rate of 120 Hz and a stimulus decay rate smaller than the display refresh rate.
Participants
Each experiment contains data from 10 participants (age 21–30 years; nine female); three participated in both experiments; five participants from Experiment 1 were tested in Amsterdam. All participants were naive as to the purpose of the study. The experiments were conducted in accordance with the Declaration of Helsinki.
Procedure
Experiment 1.
Each trial started with fixation target (black dot, radius 0.75°, 5 cd/m2) presented on a gray background (83 cd/m2). After a variable fixation period (Gaussian distribution mean 1000 ms, SD 300 ms), one or two saccade targets appeared (black dot, radius 0.75°). In the single saccade condition, participants made one saccade: a leftward/rightward horizontal saccade directed from periphery to the center of the screen or an upward/downward vertical saccade originating from the center of the screen. In the double saccade condition, participants made two saccades: a leftward/rightward horizontal saccade to the center of the screen followed by an upward/downward vertical saccade from the center of the screen. Single and double saccade conditions were blocked. On each trial, the distance between the fixation and saccade targets randomly varied between 6° and 8°. Saccade targets were visible for 700 ms. The distractor was a dark blue dot (radius 1.2°, 34.5 cd/m2, CIE Yxy 34.3, 0.184, 0.133) that was shown briefly (50 ms) at a randomly selected time (−400 to 600 ms relative to the saccade target onset). The distractor was presented randomly clockwise or counterclockwise relative to the saccade target (Fig. 1A, inset 2) halfway between the fixation and saccade targets. Each participant completed ∼4500 trials during 8–10 h of testing (depending on individual speed and availability).
Experiment 2.
All settings were the same as in Experiment 1 except for the following. Participants performed a double saccade task only. Ten filled circles (gray color, 91 cd/m2) were arranged in a circle and were present throughout the entire trial. Distractors were shown clockwise or counterclockwise relative to the second saccade target only. The distractor was a luminance increase of one of the circles, which was presented for 50 ms at the time selected randomly between 50 and 350 ms relative to the saccade target onset and was present on 50% of the trials. Distractor salience was manipulated by setting the distractor luminance at one of three levels (109, 120, or 143 cd/m2). Even low salient distractors were visible. At the end of each trial, participants indicated whether the distractor was present or absent by a button press. Each participant completed ∼3900 trials during 6–8 h of testing.
Data analysis
Trials with saccades faster than 80 ms and slower than 700 ms, saccades that did not start within 2° from fixation point, and saccades that did not land within 3° of the target were discarded from further analyses (23% in Experiment 1 and 18% in Experiment 2). Furthermore, trials with blinks (3% in Experiment 1 and 3% in Experiment 2) and with saccades landing within 2° of the distractor (0.3% in Experiment 1 and 2% in Experiment 2) were also discarded.
For all saccade curvature calculations, trajectory data were rotated and inverted so that all horizontal saccades started at the center of the display and were directed rightward and all vertical saccades were directed upward. Trajectory data were then rotated and inverted based on whether the retinotopic distractor representation stayed within the same visual hemifield across the saccade or if it moved to a different visual hemifield. Because every saccade had a different amplitude and duration, we normalized saccade duration such that each sample point was plotted relative to the total saccade duration (Dhawan et al., 2013). If the total saccade duration was 50 ms and the sample was acquired at 25 ms, the sample was drawn at 50% of saccade's normalized duration. Saccade curvature was determined by computing the angular deviations of each sample point from a straight line connecting the starting and ending points of saccade (Belopolsky and Van der Stigchel, 2013). A median of these deviations was calculated for each saccade, after which each condition curvature was quantified as a difference in curvature between the trials on which distractor was presented clockwise or counterclockwise relative to the saccade target (negative curvature corresponds to curvature away from the distractor). To examine the time course of saccade curvature, the times between distractor offset and saccade onset were divided into 100 ms bins in Experiment 1 (∼60 trials per bin) and 50 ms bins in Experiment 2 (∼100 trials per bin).
To evaluate clockwise and counterclockwise trajectories separately, trajectories were corrected for their idiosyncratic trajectory. First, we calculated baseline saccade trajectory for each participant and for each saccade sequence (rightward-up, rightward-down, leftward-up, and leftward-down) by averaging saccade trajectories for clockwise and counterclockwise distractor locations. This baseline trajectory was comparable to the trajectory in distractor absent condition (in Experiment 2, F(1,9) = 0.24, p = 0.63) that is often used as a baseline for individual variance in saccade trajectories (van der Stigchel et al., 2006). Second, we subtracted out the corresponding baseline from the trajectory in each condition (e.g., the leftward-up baseline trajectory was subtracted from leftward-up saccades). Repeated-measures ANOVAs and t tests (uncorrected) were used for statistical comparisons.
Results
Saccade latencies, amplitude, and duration in single and double saccade tasks in Experiment 1 are shown in Table 1. Both horizontal (Fig. 1B) and vertical (Fig. 1C) saccades curved away from the distractor location, suggesting the rapid suppression of the competing distractor representations in the oculomotor system. Consistent with the literature, curvature away was less pronounced for horizontal than for vertical saccades (Walker and McSorley, 2008). We also observed that, for vertical saccades (Fig. 1C), the time course of saccade curvature was clearly different between single and double saccade tasks. There was a significant effect of distractor offset time (F(6,54) = 7.45, p = 0.001), no effect of single/double saccade task (F(1,9) = 0.07, p = 0.79), and a significant interaction effect (F(6,54) = 5.89, p = 0.001). The largest differences in time course were driven by increased curvature away from distractors presented during the intersaccadic interval of the double saccade task, suggesting that the oculomotor system might be especially vulnerable to competing distractors during the intersaccadic interval.
Average saccade latencies, amplitude, and duration in Experiments 1 and 2
Experiment 1. A, Trial timing and display setup. Inset 1, All possible saccade directions (indicated by arrows) and distractor locations (filled dark blue circles). Inset 2, Distractor location relative to saccade target. Example saccade direction is indicated by an arrow; cw, clockwise distractor location; ccw, counterclockwise. B, Horizontal saccade trajectory curvature as a function of distractor offset time relative to horizontal saccade onset. Shaded area indicates SEM. Sacc H, Horizontal saccade onset time C, Vertical saccade trajectory curvature as a function of distractor offset time relative to the vertical saccade onset. Sacc V, Vertical saccade onset time; Sacc H, horizontal saccade onset time in the double saccade task. D, Vertical saccade trajectories in the single saccade task. Filled circles represent relative distractor location (green, counterclockwise; violet, clockwise). Distractor offset time was 300–200 ms before the vertical saccade onset. Shaded areas indicate SEM. E, Vertical saccade trajectories in the double saccade task. Distractor offset time was 100–0 ms before the horizontal saccade onset (∼300–200 ms before the vertical saccade onset). Inset 1, Distractor locations before saccade sequence. Arrows indicate saccade sequence; star, eye position at the time of distractor presentation. Inset 2, Retinotopic distractor locations after the horizontal saccade. Curved lines indicate predicted trajectory curvature away from the retinotopic distractor location. Inset 3, Spatiotopic distractor locations and predicted spatiotopic saccade trajectory curvature. F, Spatiotopic saccade curvature for short and long intersaccadic interval duration. ns, Nonsignificant t test comparison between the two conditions.
The main question was whether the second (vertical) saccade in the double saccade task would curve away from the distractor presented before the first (horizontal) saccade. Indeed, vertical saccades curved away from distractors presented before the onset of the horizontal saccade (Fig. 1C, red line; −200 to −100 ms before horizontal saccade onset (or ∼−410 to −310 ms before vertical saccade onset; t(9) = 2.94, p = 0.02) and −100 to 0 ms before horizontal saccade onset (or ∼−310 to −210 ms before vertical saccade onset; t(9) = 3.39, p = 0.007). To further show that saccades curved away from the spatial (Fig. 1E, inset 3) and not from the retinal (Fig. 1E, inset 2) representations of the distractor, we analyzed curvature separately for trials on which the intervening horizontal saccade brought the retinal location of the distractor into a different visual hemifield (Fig. 1E, inset 2, shown as green distractor) and for trials on which it kept it within the same hemifield (shown as a violet distractor). Figure 1E clearly shows that in both cases saccades curved away from the spatiotopic location of the distractor (different hemifield: 0.48°± 0.12°, t(9) = 3.80, p = 0.004; same hemifield: 0.39°± 0.09°, t(9) = 4.30, p = 0.002; distractor offset time −100 to 0 ms before the horizontal saccade onset). Figure 1D shows that this curvature pattern was strikingly similar to the single saccade task (clockwise distractors: 0.43°± 0.11°, t(9) = 3.85, p = 0.004; counterclockwise distractors: 0.53°± 0.13°, t(9) = 3.93, p = 0.003; distractor offset time −300 to −200 ms before the vertical saccade onset), indicating that saccade curvature followed the spatiotopic and not the retinotopic representation of the distractor.
One possibility is that updating the spatial location of the distractor occurs gradually during the intersaccadic interval. This would predict that the degree of saccade curvature away from the distractor should increase as the intersaccadic interval increases. We performed a median split of the intersaccade intervals (short intersaccadic interval: 148 ± 7 ms; long: 209 ± 9 ms). The saccade curvature (Fig. 1F) was comparable between short and long intersaccade intervals (t(9) = 1.05, p = 0.32) and, if anything, decreased with time, arguing against the slow updating of the distractor location during the intersaccade interval. Combined, the results of Experiment 1 clearly demonstrate that the location of competing distractor is updated very quickly and automatically within the oculomotor system.
In Experiment 2, we further investigated whether the degree of competition between the distractor and the target could be maintained across saccades. To do this, we manipulated the salience of the distractor by adjusting its luminance (Fig. 2A, inset). We expected that, if the degree of competition produced by the distractor is preserved across saccades, then saccade curvature away should increase with the salience of the distractor.
Results of Experiment 2. A, Vertical saccade trajectory curvature as a function of distractor offset time before vertical saccade onset and onset time after vertical saccade offset. Sacc V, Vertical saccade onset time; Sacc H, horizontal saccade onset time. Shaded areas indicate SEM. Example display setup is shown in the inset. B, Vertical saccade trajectories distractor (green, violet) and no-distractor trials (gray). Distractor offset time was 50–0 ms before the horizontal saccade onset. C, Spatiotopic saccade curvature for short and long intersaccadic interval duration. ns, Nonsignificant t test comparison between the two conditions. D, Saccade curvature for distractors presented before horizontal saccade onset and after horizontal saccade offset as a function of distractor salience level. Shaded areas indicate SEM.
Saccade latencies, amplitude, and duration in distractor present and distractor absent trials in Experiment 2 are shown in Table 1. Analyses of the curvature of saccade trajectories showed that the presence of the distractor caused saccades to curve away from the distractor location (Fig. 2A). A two-way ANOVA showed an effect of distractor offset time (F(5,45) = 8.60, p = 0.001), an effect of distractor presence/absence (F(1,9) = 8.18, p = 0.01), and an significant interaction effect (F(5,45) = 5.13, p = 0.001). As before, vertical saccades curved away from distractors presented before the horizontal saccade (150 to −100 ms: t(9) = 0.46, p = 0.65; −100 to −50 ms: t(9) = 3.56, p = 0.006; −50 to 0 ms: t(9) = 4.02, p = 0.003). The distractor effect on saccade curvature was found to emerge later (−100 to 0 ms before horizontal saccade onset) than was observed in Experiment 1 (−200 to 0 ms before horizontal saccade onset). This difference is likely due to the way that the salient distractor was presented. Recent work using cluttered displays similar to Experiment 2 demonstrated that the influence of salient distractor on selection and saccade trajectory is short lived (van Zoest et al., 2012).
As in Experiment 1, we observed that saccade trajectories curved away from the spatiotopic location of the distractor (different hemifield: 1.74°± 0.42°, t(9) = 4.08, p = 0.002; same hemifield: 1.74°± 0.40°, t(9) = 4.30, p = 0.002; distractor offset time −50 to 0 ms before the horizontal saccade onset; Fig. 2B). Furthermore, as in Experiment 1, the saccade curvature was comparable between short (161 ± 6 ms) and long intersaccade intervals (216 ± 9 ms; t(9) = 0.23, p = 0.82; Fig. 2C).
Finally, we investigated whether a distractor's salience affected saccade trajectories by analyzing the time bins right before saccade onset (distractor offset −50 to 0 ms) and after saccade offset (distractor onset 0 to 50 ms). The curvature away increased with the salience of the distractor for both time bins (Fig. 2D). Two-way ANOVA showed an effect of distractor salience (F(2,18) = 3.90, p = 0.03), no effect of distractor offset time (F(1,9) = 0.53, p = 0.49), and no interaction (F(2,18) = 0.22, p = 0.80). The results of Experiment 2 indicate that, not only does the information about the distractors' spatial location get updated across saccades, but also information about the distractors' salience level.
Discussion
The present study clearly demonstrates that target–distractor competition within the human oculomotor system remains spatially stable across saccades. In two experiments, we showed that when the distractor was presented before the first saccade in the sequence, the second saccade curved away from its spatial location. The fact that the saccade curvature away was similar for short and long intersaccadic intervals suggests that the location of a task-irrelevant distractor was updated rapidly, within ∼150 ms before the start of second saccade. Furthermore, we observed that the degree of saccade curvature away from the distractor increased with increases of the salience of the distractor, suggesting that information about the distractor's salience was also transferred across saccades. Our findings show that the oculomotor system keeps track of both the spatial locations of distractors and the degree of competition that they evoke.
The present results argue against the slow transition from retinotopic to spatiotopic representations across eye movements that has been suggested previously (Golomb et al., 2008; Zimmermann et al., 2013). Instead, we demonstrated that spatiotopic effects emerge rapidly and automatically. One possibility for this discrepancy is that previous work has largely relied on tasks involving perceptual judgments (Golomb et al., 2008; Zimmermann et al., 2013), whereas our task focused on the performance of the oculomotor system itself. This is consistent with the idea of potential differences between the visual stability used for perception and the visual stability used for action (Bays and Husain, 2007; Medendorp, 2011) because spatiotopic representations are crucial for successful interaction with objects. In addition, studies demonstrating spatiotopic visual selection across saccades (e.g., spatiotopic IOR: Pertzov et al., 2010; Hilchey et al., 2012, but see conflicting results in Mathôt and Theeuwes, 2010; illusory motion perception: Szinte and Cavanagh, 2011; Melcher and Fracasso, 2012; allocation of spatial attention: Jonikaitis et al., 2013) used salient, attention capturing stimuli that are known to automatically activate the oculomotor system. In contrast, the studies showing gradual emergence of spatiotopic representation often used stimuli that were required to be committed to memory over long periods of time (Golomb et al., 2008; Golomb and Kanwisher, 2012).
Furthermore, saccade curvature itself may allow for a more direct measure of competition within the oculomotor system. The presence of a distractor leads to preparation of a saccade and the corresponding neural activity within the oculomotor map in the SC or frontal eye fields (FEFs; McPeek et al., 2003; McPeek, 2006). Stimulation of SC evokes saccade trajectory curvature toward the stimulated receptive field (RF; McPeek et al., 2003), whereas local inactivation evokes saccade curvature away (Aizawa and Wurtz, 1998). It is assumed that the oculomotor activity at the distractor location needs to be inhibited, which causes the overall vector of the prepared saccade to shift in the opposite direction (Aizawa and Wurtz, 1998; Doyle and Walker, 2001). Such competing oculomotor programs from SC are interpreted by the brainstem saccade generator and this results in curved saccade trajectories (Walton et al., 2005).
A few mechanisms could mediate the formation of spatiotopic representations within the oculomotor system. First, spatiotopic maps (with neurons representing spatial, not retinal, stimulus location) could be formed in parallel to retinotopic maps (Burr and Morrone, 2011), which could maintain target–distractor competition in spatiotopic coordinates. However, it has been suggested that the formation of such spatiotopic maps takes longer than was found in our study (Zimmermann et al., 2013) and it is not known how such maps interact with the saccadic system. Second, perisaccadic remapping could act as a mechanism of updating stimulus representation from presaccadic retinotopic RF to postsaccadic retinotopic RF (Wurtz, 2008). Perisaccadic remapping is localized to oculomotor structures such as FEF and SC and could be closely bound to updating of oculomotor representations (Melcher, 2007; Melcher and Colby, 2008; Wurtz, 2008; Rolfs et al., 2011). Third, object-based reference frames could facilitate the formation of spatiotopic representations. Objects within the visual scene could act as anchors in localizing stimuli both with and without saccades (Hayhoe et al., 1991; Boi et al., 2011; Boon et al., 2014). Any of these three mechanisms could be used by the visual and oculomotor systems.
Our findings suggest that the oculomotor system not only represents competing motor programs, but also rapidly updates the competing representations across eye movements. Even though saccade initiation is fast and requires little resources, losing track of important or rewarding objects is costly and could leave the system unable to respond appropriately after the saccade has finished. Therefore, a mechanism that can update the location and the salience of competing distractors is essential to allow appropriate postsaccadic behavior regardless of the intervening retinal stimulus displacements.
Footnotes
This work was supported by the Deutsche Forschungsgemeinschaft (Grant JO 980/1-1 to D.J.).
The authors declare no competing financial interests.
- Correspondence should be addressed to Donatas Jonikaitis, Allgemeine und Experimentelle Psychologie, Leopoldstr 13, 80802 Munich, Germany. donatas.jonikaitis{at}psy.lmu.de
