Inhibition of return (IOR), a performance decrement for stimuli appearing at recently cued locations, occurs when the target and cue share the same screen position. This is in contrast to cue-based attention facilitation effects that were recently suggested to be mapped in a retinotopic reference frame, the prevailing representation throughout early visual processing stages. Here, we investigate the dynamics of IOR in both reference frames, using a modified cued-location saccadic reaction time task with an intervening saccade between cue and target presentation. Thus, on different trials, the target was present either at the same retinotopic location as the cue, or at the same screen position (e.g., spatiotopic location). IOR was primarily found for targets appearing at the same spatiotopic position as the initial cue, when the cue and target were presented at the same hemifield. This suggests that there is restricted information transfer of cue position across the two hemispheres. Moreover, the effect was maximal when the target was presented 10 ms after the intervening saccade ended and was attenuated in longer delays. In our case, therefore, the representation of previously attended locations (as revealed by IOR) is not remapped slowly after the execution of a saccade. Rather, either a retinotopic representation is remapped rapidly, adjacent to the end of the saccade (using a prospective motor command), or the positions of the cue and target are encoded in a spatiotopic reference frame, regardless of eye position. Spatial attention can therefore be allocated to target positions defined in extraretinal coordinates.
Understanding the intricacies of a complex visual scene often requires scanning it with multiple eye movements that direct the high-resolution fovea to behaviorally relevant targets. With each eye movement, the retinotopic (eye-centered) coordinates of a visual object change while the head-centered coordinates remain stable. Although the predominant representation associated with early visual processing is retinotopic (Gardner et al., 2008), it has been demonstrated that the brain constructs multiple spatial representations to guide action (Andersen et al., 1997; Colby, 1998; Colby and Goldberg, 1999). More complex spatial representations are needed to integrate information across different senses (such as in audiovisual processing) as well as to guide motor movements (such as reaching), which can be accurate even when the object of interest is outside the visual field. Recently, however, Golomb et al. (2008), who studied the reference frame of attentional facilitation (using a task-relevant cue), suggested that the native representation of spatial attention in the human visual system is retinotopic. This claim seems to be somewhat overstated as there is no a priori reason to believe that there is a single “native” coordinate frame in which attention operates. For example, hemispatial neglect, which is usually considered as an attentional deficit, can be evident in various reference frames (e.g., object, head, body, or world based) other than the retinotopic one (Behrmann and Tipper, 1999; Vuilleumier et al., 1999; Landis, 2000). We use here the inhibition of return (IOR) effect (Posner and Cohen, 1984), which is characterized by performance decrement for a target stimulus appearing at a previously cued location. Performance decrement is often found when the interval between the appearance of an attention-grabbing cue and the target is long enough (∼300 ms or more) and the cue is irrelevant for the task. Current interpretations of the IOR effect differ in the suggested underlying mechanisms but concur in viewing inhibition as the expression of some form of spatially selective refractoriness of the attentional system (Kingstone and Pratt, 1999; Klein, 2000). The original “Posner task” (Posner and Cohen, 1984) was modified here such that an intervening saccade was executed after the appearance of the cue and before the presentation of the target. This allowed separating between retinotopic trials, in which the position of the target on the retina was the same as the cue (but the two appeared at different positions on the screen), and spatiotopic trials in which the cue and target shared the same screen position (but their projections occupied different loci on the retina). Previous studies (Maylor and Hockey, 1985; Sapir et al., 2004) indicated that, under such conditions, the IOR is evident also at the spatiotopic location of the cue. In the present study, we used a gaze-dependent display technique, which allowed us to examine the dynamics of target mapping around the time of the saccades (in the two coordinate systems) by measuring the dependence of the magnitude of the IOR effect on the time that elapsed from the completion of the intervening saccade to the appearance of the target.
Materials and Methods
Twenty-five (16 females) university students (ages, 23.1 ± 1.3 years) took part in ∼1 h of experiment and completed three or four experimental runs in return for course credit. Each run included a total of 102 trials, randomized independently for each run and subject. Six additional subjects did not complete all required experimental runs because of having a large number of discarded trials, mainly because of difficulty in performing the required eye movements. The data of these subjects were not considered in the reported analysis. All subjects gave written informed consent and had normal or corrected-to-normal visual acuity by self-report. Experimental procedures were approved by the ethics committee of the Psychology Department at Ben-Gurion University in Israel.
Stimuli and experimental settings.
Stimuli were displayed on a 19 inch CRT monitor (Graphics Series G90fB; View Sonic) at a resolution of 800 × 600 pixels with a refresh rate of 100 Hz. Subjects were seated 60 cm from the monitor with their head positioned at a forehead and chin rest so that the entire display subtended ∼34 × 25° of visual angle. The experiments were conducted in a dimly lit room with ambient illumination from the monitor and eye-tracking equipment. All stimuli were white (118 cd/m2) and were plotted against a gray background (7 cd/m2). The cue was a square frame subtending 1.8 × 1.8° of visual angel and could appear in one of four possible locations: 5° from fixation in 45 and 135° to either side of the vertical meridian. The temporal flow of a typical trial is described in Figure 1.
A run had the following possible combinations: 4 (cue locations) × 2 (directions of intervening saccade) × 3 (delay durations) × 4 (target positions), adding up to 96 different trials. Thus, there were 12 trials for each of the major conditions (retinotopic, spatiotopic, and two controls) when the cue and intervening-saccade destination were on the same side of the initial fixation, and when the two were on opposite sides (see Fig. 2). In terms of the design of the control conditions, the present study provides an improvement of the design of the study by Golomb et al. (2008), since there are individually matched control locations for spatiotopic and retinotopic conditions, rather than just a single control condition. This modification is important because spatiotopic and retinotopic conditions may involve different attentional shifts with respect to the intervening saccade direction. An additional six catch trials with no target (hence requiring no response) were added to each run. Auditory and visual feedback was given after each trial, indicating whether subjects succeeded or failed (see below, Eye tracking) to initiate the proper responses. Incorrect trials were discarded and repeated later in the run.
A video-based desk-mounted eye tracker (Eye Link1000; SR Research) with a sampling rate of 1000 Hz was used for recording eye movements. We used built-in programs provided with the eye tracker for calibration and validation purposes (nine points presented in a random sequence). All the data analyzed in the present paper were obtained from recordings with an average Cartesian prediction error of <1° during the validation procedures. A trial began by presentation of a fixation point at the center of the screen. The subjects triggered the stimulus display while fixating this point at their own pace (by key press). The data obtained during this control fixation period were used to correct for slow drifts of the eye tracker.
Eye tracking was used for the following three purposes.
The first purpose was to determine on-line the time at which the target should be presented (10, 250, or 600 ms after the completion of the intervening saccade). This obviously required on-line assessment of the time at which the intervening saccade ended and subjects attained the new fixation location. Saccade duration was defined as the time in which the eye velocity (30°/s) and acceleration (8000°/s2) thresholds were crossed for at least two consecutive samples (and the pupil is visible, for dissociating blinks). Subthreshold velocity and acceleration imply the end of saccade and the beginning of fixation, signaling the onset of the delay period.
The second purpose was to disqualify trials in which the subjects failed to initiate the required saccades on time. A saccade was regarded valid if it ended within a 1.4° radius from the destination point and was initiated within predefined time limits (150–500 ms after target onset). In catch trials, in which no target was presented, any crossing of the velocity and acceleration threshold during the 1.5 s after the first intervening saccade disqualified the trial. Trials were disqualified when gaze was out of the 1.4° radius perimeter during the cue presentation and fixation period.
The third purpose was to report the saccade response time (SRT) after the target appearance. SRT was determined as the time between target presentation and the beginning of a saccade with valid destination and within the predefined time limits (150–500 ms).
Subjects performed a modified cued-location saccadic reaction time task, in which an intervening saccade was executed between cue and target presentation. After a brief cue presentation, they made an intervening saccade to the left or right of their initial fixation. Then, after a variable postsaccadic interval, the target appeared and they responded by executing a saccade toward the target (Fig. 1). When the target was shown at the same spatiotopic or retinotopic locations as the initial cue, SRTs were longer compared with control locations (two-tailed t test; p < 0.0005; collapsed across all delay durations), demonstrating the known phenomenon of IOR. We measured the IOR as the difference in the median response time between each condition (retinotopic and spatiotopic) and its specific control condition, separately for each subject and delay duration. We further divided the trials to (1) those in which the cue and the intervening-saccade destination were at the same side relative to the initial fixation (“same side”) (Fig. 2, left panels) and (2) trials in which the cue and intervening-saccade destination were at opposite sides of fixation (“opposite sides”) (Fig. 2, right panels). Note that, in same-side trials, the intervening saccade changed the gaze, such that the spatiotopic target was now present in the hemifield opposite to the cue (Fig. 2, new vertical midline, depicted in red dashed line), whereas in the opposite-sides trials, the spatiotopic target was shown in the same hemifield as the cue. Note, furthermore, that the retinotopic matched targets (by definition) are always on the same hemifield as the cue.
A repeated-measures ANOVA with reference frame (cue location: spatiotopic/retinotopic) by cue-intervening-saccade directions (same/opposite) by delay duration (10, 250, and 600 ms interposed between the end of the intervening saccade and the appearance of the target) on the IOR effect revealed significant main effects of the cue-intervening-saccade directions (F(1,24) = 16.1, p < 0.001) and delay duration (F(2,48) = 5.3, p < 0.01), indicating that opposite-sides trials tend to have bigger IOR effect and that the effect diminishes with time after the termination of the intervening saccade. Interestingly, the ANOVA also revealed a significant interaction between cue-intervening-saccade directions and reference frame (F(1,24) = 5.0; p < 0.05), suggesting that behavior was sensitive to the coordinate frame in which the cue and target matched, only in opposite-sides trials. A direct comparison between the two conditions (cue–target match in retinotopic vs spatiotopic coordinates) showed that the IOR of the 10 ms delay was significantly larger when the target matched the cue position in their spatiotopic coordinates (two-tailed paired t test, p < 0.05) (Fig. 3B). This was evident only in opposite-sides trials. The crucial element in these trials was that, when the cue and target matched in spatiotopic coordinates, they were present in the same hemifield with respect to fixation at their corresponding time of presentation [the hemifield of the cue and target should be assessed with respect to the vertical midline before and after the intervening saccade, indicated in blue and red, respectively (Fig. 2)]. In contrast, the same-side trials were characterized by the fact that the spatiotopic-matched target was present in the other hemifield compared with the original cue. The fact that the enhanced IOR was limited to the visual hemifield in which the cue was presented is in agreement with several previous studies that suggested that there are limits to IOR transference between the two hemispheres (Rizzolatti et al., 1987; Ro et al., 2000) (but see Dori and Henik, 2006). Note also that the enhanced spatiotopic IOR effect (at the screen location of the cue) could not be attributed to a difference in the absolute SRT, which might be caused by the larger distance of the spatiotopic targets (from fixation) in the opposite-sides trials: The enhanced IOR effect was composed of a longer SRT to spatiotopic matched targets as well as decreased SRT to the specific control location (Fig. 3A). Linear-trend analysis for the delay duration for each of the four conditions revealed a significant linear trend in the spatiotopic IOR, only in opposite-sides trials. The spatiotopic IOR diminished as the interval increased (F(1,24) = 7.0; p < 0.02). To summarize, inspecting the IOR effects in various delays after the intervening saccade, reveals that the most prominent spatiotopic effects are evident soon after the end of the saccade and decrease with time, only when the cue and eventual target (after the intervening saccade) were presented in the same visual hemifield (Fig. 3B).
In their seminal study, Posner and Cohen (1984) have shown that attentional facilitation is encoded in retinotopic coordinates, whereas the IOR is primarily coded in environmental/spatiotopic coordinates. With modern gaze-dependent presentation techniques, we were able to carefully study the dynamics of the IOR effect in the two coordinate systems. IOR was primarily found for a target appearing at the same spatiotopic position as the cue. Critically, these spatiotopic effects occurred when the target was shown as early as 10 ms after the completion of the intervening saccade. Moreover, the magnitude of the IOR spatiotopic effect was maximal at the shortest delay and was attenuated in longer delays. The latter finding may seem somewhat counterintuitive, because one might expect that remapping from the initial retinotopic coordinate frame (to take into account the new eye position) takes time, and therefore the spatiotopic effects should be seen late rather than immediately after the completion of the intervening saccade. Indeed, Golomb et al. (2008) found that the perceptual advantage for a target positioned at the same spatiotopic location as the cue was not present immediately after the intervening saccade and increased with time. Interestingly, this spatiotopic effect occurred only when task demands required maintaining the veridical position of a cue (across an eye movement). Golomb et al. (2008) concluded that “endogenously maintained spatial attention is retinotopic, and remapping of attention to spatiotopic coordinates occurs slowly and only when behaviorally necessary.” We focus here on the inhibitory phase of spatial attention (IOR); our data indicate that an updated spatiotopic representation of a previously attended location is not slowly generated after the execution of the intervening saccade. Rather, it seems to be rapidly formed prior or adjacent to the completion of the intervening saccade. This immediate spatiotopic cue tagging occurs even when the cue is not behaviorally informative (i.e., it cannot predict the future target position). Thus, the dynamics of the attentional effect reported here [and in another recently published paper by Mathôt and Theeuwes (2010)] are markedly different from those reported by Golomb et al. (2008). It is possible that a purely retinotopic representation of attention occurs in the case of attentional facilitation when using a task-relevant cue and short delays between cue and target. However, the discrepancy between the studies could also be attributed to the different mode of response that was used, as we recorded saccadic reaction times, whereas Golomb et al. (2008) used manual reaction time (but see Abrams and Pratt, 2000). In any case, our study clearly demonstrates that a definitive conclusion about the native reference frame of attention is not tenable. Thus, although some experimental conditions manipulating spatial attention may favor the use of retinotopic representations as in the case of the study by Golomb et al. (2008), other conditions may reveal spatiotopic representations that perhaps involve neuronal populations with more complex receptive fields. A similar example for a sensitivity to the exact experimental factors is evident in the case of transsaccadic integration: Using static visual patterns, McKyton et al. (2009) found that integration is strictly retinotopic, whereas integration of subthreshold motion signals was evident in a spatiotopic coordinate frame (Melcher and Morrone, 2003).
We show that the representation of attended locations, in the inhibitory stage (IOR), is spatiotopic immediately after the intervening saccade ends. Additional experiments should be conducted to understand whether the inhibitory tag is initially coded in (at least) head-based coordinates, or if it is encoded in retinotopic coordinates and remapped before (or in conjunction with) the intervening saccade in a predictive manner (possibly using the motor efference copy signal). Both alternatives have some neurophysiological evidence to support them as viable options. When an eye movement is about to occur, many of the neurons in monkeys' parietal cortex transiently shift the location of their receptive field, as expected according to the predictive remapping scenario. Such neurons start responding to the presence of a visual stimulus even before the saccade (that will eventually bring the stimulus to the receptive field of the neuron) is initiated (Duhamel et al., 1992; Colby et al., 1995). Functional imaging studies in humans show results along the same line in the parietal (Merriam et al., 2003) and occipital cortex (Merriam et al., 2007). Recent behavioral studies in humans also showed evidence for predictive remapping using adaptation techniques (Melcher, 2007; Melcher and Colby, 2008), object-based remapping across saccades (Melcher, 2008), and remapping of spatial attention (Mathôt and Theeuwes, 2010). However, there is also ample evidence for visual encoding of information in a head-based reference frame in the monkey's parietal cortex. Neurons in the posterior parietal lobe are typically sensitive to both the position of the stimulus on the retina and the position of the eye in the orbit. The eye position sensitivity of visually responsive neurons is often termed “gain field” to indicate that the neurons typically have a receptive field in retinotopic coordinates that is modulated by the position of the eye in the orbit (Andersen et al., 1985, 1990). Eye position-independent location could be extracted from a population of such neurons with gain field properties (Zipser and Andersen, 1988). Alternatively, this information could be represented explicitly by single neurons with spatiotopic receptive fields; however, we note that such neurons were only rarely found in the posterior parietal lobe (Duhamel et al., 1997; Galletti et al., 1993; Mullette-Gillman et al., 2005).
Finally, our results may also have implications for gaining better understanding of the neural processes underlying hemispatial neglect. Neglect patients vary significantly in the locus of their lesion, its extent, and the coordinate frame in which their neglect is most pronounced (Vuilleumier et al., 1999; Landis, 2000). Consider the performance of neglect patients with a right parietal lesion on a visual search (cancellation) task. In such a task, patients typically refixate visited targets in their supposedly intact right visual field much more frequently than healthy controls (Husain et al., 2001; Mannan et al., 2005). Indeed, Mannan et al. (2005) suggested that “difficulties in keeping track of previously inspected targets (because of impaired transsaccadic remapping and/or memory) may be one contributing component to visual neglect … .” According to this view, the combination of an ipsilesional attentional bias together with an impairment to retain memory for locations already searched, may explain why stimuli on the right are repeatedly reexamined, as if being searched for the first time, and hence why stimuli on the left continue to be ignored even with unlimited viewing time.
A more daring suggestion argues that an impaired transsaccadic spatial remapping process may be responsible for various symptoms in neglect patients (Pisella and Mattingley, 2004). In an apparent agreement with such an account, spatiotopic IOR (but not the retinotopic one) was found to be absent in patients with parietal lesions, especially in the right hemisphere (Sapir et al., 2004). Furthermore, a recent study generating a virtual parietal lesion in healthy controls using transcranial magnetic stimulation (TMS) showed that a double-pulse TMS to the right (but not left) intraparietal sulcus abolished the spatiotopic IOR effect (van Koningsbruggen et al., 2010). Critically, in this case, the TMS was applied 150 and 250 ms after the saccade onset, hence ∼100–200 ms after it ended. Given our results, this interval seems long enough to suggest that, at the time of stimulation, the spatiotopic IOR was probably already fully developed. Therefore, it is unlikely that the remapping process was impaired. Rather, we suggest that the already existing spatiotopic representations were affected by the stimulation. Therefore, failure in the maintenance of spatiotopic representations (rather than a remapping process), after right parietal lesion, may explain some of the symptoms of hemispatial neglect.
In conclusion, the goal of this study was to investigate, using behavioral means, the coordinate frame of the inhibitory trace of attended locations. To that end, we used a gaze-dependent display, allowing investigation of the dynamics of IOR in a cued-target reaction time task with an intervening saccade between cue and target presentation. When the cue and target were shown in the same visual hemifield, the IOR effect was most prominent when the target was present at the same spatiotopic position as the cue. This effect was most notable when the target was presented 10 ms after the intervening saccade ended. These findings suggest that the representation of cued locations (in the inhibitory stage) is not slowly remapped after the execution of a saccade. Rather, either a retinotopic representation is remapped adjacent to the end of the saccade (using the prospective motor command) or the position of the cue and target are encoded in (at least) a head-based reference frame, regardless of eye position. This study complements a handful of studies showing evidence for coordinate transformation in the human brain, by establishing the fact that the inhibitory effect of attended locations is mapped in spatiotopic coordinates. Importantly, this demonstrates that attentional processes are not exclusively mapped in a retinal coordinate frame, as previously claimed (Golomb et al., 2008).
This work was supported by Israel Science Foundation Grant 39/09 (E.Z.) and by National Institute for Psychobiology in Israel Grant 2-2008-09 (G.A.). We thank Shai Gabay for helpful discussions and comments, and Daniel Reznik for his help with data acquisition.
- Correspondence should be addressed to Galia Avidan, Department of Psychology, Ben Gurion University of the Negev Israel, Beer-Sheva 84105, Israel.