Abstract
It is known that spatial localization of flashed objects fails around the time of rapid eye movements (saccades). This mislocalization is often interpreted in terms of a combination of shifts and deformations of the brain's representation of space to account for the eye movement. Such temporary remapping of positions in space should affect all elements in a scene, leaving ordinal relationships between positions intact. We performed an experiment in which we presented flashes on a background with red and green regions to human subjects. We found that flashes that were presented on the green part of the background around the time of a saccade were readily reported to have been presented on the red part of the background and vice versa. This is inconsistent with the notion of a temporary shift and deformation of perceived space. To explain our results, we present a model that illustrates how temporal uncertainty could give rise to the observed spatial mislocalization. The model combines uncertainty about the time of the flash with a bias to localize targets where one is looking. It reproduced the pattern of mislocalization very accurately, showing that perisaccadic mislocalization can best be explained in terms of temporal uncertainty about the moment of the flash.
Introduction
When people shift their gaze, retinal information needs to be remapped using extra-retinal information about eye orientation to maintain a stable representation of objects' locations. If the eye orientation changes rapidly, as it does during saccades, small time differences between the retinal and extra-retinal signals will lead to transient errors in the mapping. This does not necessarily cause any problems in judging the objects' locations, because most objects remain visible well before and after the saccade. However, for briefly presented objects near the time of a saccade, even a small time difference, such as could arise from low-pass characteristics of retinal (Pola, 2004) or extra-retinal (Dassonville et al., 1992; Schlag and Schlag-Rey, 2002) signals, could give rise to the systematic shift of apparent positions; a phenomenon that has often been reported (Matin and Pearce, 1965; Bischof and Kramer, 1968; Mateeff, 1978; Honda, 1991; Dassonville et al., 1992; Schlag and Schlag-Rey, 2002). Apart from overall shifts, a compression of apparent positions toward the saccade target has also been found (Morrone et al., 1997; Ross et al., 1997; Lappe et al., 2000). It is not directly evident how timing errors could account for this compression just before and during saccades. The deformation of the pattern of apparent positions may tell us something about the brain's remapping of visual space (Ross et al., 1997; Lappe et al., 2006).
The fact that the shape, color (Lappe et al., 2006), and size (Matsumiya and Uchikawa, 2001) of an object, and the separation between objects (Reeve et al., 2008) flashed near the time of saccades is perceived veridically suggests that the effects are specific to localization. Temporarily shifting and compressing a representation of positions in space should displace background elements as well as flashed objects (Ross et al., 1997), leaving ordinal relative positions intact. However, Lappe and colleagues (2006) show that objects presented just before saccades can be perceived to be on a different background element than the one on which they were flashed. If so, then the ordinal positions do change, and the errors cannot be attributed to an overall compression of perceived positions. To investigate this systematically, we explicitly asked subjects to indicate the color of the background onto which a white bar was flashed.
Materials and Methods
Six subjects (one male and five females; including one of the authors) sat in front of a touch screen (EloTouch; CRT 19”, 85 Hz, 1024 × 768 pixels, 36 × 27 cm, 40° × 30° of visual angle) in a room with normal illumination. Eye movements were recorded using an Eyelink II (SR Research; standard nine-point calibration procedure) at a sample frequency of 500 Hz. The orientation of the right eye was used to identify saccades (speed threshold of 35°/s).
Subjects followed a 0.5° diameter jumping black dot with their eyes. The dot made three to six 12° jumps in various directions (one every 400 ms) before making a last 12° horizontal jump, either to the right or to the left. At a variable time after the last jump, a 0.5° × 12.3° vertical white bar was flashed for one frame at one of three different locations: −2.4°, 2.4°, or 15.6° from where the saccade most likely started (Fig. 1A). The background was red or green (isoluminance determined individually by flicker photometry before the experiment) with a 14.4° wide vertical band of the other color (from the saccade start location to a position 2.4° beyond the saccade target). In one session, subjects indicated the color of the background of the white bar by pressing the R or G key (484 trials). In another four sessions (600 trials each), they indicated the location of the bar by touching the screen. In these four sessions, there were 150 trials with red–green backgrounds (the other 450 contained conditions that were not analyzed; the trials were presented in random order). As in previous studies (Maij et al., 2010), we excluded trials from further analysis if either the saccade or the touched location were clearly inappropriate.
Results
Subjects regularly reported the incorrect background color (Fig. 1B). All six subjects reported the incorrect background color some of the time for bars flashed at −2.4°, and five subjects did so for some of the bars flashed at 15.6°. The number of reports of the incorrect background color depended on when (relative to the saccade) and where the bar was flashed (Fig. 2A). The pattern of errors in judging the background color was consistent with the pattern of mislocalization that we found when subjects were asked to touch the screen where they had seen the white bar (Fig. 1C). All six subjects localized some of the flashed bars at −2.4° and 15.6° on the incorrect background color. The proportion of trials in which the mislocalization was so large that the bar was localized on the incorrect background color (Fig. 2B) corresponds with the number of trials in which the color of the background was misreported (Fig. 2A).
Overview of the task and representative results. A, Subject was instructed to follow a jumping black dot with their eyes. A variable time after the dot jumped horizontally, a vertical bar was flashed for one frame. The bar was presented in one of three possible locations. Subjects were either asked to indicate on which background color they had perceived the bar or to indicate the location of the bar by touching that location. B, The number of flashes for which a representative subject named the correct (green) or incorrect (red) background color for flashes at −2.4°, as a function of the time of the flash (relative to saccade onset). Gray shading indicates the average saccade duration. C, The indicated location of the bar for the same subject as in B. The curved lines are averages based on a moving Gaussian window (σ = 7 ms). Black line, Orientation on a representative trial; dotted lines, the three flash positions; dashed lines, location of the start of and target for the saccade.
Percentage of trials in which the subjects perceived the flash on a differently colored part of the background. A, Percentage of trials in which subjects named the color of the background incorrectly. B, Percentage of trials in which subjects localized the bar at a position with the wrong background color. Purple, violet, and blue shading indicates SEM across subjects. Gray shading indicates the average saccade duration. Asterisks indicate times at which the percentages are significantly larger than zero (across subjects, p < 0.05).
Discussion
The pattern of mislocalization shows a similar compression of visual space around the moment of the saccade, as has been previously reported (Morrone et al., 1997; Ross et al., 1997; Lappe et al., 2000). The reported color of the background confirms the findings of Lappe and colleagues (2006) and expands the evidence to bars flashed during saccades. The errors in reporting the background color imply that the background color is not judged within a scene that is temporarily deformed in accordance with the mislocalization of objects flashed during the saccade (Ross et al., 1997). This could mean that the background is not deformed, but it is also possible that only the location of the flashed bar is judged during the saccade, and that the color of the background is determined after the saccade. The latter explanation allows temporary deformations of the background during the saccade, but does not require it. In combination with evidence that shapes (Matsumiya and Uchikawa, 2001) and separations (Reeve et al., 2008) are perceived correctly for targets flashed during saccades, a parsimonious account of the findings is that temporal errors, when combining information about the flashed object's retinal location with information about eye orientation, are responsible for the observed perisaccadic mislocalization of flashed objects (see Model interpretation, below). Such a temporal explanation is also in line with our finding that temporal information can change perisaccadic mislocalization (Maij et al., 2009).
Model interpretation
It has been suggested that a combination of temporal uncertainty and a bias toward believing that one is looking at what one sees is the basis of various localization errors, including ones related to eye movements (Brenner et al., 2006). When uncertain, people's judgments are influenced by prior expectations. There is usually a strong correlation between where one is looking and where one sees things, so when one is uncertain about the position of a flash, there will be a bias toward perceiving it where one was looking at the time of the flash (Brenner et al., 2008). We present a simple model that combines temporal uncertainty with a foveal bias to explain the mislocalization pattern.
The model combines a normally distributed temporal uncertainty about the time of the flashed object (Fig. 3A, pink area) with a saccadic eye movement [saccade modeled as a minimum jerk movement (Flash and Hogan, 1985) of 11° in 50 ms] (Fig. 3A, black curve) to obtain the likely orientation of the eye (Fig. 3A, orange area). The average of this orientation corresponds to the mislocalization because of temporal uncertainty alone (Fig. 3B). The pattern of the predicted locations depends on the time of the flash in a manner similar to the experimentally obtained pattern of mislocalization occurring when flashed objects are presented near the time of saccades in the dark: a time-dependent shift of the apparent positions of the objects that is independent of their spatial locations (Honda, 1991). This influence of temporal uncertainty is equivalent to the proposed shift of apparent positions as a result of low-pass characteristics of retinal (Pola, 2004) or extra-retinal (Schlag and Schlag-Rey, 2002) signals. It is also consistent with temporary shifts in neuronal responses in various brain areas near the time of saccades (Duhamel et al., 1992; Walker et al., 1995; Colby et al., 1996; Umeno and Goldberg, 1997). Since the localization errors arise from misjudging the eye orientation, the same mislocalization is predicted for targets at all positions. We show that, if one considers the bias toward the direction of gaze at the time of the flash, the model will predict deviations of the mislocalization patterns from uniform shifts.
A simple model that can account for the mislocalization pattern. A, Combining a normally distributed temporal uncertainty (illustrated in pink for two moments of the flash; σt = 15 ms; top, μt = 10 ms; bottom, μt = 30 ms) with the eye orientation, gives the likely distribution of eye orientations shown on the left (orange). Note that the mean values of the orientations (arrows at 2.9° and 6.6°) do not correspond with the orientations of the eye (blue dots at 1.1° and 7.5°). B, Combining these orientation distributions with the retinal stimulation gives likely distributions for the position of the flashed object in space for each moment of time. The means of the distributions for the three locations (curved lines) look similar to the mislocalization curves that are found when one performs these kinds of experiments in the dark. C, The mislocalization curves are different if the distribution for the position of the flashed object in space is combined with a foveal bias (σs = 5°), centered where an efferent eye signal that precedes the actual eye movement by 20 ms indicates that the eye is looking. Note the resemblance with Figure 1C.
Our model does not include any explicit spatial uncertainty. However, the width of the above-mentioned likelihood of the eye orientation (Fig. 3A, orange area) indicates the uncertainty about the orientation of the eye. Assuming that the uncertainty about the retinal location of the flashed object is negligible, the uncertainty about the orientation of the eye is equivalent to the spatial uncertainty about the flashed object. To take the foveal bias into account, we multiply the distribution of possible object locations with a normally distributed foveal bias centered on the position toward which gaze is directed just after the flash. The mean of the resulting distribution is the outcome of the model. If we determine this for each moment for each of the three locations, we obtain mislocalization curves (Fig. 3C) that are similar to the experimentally obtained ones shown in Figure 1C.
General discussion
One important experimental finding about perisaccadic mislocalization is that localization errors are independent of the (retinal) location of the flashed object when experiments are conducted in complete darkness, whereas the errors depend systematically on the location of the flash when visual references are available after the saccade (Lappe et al., 2000). More recently, it has been shown that there is a gradual transition, with performance depending on the stimulus luminance (Georg et al., 2008) and contrast (Michels and Lappe, 2004). Our model can account for this transition through the weight given to the foveal bias. For little spatial uncertainty, the bias gets little weight, so the pattern looks like a uniform shift (Fig. 3B). In situations of high uncertainty, the bias will get more weight, resulting in a compression of the perceived locations toward the saccade target (Fig. 3C). It is possible that a background shifting across the retina after the flash (when experiments are not conducted in complete darkness) increases the uncertainty. This increase in uncertainty will be higher for a high-contrast background, leading to the increase in compression that has been reported (Michels and Lappe, 2004). Reducing the luminance to near-threshold values is also likely to increase uncertainty, and will, according to our model, thus lead to more compression, which indeed has been found (Georg et al., 2008).
We are not the first to model the effects of eye movements on localization of flashed stimuli. The uniform shift has been modeled by Pola (2004), based on retinal signal persistence, and by Dassonville et al. (1992), using a damped representation of the eye-movement signals. Other models only focus on the perisaccadic compression. Ross et al. (1997) modeled the compression using a complex exponential. Richard et al. (2009) modeled deformations on the basis of nonlinearities in brain representations. Niemeier et al. (2003) have argued that using the saccade target as a prior for the flash location results in optimal trans-saccadic integration. Their model is similar to ours, but their model is not directly applicable to the compression during saccades. Hamker et al. (2008) have a different modeling approach. They used neuronally inspired simulations with many parameters to link perisaccadic mislocalization to attentional phenomena. Our model differs from the latter models (Niemeier et al., 2003; Hamker et al., 2008) in that the bias is toward fixation rather than toward the saccade target. It differs from all others in being able to account for uniform shifts as well as compression.
Objects are only mislocalized near the time of saccades if they cannot be localized reliably well before or after the saccade. For such objects, one has to combine signals at a time at which doing so is sensitive to temporal uncertainty. We show that the errors that one observes under such conditions can be accounted for by temporal uncertainty combined with a foveal bias. Thus, eye-movement related errors probably tell us more about timing in vision and oculomotor control than about special mechanisms for achieving visual stability.
Footnotes
-
This research was supported by the Netherlands Organization for Scientific Research (Grant 816-02-017). We thank Siro Otten and Frans-Josef Halkes for their technical support and Rouwen Cañal-Bruland, Idsart Kingma, Lieke Peper, and Rob van Beers for their comments.
- Correspondence should be addressed to Femke Maij, Research Institute MOVE, Faculty of Human Movement Sciences, VU University, Van der Boechorststraat 9, 1081 BT Amsterdam, The Netherlands. f.maij{at}fbw.vu.nl
