Abstract
The dorsal aspect of the medial superior temporal area (MSTd) is involved in the computation of heading direction from the focus of expansion (FOE) of the visual image. Our laboratory previously found that MSTd neurons adjust their focus tuning curves to compensate for shifts in the FOE produced by eye rotation (Bradley et al., 1996) as well as for changes in pursuit speed (Shenoy et al., 2002). The translation speed of an observer also affects the shift of the FOE. To investigate whether MSTd neurons can adjust their focus tuning curves to compensate for varying translation speeds, we recorded extracellular responses from 93 focus-tuned MSTd neurons in two rhesus monkeys (Macaca mulatta) performing pursuit eye movements across displays of varying translation speeds. We found that MSTd neurons had larger shifts in their tuning curves for slow translation speeds and smaller shifts for fast translation speeds. These shifts aligned the focus tuning curves with the true heading direction and not with the retinal position of the FOE. Because the eye was pursuing at the same rate for varying translation speeds, these results indicate that retinal cues related both to translation speed and extraretinal signals from pursuit eye movements are used by MSTd neurons to compute heading direction.
Introduction
It has been proposed that, for an individual in motion, the focus of expansion (FOE) of the retinal image could be used to determine the direction of heading (Gibson, 1950). However, when eye and head movements are present, the FOE no longer corresponds with heading direction (Longuet-Higgins and Prazdny, 1980; Koenderink and van Doorn, 1986). Imagine an observer walking down the road while their eyes are fixating on an airplane flying across the horizon from left to right. As the observer moves forward and follows the moving plane with their eyes, the eyes rotate in the head. This eye movement adds a laminar motion to the forward translation component of the visual field that is in the opposite direction of the eye movement. As a result, the retinal FOE is shifted in the direction of eye movement and no longer indicates the true heading direction (Fig. 1A).
Schematic of flow fields on the retina. Retinal motion patterns caused by observer translation and pursuit eye movements. Comparing row A to row B shows that the FOE shifts more on the retina for faster pursuit speeds than for slower pursuit speeds. The FOE shift is also affected by the speed of translation. This can be seen by comparing rows B and C. Note that, although the pursuit speeds and translation speeds are different for row A and row C, the FOE location is the same.
Neurons in the dorsal aspect of the medial superior temporal area (MSTd) have many properties that suggest they are involved in self-motion perception such as large receptive fields, selectivity for optic flow patterns (Komatsu and Wurtz, 1988a,b; Duffy and Wurtz, 1991a,b; Graziano et al., 1994; Lagae et al., 1994), tuning for the retinal position of the FOE (Duffy and Wurtz, 1995), and tuning for the rate of expansion (Duffy and Wurtz, 1997). In addition, Britten and van Wezel (1998) found that microstimulation of MSTd neurons while monkeys made heading judgments biased their perceived heading direction during pursuit eye movements.
Research from our laboratory has found that neurons in MSTd adjust their focus tuning in response to the laminar motion produced by eye rotations, thereby recovering the true FOE, which corresponds to the true heading direction (Bradley et al., 1996; Shenoy et al., 1999). Because the FOE shifts more on the retina for faster pursuit speeds (PSs), greater focus tuning compensation is required as pursuit speed increases. In line with this, MSTd neurons have been shown to adjust their focus tuning in accordance with changes in eye pursuit speed (Shenoy et al., 2002).
The translation speed (TS) of an observer also affects the shift of the FOE. This can be seen by comparing rows B and C in Figure 1. Faster translation causes the FOE to shift less on the retina than slower translation. If MSTd neurons are involved in self-motion perception, then they should demonstrate varying compensation as translation speed changes. We recorded neural responses from 93 MSTd neurons as monkeys were presented with optic flow stimuli simulating 11 heading directions. The forward translation was simulated at three different translation speeds while the monkeys were required to fixate or pursue across the display. This experimental paradigm allowed us to examine the effects of varying the translation speed for a given pursuit speed. The results we report here indicate that area MSTd does in fact compensate for changes in translation speed, which suggests it combines retinal cues related to translation speed with extraretinal signals related to eye movements to compute heading direction.
Materials and Methods
The methods described in this section have been previously reported (Bradley et al., 1996; Shenoy et al., 1999, 2002). We will briefly review them here.
Animal preparation.
The Caltech Institutional Animal Care and Use Committee approved all protocols. We recorded extracellular responses from 93 MSTd neurons in two rhesus monkeys (Macaca mulatta). Both monkeys were naive to the experimental paradigm. We implanted bone screws into the skull on which a methylmethacrylate head cap was built. This head cap featured a titanium head post that allowed the monkey's head to be immobilized. In monkey DON, a scleral search coil was implanted between the conjunctiva and sclera and used to monitor eye position at 1000 Hz (Judge et al., 1980). The search coil was connected to a coaxial connector located on the methylmethacrylate head cap. For monkey ROY, we used an optical eye tracker to monitor eye position at 240 Hz (ISCAN, Burlington, MA). Behavioral training commenced ∼1 week after surgery. During training, the monkeys were given a juice reward for correctly performing each trial. After several weeks of training, performance levels reached >90%. At this time, we performed a second surgery to open a craniotomy and implanted a surface-normal chronic recording chamber. The recording chamber was located at 5 mm posterior, 17 mm lateral, on the right hemisphere in both DON and ROY.
Recording techniques.
We recorded extracellular action potentials with glass-coated tungsten microelectrodes with impedances between 1.0 and 1.5 MΩ (Alpha-Omega, Nazareth, Israel). Below the chronic recording chamber, a stainless-steel guide tube was manually advanced through the dura, and then the electrode was further lowered into the cortex via an FHC hydraulic microdrive (FHC, Bowdoinham, ME). As the electrode was advanced, a visual stimulus consisting of an expansion flow pattern was displayed on the screen. Once neural activity that corresponded with the visual stimulus was recorded, the electrode was allowed to settle for 30 min. MSTd neurons were identified by stereotaxic coordinates, magnetic resonance anatomical images, depth in the chamber, position relative to other cortical areas, and response properties such as optic flow tuning and large receptive fields that were both ipsilateral and contralateral. Neural data were sampled at 20,000 Hz and recorded and analyzed with custom software. All experiments were performed in an acoustically and radio frequency shielded room.
Visual stimuli.
The monkeys performed all the tasks in complete darkness, except for the visual stimuli. The visual stimulus was displayed on a high-resolution flat screen cathode ray tube monitor at a resolution of 800 × 600 and frame rate of 120 Hz. Each pixel was ∼0.07°. This monitor was placed 38.1 cm from the eyes. The stimulus was 20 × 20° in size and contained 400 stimulus dots. The dots were white (10 cd/m2) on a black background and antialiased for smooth movement. Each dot was given an age between 0 and 287 ms (uniform distribution) and traveled at a constant velocity until 300 ms expired, or until it crossed the edge of the stimulus window, in which case it was moved to a new random position in the stimulus. The speed of the stimulus dots increased in proportion to the distance from the focus of expansion. Both monkeys viewed the stimulus monocularly with the left eye to eliminate stereo cues. Stimulus dots were 2.5 × 2.5 pixels in size, whereas fixation points were larger at 5 × 5 pixels.
Once a neuron was isolated, we mapped out the receptive field by displaying an expansion optic flow stimulus at different positions on the screen. The center of the stimulus was tested at 0,0°; +5,+5°; +5,−5°; −5,−5°; and −5,+5° with respect to the fixation point, which was always located at 0,0°. The position with the strongest response to the expansion stimuli was then used for all subsequent experiments. In addition, all pursuit trajectories, real or simulated, were centered around the fixation point (0,0°) on the monitor to remove gaze angle effects.
Characterization tasks.
We first ran three sets of characterization tasks to determine the preferred spiral space pattern, the preferred laminar motion, and the preferred pursuit direction. After the neuron was characterized, we then ran the translation speed compensation task.
In the first characterization task, we displayed eight spiral space patterns to measure the spiral space selectivity of each neuron. Spiral space includes expansions, contractions, rotations, and their combinations. These visual patterns are generated on the retina during self-motion toward and away from a frontoparallel plane, as well as during self-rotation about the axis of heading. Activity in spiral space can be represented in two dimensions with expansions/contractions represented along the horizontal axis and rotations along the vertical.
Spiral space is constructed by rotating the motion vectors in an expansion stimulus by different counterclockwise angles: 0° for expansion, 45° for a counterclockwise-expanding spiral, 90° for counterclockwise rotation, 135° for a counterclockwise-contracting spiral, 180° for contraction, 225° for a clockwise-contracting spiral, 270° for clockwise rotation, and 315° for a clockwise-expanding spiral. The preferred spiral space pattern of the neuron can be seen by plotting the tuning curves in spiral space. The tuning curves for MSTd neurons in response to spiral space stimuli are typically single-peaked and smoothly varying (Graziano et al., 1994; Geesaman and Andersen, 1996).
We also determined the laminar motion tuning of each neuron, because laminar motion tuning is an identifying characteristic of area MSTd. Laminar motions are generated on the retina during lateral self-motion. They are also generated during eye and head rotations. We used the laminar motion task to estimate the response of MSTd neurons to purely visual laminar motion in the absence of either self-motion or eye and head movements. The laminar motion stimuli consisted of random dots moving unidirectionally in one of eight directions, spaced 45° apart.
Other than differences in the visual stimuli, both the preferred spiral space pattern and preferred laminar motion tasks used the same behavioral task. At the beginning of each trial, a fixation point appeared on the screen and the monkey was required to fixate within 800 ms; otherwise, the trial would abort. Fixation was required to be maintained within a ±2° window around the fixation point until the end of the trial. Once the monkey obtained fixation, the optic flow pattern was displayed for 1200 ms resulting in a total trial duration of 2000 ms. The last 1000 ms of this period was used to measure the activity of the neuron. By discarding the first 200 ms of the period in which the optic flow pattern was displayed, we avoided the phasic response to the onset of the stimuli and only analyzed the tonic response. Overall, we ran three trials per condition to determine the mean rate of activity. Three trials per condition have proven sufficient to measure the response of the neuron, yet the number of trials was low enough that all the conditions could be tested in a single recording session (Shenoy et al., 2002).
In the final characterization task, we determined the preferred pursuit direction of each MSTd neuron. Spiral space and laminar motion stimuli involve purely retinal cues, whereas pursuit eye movements also involve extraretinal signals related to the eye movement itself. These extraretinal signals may reflect the eye movement command or efference copy, from other brain areas. We used the pursuit task to quantify the extent to which extraretinal signals are represented in area MSTd. In this task, the monkey pursued in eight directions spaced 45° apart to determine the preferred pursuit direction of the neuron. This preferred direction was noted and used in the translation compensation task.
The preferred pursuit direction was determined by having the monkey pursue a small white dot on a black background with no other visual stimuli on the screen. Because MSTd neural firing rates increase with increasing pursuit speeds, we used a rather fast pursuit speed of 8°/s to elicit a large neural response (Shenoy et al., 1999, 2002). We tested eight pursuit directions spaced 45° apart. Within 800 ms of the stimulus appearance, the monkey was required to obtain and maintain fixation inside a ±2° moving window that surrounded the moving fixation point. The fixation point continued to move for an additional 1200 ms for a total trial duration of 2000 ms. To remove gaze angle effects, pursuit trajectories were centered around the straight ahead (0,0°) location on the monitor.
Translation speed compensation task.
In this block of trials, we examined the effects of different speeds of translation and pursuit on compensation in three randomly interleaved conditions: fixed gaze, real pursuit, and simulated pursuit. In all three conditions, the monkey had 800 ms to acquire fixation, and then had to maintain fixation for an additional 1200 ms for a successful trial. The heading stimuli consisted of an expansion flow field with the foci positioned at 11 locations in 6° steps (range, ±30°) along the preferred axis of pursuit as determined earlier. Because the stimulus window was 20 × 20° and always located at the same position on the screen, the FOE would sometimes be outside the window, but the centrifugal dots from the expansion pattern were always visible within this window (Fig. 2).
Translation speed task diagram. The monkey was required to fixate at 0,0°. The heading stimuli consisted of an expansion flow field with the foci positioned at 11 locations in 6° steps (range, ±30°) along the preferred axis of pursuit. The activity at the 11 FOE locations was used to generate a tuning curve. The stimulus window was 20 × 20° in size and contained 400 dots. Because the stimulus window was always located at the same position on the screen, the FOE would sometimes be outside the window, but the centrifugal dots from the expansion pattern were always visible within this window (shown in figure). In the fixed-gaze condition, the monkey simply maintained fixation on a static fixation point. In the real-pursuit condition, the monkey pursued a moving fixation point across the stimulus window. In the simulated-pursuit condition, the monkey fixated at a static fixation point, and the entire stimulus was drifted across the screen at the same speed, but in the opposite direction, as in the real-pursuit condition.
In the fixed-gaze condition, the stimulus was presented on the monitor as the monkey fixated on a stationary fixation point. Because there was no eye movement in this condition, the visual image and retinal image were identical and represented the actual direction of heading. In the real-pursuit condition, the stimulus was again presented at a fixed location on the monitor; however, the monkey was required to pursue a moving fixation point across the stimulus window. The moving fixation point traveled in the preferred direction of the neuron as determined previously. This eye movement caused the focus of expansion to shift on the retina. As a result, the focus of expansion no longer represented the direction of heading. In the simulated-pursuit condition, the retinal image was identical to the retinal image in the real-pursuit condition. This was accomplished in the simulated-pursuit condition by drifting the entire stimulus across the screen at the same speed, but in the opposite direction as in the real-pursuit condition to ensure that all aspects of the retinal stimulus were identical in the two conditions. Of the three conditions, only the retinal image in the fixed-gaze condition accurately represented the direction of heading. The real- and simulated-pursuit conditions both had shifted focus of expansions that no longer represented the direction of heading. The difference between the real- and simulated-pursuit conditions was that the eyes actually moved in the real-pursuit condition, but not in the simulated-pursuit condition.
For both the real-pursuit and simulated-pursuit conditions, we used three different (real and simulated) pursuit speeds (2.5, 5.0, and 8.0°/s) and three different simulated translation speeds (10, 16, 20 cm/s). Because there was no pursuit in the fixed-gaze condition, we only varied the three translation speeds. This gave us a total of 21 conditions (three fixed gaze plus nine real pursuit plus nine simulated pursuit). The different pursuit speeds at different translation speeds produced focus shifts ranging from 4.8 to 30.5° (Table 1). At the different pursuit speeds, the eye would travel for at least 1200 ms (800 ms acquire time plus 1200 ms fixation time) covering a minimum distance of 3, 6, and 9.6° for the pursuit speeds of 2.5, 5.0, and 8.0°/s, respectively.
Focus of expansion shifts
Data analysis.
We calculated the preferred spiral space pattern and preferred laminar motion directions by plotting tuning curves in spiral and laminar space and computing the response weighted vector sum (Geesaman and Andersen, 1996; Shenoy et al., 1999, 2002). The preferred pursuit direction was also determined using this technique. A significant modulation of activity by direction was measured by using the Rayleigh test (Geesaman and Andersen, 1996; Zar, 1996).
The governing equation of the retinal image when approaching a frontoparallel wall is the following:
where x is the distance of a point on the wall from the center, dx/dt is the radial speed, Tz is the translation speed, and z is the distance from the wall.
To determine the retinal location of the focus of expansion during smooth pursuit eye movements, we set dx/dt as the pursuit speed, Px, and solve Equation 1 for x as follows:
This equation shows how the location of focus x changes as a function of the pursuit and translation speed. The value of z in Equation 2 is the distance of the monkey's eyes to the monitor and is fixed at 38.1 cm. The retinal location of the focus of expansion depends linearly on pursuit speed (Px) and hyperbolically on translation speed (1/Tz). We chose pursuit speeds between 0 and 10°/s because of constraints of the monitor size. Also, faster pursuit is more difficult for the monkeys to perform. Similarly, translation speed was constrained between 10 and 20 cm/s, so that we did not encounter translation/pursuit speed combinations in which pursuit speed had little effect on focus shift (Tz > 20 cm/s) or a large effect on focus shift (Tz < 10 cm/s).
Compensation is calculated by measuring the horizontal shift of the tuning curves obtained in the real- and simulated-pursuit conditions with respect to the tuning curve measured in the fixed-gaze condition. We calculated the cross-correlation coefficient at each 6° step of the 11 FOE locations along the pursuit axis (Bradley et al., 1996; Shenoy et al., 1999, 2002). Cross-correlation is well suited for this type of well sampled data because of its sensitivity to the horizontal alignment of tuning curves regardless of their exact functional form. In addition, it is insensitive to any vertical shifts or gain changes between the curves that may be present. One potential drawback of using cross-correlation is that the range of shifts tested must be restricted to avoid computing cross-correlations where there are not enough overlapping points. We avoided this problem by designing the analysis such that, of the 11 focus positions, there were always 6 or more overlapping sample points.
To interpolate between the 6° measures, the tuning curves were spline interpolated with 1° sampling. In previous studies, we found that the results are similar without such smoothing (Shenoy et al., 1999, 2002). However, using spline interpolation allows us to detect small shifts. The shift that produced the highest correlation coefficient was used to compute the compensation. Compensation was defined as the difference between the theoretical shift of the focus on the retina and the measured shift using cross-correlation, divided by the theoretical shift. Percentage compensation was calculated by multiplying this term by 100. For example, if the theoretical retinal shift during a pursuit condition was 12°, and the actual shift was only 3°, then the compensation is (12° − 3°)/12° = 0.75, and the percentage compensation is 75%.
To examine how each neuron changed its compensation as a function of real and simulated translation speed, we calculated a compensation index. To do this, we first calculated a regression of the compensatory shift as we varied real and simulated translation speed. We performed the regression by calculating the compensatory shift for each translation speed, and then fitting a linear model with the 1/(translation speed) as the independent variable and the compensatory shift as the dependent variable. We then divided the slope of each regression line by the slope of the line expected for perfect compensation. This gave us the compensation index as a percentage for real and simulated translation speed.
A linear model was chosen to fit the data because, mathematically, the focus of expansion, and thus the amount of compensatory shift needed, is linearly dependent on pursuit speed and 1/(translation speed) as shown by Equation 2. We included the 0° compensation at the translation speed = 0 data point in the regression, because when translation speed becomes large enough, the value of 1/(translation speed) must be zero. Intuitively, we can understand this effect by visualizing the movement of dots in the visual stimuli as we increase translation speed. With high translation speed, the dots of the visual stimuli move toward you much more than they move across the screen. Therefore, increasing translation speed compresses the horizontal shift attributable to pursuit movements.
To examine the effects of pursuit speed and translation speed on compensation in area MSTd, we performed a two-way ANOVA on the 3 × 3 array of compensations. A significant result for pursuit speed or translation speed would indicate that the compensation was not uniform but depended on one of the factors. We then explored whether the two factors were combined independently or whether there was an interaction between them by examining the interaction term from the ANOVA.
To further quantify the effects of translation and pursuit speed, we performed a regression analysis on the measured shifts of the tuning curves. Because we know the association between the retinal shifts, pursuit speed, and translation speed (Eq. 2), we can estimate how well the compensation follows this relationship. We first examined simplified versions of Equation 2 by regressing single variable models using pursuit or translation speed alone. The goodness of fit of the regressions indicated how well these single variable models represented the data. We then fit the complete model of Equation 2 and examined the significance of fit. A significant improvement in fit was assessed using the Akaike Information Criterion for each model. A significant improvement in the fit would suggest that the neurons are doing the appropriate calculations and the two factors are interacting as predicted by Equation 2.
Results
We analyzed neural activity from 93 neurons in two monkeys, 64 from monkey DON and 29 from monkey ROY. These neurons were selected from a larger population for demonstrating a significant response using an ANOVA analysis in all three characterization tasks of spiral space, laminar motion, and pursuit. These neurons were then tested in the translation speed compensation tasks.
Characterization tasks
Across the population, 86 of 93 (93%) neurons recorded were significantly tuned to one direction in spiral space (p < 0.05, Rayleigh's test) (Fig. 3A). Most of the neurons showed the greatest response to an optic flow pattern that contained an expansion component for 82 of 93 (88%) neurons. The preferred direction across the population of neurons was −11 ± 10° (95% confidence interval), and the distribution was not uniform (p < 0.05, Rayleigh's test). Because we initially searched for neurons using an expansion stimulus, this distribution may not be reflective of the overall representation of spiral space patterns in MSTd.
Population: spiral/laminar/pursuit/antialignment. A, Spiral space: 86 of 93 (93%) neurons recorded were significantly tuned to one direction in spiral space (p < 0.05, Rayleigh's test). The preferred response across the population of neurons was −11 ± 10° (95% confidence interval) and the distribution was not uniform (p < 0.05, Rayleigh's test). Most of the neurons showed the greatest response to an optic flow pattern that contained an expansion component: 82 of 93 (88%) neurons. B, Laminar: 80 of 93 (86%) MSTd neurons showed significant tuning to laminar motion stimuli (p < 0.05, Rayleigh's test). The distribution of preferred directions was not uniform and was slightly biased toward upward motion (p < 0.05, Rayleigh's test). There was a slight ipsilateral bias: 50 of 93 (54%) neurons. C, Pursuit: 76 of 93 (82%) MSTd neurons showed significant tuning to pursuit (p < 0.05, Rayleigh's test). The distribution of preferred pursuit directions was not uniform and was slightly biased toward downward motion (p < 0.05, Rayleigh's test). There was a slight contralateral bias: 56 of 93 (60%) neurons. D, Antialignment of laminar and pursuit tuning. On a population level, laminar and pursuit tuning showed an antialignment of 178 ± 14° (95% confidence interval), and the distribution was not uniform (p < 0.05, Rayleigh's test). Most neurons had an antialignment between 90 and 270°: 67 of 93 (72%) neurons.
A total of 80 of 93 (86%) MSTd neurons showed significant tuning to laminar motion stimuli (p < 0.05, Rayleigh's test) (Fig. 3B). The distribution of preferred directions was not uniform and was biased toward upward motion (p < 0.05, Rayleigh's test). There was also a slight ipsilateral bias: 50 of 93 (54%) neurons.
A total of 76 of 93 (82%) MSTd neurons showed significant tuning to pursuit (p < 0.05, Rayleigh's test) (Fig. 3C). The distribution of preferred pursuit directions was not uniform and was biased toward downward motion (p < 0.05, Rayleigh's test). There was also a contralateral bias: 56 of 93 (60%) neurons.
On a population level, laminar and pursuit tuning showed an antialignment of 178 ± 14° (95% confidence interval) with a nonuniform distribution (p < 0.05, Rayleigh's test) (Fig. 3D). Most neurons had an antialignment between 90 and 270°: 67 of 93 (72%) neurons. This is consistent with previous reports (Sakata et al., 1978, 1983; Komatsu and Wurtz, 1988b; Shenoy et al., 2002).
Translation/pursuit speed compensation
Computing heading direction involves compensating for eye movements by subtracting their influence from the optic flow pattern. After determining the axis of preferred pursuit, we tested how well MSTd neurons compensate for pursuit eye movements by orienting the focus of expansion stimuli along the axis of preferred pursuit. Figure 4 shows the response of an example neuron to these stimuli at three different translation speeds and three different pursuit speeds. For this neuron, the simulated headings (FOE) were situated from −30 to +30° in 6° increments along the vertical axis (90°). We will explain in detail the tuning of this neuron using the middle panel in the top row of Figure 4: TS, 10 cm/s; PS, 5.0°/s. The fixed-gaze tuning curve (blue solid) peaks around −18 to −24° and is the neural response to the focus of expansion at the 11 points spanning −30 to +30°. Because the eyes were not moving, the focus position on the retina corresponds to the true heading in this fixed-gaze condition.
Single neuron compensation: three translation speeds by three pursuit speeds. Tuning curves and calculated compensatory shifts of an example neuron at three translation speeds and three pursuit speeds. Simulated headings (FOE) were situated from −30 to +30° in 6° increments along the vertical axis (90°). Fixed-gaze tuning curves are shown by a blue solid line, real pursuit by a red dashed line, and simulated pursuit by a green dotted line. Because the eyes were not moving, the focus position on the retina corresponded to the true heading in the fixed-gaze condition. Theoretical shift is the amount the real- and simulated-pursuit condition tuning curves would need to shift for perfect (100%) compensation. Horizontal shifts were calculated using cross-correlation. For the TS = 10 cm/s, PS = 8.0°/s condition, the simulated pursuit compensation could not be calculated because the tuning curve features were too far apart and resulted in an erroneous calculation.
In the real-pursuit condition, the eye movement caused a shift of the focus of expansion on the retina in the same direction as the pursuit, in our example, 19.1°. If this neuron were simply reporting the focus position on the retina, then we would expect the focus tuning curve to move downward by 19.1° or left along the x-axis. However, if this neuron was reporting the perceived heading direction, then the focus tuning curve would not shift on our plot, although the image on the retina was shifted. In this case, the neuron would be compensating for the shift of the focus of expansion caused by eye movements by shifting its tuning curve to represent the perceived heading direction and not the actual image on the retina. A third possibility is the neuron had an incomplete shift somewhere between the true heading and retinal image. This would occur if the neuron were computing perceived heading direction but did not shift its focus tuning curve far enough to represent the true heading direction.
In our example neuron, for the real-pursuit condition (red dashed), the focus tuning curve is close to the focus tuning curve during fixation (fixed gaze; solid blue), indicating that this neuron is reporting the approximate true heading instead of the retinal image. To quantify how much shift actually occurred, we cross-correlated the fixed-gaze and real-pursuit tuning curves. The maximum correlation coefficient is found at an offset of 3°. If this neuron were simply reporting the location of the retinal image, there would be a shift of 19.1°. A value of 3° tells us that this neuron was almost fully compensating for the effect of pursuit eye movements and was shifting its tuning curves to represent true heading. Using our compensation index formula, we calculate that this neuron compensates by 16.1°/19.1° = 0.843 or 84.3%.
We next asked whether this neuron shifted its tuning curves in the simulated-pursuit condition when there were only retinal cues present and no actual eye movements. Again, using cross-correlation, we find that the simulated pursuit tuning curve (green dotted) is 16.0° away from the fixed gaze focus tuning curve. It is very close to the tuning curve representing the retinal image, not the direction of heading. This neuron shows 3.1°/19.1° = 0.16 or 16% compensation. In the simulated-pursuit condition, this neuron shows significantly less compensation than in the real-pursuit condition indicating that the actual pursuit eye movement is important in heading computation. However, the compensation value for the simulated-pursuit condition is not zero, which demonstrates that MSTd neurons can use purely retinal cues to shift the focus tuning curve toward the fixed gaze tuning curve (true heading).
Because we know that MSTd neurons can partially compensate for simulated pursuit, which is based on retinal information and not actual eye movement or efference copy, we asked whether MSTd neurons can adjust their tuning curves to changes in translation speed that are also only based on retinal input. In the middle column of Figure 4, the pursuit speed was kept the same (5.0°/s); however, the translation speed changed from 10 to 16 to 20 cm/s. In this manner, we were able to keep pursuit speed the same and examine the effects of varying the translation speed.
By inspection, it is apparent that the real-pursuit tuning curves are very close to the fixed-gaze tuning curves for all three translation speeds. When the actual shift is calculated for the real-pursuit condition, this MSTd neuron shifted its tuning curves to compensate for eye movements by 16.1 to 10.9 to 10.5° as translation speed increased. Because there is an inverse relationship between translation speed and the shift of the focus of expansion, increases in translation speed result in smaller focus of expansion shifts. The real pursuit tuning curves align closely to the fixed gaze tuning curves (true heading). This means that neurons in MSTd were able to use purely retinal information related to translation speed to shift their tuning curves toward true heading. The pursuit speeds in the three conditions were identical so the changes in the tuning curves are attributable to retinal information alone.
Because this effect is based on purely retinal information, we would expect the simulated-pursuit condition, which does not have any eye movement component and only has retinal information, to also compensate for changes in translation speed. We found this is the case. The tuning curves shift from 3.1 to 1.9 to 1.5° as translation speed increased. Although this is not enough for full compensation to represent true heading, it shows that the retinal information alone can be used by neurons in MSTd to partially compensate and shift their tuning curves toward the true heading direction.
Translation speed compensation: single neuron example
To examine the response of a single neuron across varying translation and pursuit speeds, we plotted the compensatory shift as a function of the inverse of translation speed in the real-pursuit and simulated-pursuit conditions. Figure 5 shows the results of translation speed compensation for a single neuron that is different from the one in Figure 4. The thick blue line represents 100% compensation, whereas 0% compensation would be represented by a perfectly horizontal line. The dashed red line is real pursuit and the dotted green line is simulated pursuit. The compensatory shift increased as 1/(translation speed) increased for both the real- and simulated-pursuit conditions.
Single neuron: translation speed compensation. A–C, Compensatory shift is plotted as a function of the inverse of translation speed in the real-pursuit and simulated-pursuit conditions. This neuron is different from the one in Figure 4. The thick blue line represents 100% compensation. Compensation of 0% would be represented by a perfectly horizontal line. The dashed red line is real pursuit, and the dotted green line is simulated pursuit. The thin black lines are regressions through the real- and simulated-pursuit conditions. We calculated the compensation index from the regression lines by dividing each regression line by the slope of the perfect compensation line. Although some neurons increased their compensation in direct proportion to 1/(translation speed), most neurons did not display a linear dependence. We averaged the regression lines across all three pursuit speeds to generate a single compensation index for both real and simulated pursuit. On the whole, this neuron compensated 69% for real pursuit and 42% for simulated pursuit.
To calculate the compensation index, we first regressed lines (thin black lines) through the real and simulated translation data. A linear model was chosen based on theoretical consideration (see Materials and Methods). Although individual neurons did not display a linear dependence, the population data nicely fit with the linear model as we will show later. Because of this, we applied a linear regression to the data from each neuron. We then divided each regression line by the slope of the perfect compensation line. This allowed us to quantify how much each neuron changed its compensation as a function of 1/(translation speed). For this neuron, at a pursuit speed of 2.5°/s, real pursuit had a compensation index of 57%, and simulated pursuit resulted in a 35% compensation index.
For both the real- and simulated-pursuit conditions, the slopes were positive, indicating that compensatory shift increased as 1/(translation speed) increased. The slope for the real-pursuit condition was also greater than the slope for the simulated-pursuit condition, which means that real pursuit results in greater compensation compared with simulated pursuit.
Regression lines for different pursuit speeds had similar compensation indices, as predicted by Equation 2. We averaged the compensation indices across all three pursuit speeds to generate a single compensation index for both real and simulated pursuit. On the whole, this neuron compensated 69% for real pursuit and 42% for simulated pursuit.
ANOVA
To analyze the effects of pursuit speed and translation speed, we performed a two-way ANOVA on the 3 × 3 array of compensatory shifts. A significant result for one factor or another would suggest that the compensation is affected by pursuit speed and/or translation speed. The interaction term from the ANOVA indicates whether the two factors are combined independently or whether there is an interaction between them.
We found that translation speed is significant in 76 of 93 (82%) of neurons during real pursuit and 64 of 93 (69%) of neurons during simulated pursuit (Table 2, ANOVA). The similarity between the number of neurons that are significant for translation speed in the real- and simulated-pursuit condition is expected because translation speed is a retinal cue that is identical in both conditions. This result demonstrates that translation speed is used by area MSTd to compute heading.
ANOVA/regression
Pursuit speed is significant in 86 of 93 (93%) of neurons during real pursuit and 58 of 93 (62%) of neurons during simulated pursuit. The difference in the number of neurons that are significant to pursuit speed may be because during real pursuit, there is an actual eye movement, whereas in simulated pursuit, the eye is stationary and the stimulus is dragged across the visual field to simulate eye pursuit. It appears that the actual eye movement in the real-pursuit condition recruits a larger number of neurons when calculating the translation and pursuit speed.
There is a significant interaction effect in 36 of 93 (39%) of neurons during real pursuit and 34 of 93 (37%) of neurons during simulated pursuit. A significant interaction effect means that pursuit speed and translation speed are interacting in a nonlinear manner.
Regression
To understand the relationship between the effect of translation speed and the effect of pursuit speed, we performed a regression analysis on the measured compensatory shifts of the tuning curves (Table 2, Regression). Equation 2 predicts that tuning curve shifts depend on both pursuit speed and translation speed. We tested this prediction by calculating how closely the compensatory shifts followed this relationship. We first regressed single variable models (pursuit or translation speed alone) using simplified versions of Equation 2. We then fit the full model given by Equation 2 and assessed the significance of the fit. A significant improvement in the fit would suggest that the two factors interact in the way predicted by Equation 2, and that the neurons are doing the appropriate calculations to estimate heading.
We first regressed the measured shift of the focus of expansion tuning curves to the single variable model of FOE = 1/(translation speed) × 38.1. We found that only 5 (5%) neurons during real pursuit and 13 (14%) of neurons during simulated pursuit had a significant fit to this model. Similarly, when we regressed the single variable model of FOE = pursuit speed × 38.1, we found that only 13 (14%) of neurons during real pursuit and 20 (22%) of neurons during simulated pursuit demonstrated a significant fit. This regression analysis shows that the computation does not functionally follow FOE = 1/(translation speed) × 38.1 only or FOE = pursuit speed × 38.1 only.
However, when we regress the compensation shifts to the full model of FOE = PS/TS × 38.1, we find 54 of 93 (58%) and 50 of 93 (54%) of neurons demonstrate a significant fit for real pursuit and simulated pursuit, respectively. This indicates that many MSTd neurons are doing calculations in accordance with Equation 2.
Translation speed compensation: population
In Figure 6 we calculated and plotted the mean population compensation shift during real and simulated pursuit as a function of 1/(translation speed). We regressed lines through the mean compensatory shifts for both real and simulated pursuit. Error bars are 95% confidence intervals. On a population level, there was greater compensation as 1/(translation speed) increased. There was also greater compensation for real pursuit compared with simulated pursuit. For the population, the average compensation index across all pursuit speeds was 70% for real pursuit and 36% for simulated pursuit.
Population: translation speed compensation. A–C, Plots of the mean population compensation shift during real and simulated pursuit as a function of 1/(translation speed). Error bars are 95% confidence intervals. For the population, the average compensation index across all pursuit speeds was 70% for real pursuit and 36% for simulated pursuit. Although most individual neurons did not display a linear dependence on 1/(translation speed), across the population of MSTd neurons we found that the compensation increased with 1/(translation speed) in an approximately linear manner.
Although most individual neurons did not display a linear dependence of compensatory shift with 1/(translation speed), across the population of MSTd neurons we found that the compensation increased with 1/(translation speed) in a linear manner. This is advantageous from a computational standpoint: regardless of the translation speed, the percentage compensation would remain the same because of the linear dependence of compensation on translation speed.
Pursuit speed compensation: population
In addition to translation speed compensation, we also examined the pursuit speed compensation of our population and compared the results with previous studies from our laboratory. In Figure 7, we plot our population data with respect to increasing pursuit speed for a fixed translation speed, in essence repeating the experiments of Shenoy et al. (2002). Our results are in line with theirs. Under similar conditions, at a translation speed of 16 cm/s, we found a compensation index of 65% during real pursuit and 28% during simulated pursuit. Shenoy et al. (2002) calculated 55 and 42%, respectively. Across all the conditions, we found a compensation index of 66% during real pursuit and 33% during simulated pursuit.
Population: pursuit speed compensation. A–C, Plots of the mean population compensation shift during real and simulated pursuit as a function of pursuit speed. We included the 0° compensation shift at 0°/s pursuit speed data point in the regression. For the population, the average compensation index across all pursuit speeds was 66% for real pursuit and 33% for simulated pursuit. Across the population of MSTd neurons, we found that the compensation increased with pursuit speed in an approximately linear manner.
Compensation comparisons for FOE shifts of the same value
In the translation speed experiment, three translation speeds were tiled with three pursuit speeds. Of the nine FOE shifts, there were two sets of replicates, namely 9.53 and 19.05° (Table 3). The 9.53° focus shift arises from TS = 10/PS = 2.5 and TS = 20/PS = 5, whereas the 19.05° shift is the result of TS = 10/PS = 5 and TS = 16/PS = 8. Although these two sets of FOE shifts have the same value, they arrive from different translation speed and pursuit speed combinations. This is essentially comparing rows A and C in Figure 1. We can compare the amount of compensation within each pair to see whether the compensation is the same. If neurons in MSTd only used extraretinal signals (i.e., pursuit speed), then the compensation would be different when comparing the two variants for each FOE shift. However, if MSTd neurons use both extraretinal and retinal signals because of translation speed, then the compensation for both variants of each FOE shift should be similar.
Two sets of focus of expansion shifts
We ran a t test on the means for each pair of FOE shifts and found that the means were the same (p < 0.05, t test) (Table 4). This shows that neurons in MSTd used both retinal and extraretinal cues in computing heading direction during translation. MSTd neurons were able to shift their focus tuning curves to represent true heading in response to changes in both translation speed and pursuit speed.
Focus shifts for 9.53 and 19.05° for different PS/TS values
Discussion
We examined whether neurons in MSTd are able to shift their tuning curves toward the actual direction of heading to compensate for the shift of the FOE brought about by changes in translation speed. The neurons were first tested for their optic flow preferences as well as for preferred pursuit direction. Consistent with previous studies, we found that MSTd neurons are selective to spiral space patterns, especially for those that have an expansion component (Saito et al., 1986; Duffy and Wurtz, 1991a,b; Orban et al., 1992; Graziano et al., 1994; Sakata et al., 1994). The majority of neurons in MSTd show significant tuning to laminar motion; however, we did not find a contralateral visual motion bias that others have reported (Komatsu and Wurtz, 1988a; Shenoy et al., 2002). We found that most MSTd neurons show significant pursuit activity as reported in other studies (Kawano et al., 1984; Komatsu and Wurtz, 1988a; Erickson and Thier, 1991; Bradley et al., 1996; Shenoy et al., 2002). The distribution of preferred directions was not uniform because there was a slight bias toward the contralateral side. This contralateral preference was seen in other studies as well (Komatsu and Wurtz, 1988a).
We also found an anticorrelation between preferred laminar motion and preferred pursuit direction, which was also previously noted by Shenoy et al. (2002). This combination of extraretinal information and purely retinal information may be necessary for visual processing and heading determination. Laminar and pursuit integration might be explained by a mechanism discovered by Sakata et al. (1978, 1983) and further explored by Komatsu and Wurtz (1988b). When pursuing an object across the visual scene, the background moves in the opposite direction of the eye movement. If area MSTd is integrating laminar motion with pursuit eye movements, then one would expect a higher firing rate when pursuing across a static background versus pursuing across a background without any salient features. Komatsu and Wurtz found that MSTd neurons do enhance their firing rate when pursuing across a visible background compared with pursuit in the dark (Komatsu and Wurtz, 1988b).
We found that individual MSTd neurons are able to shift their tuning curves to compensate for changes in translation speed. This was true in both the real- and simulated-pursuit conditions. Although some neurons increased their compensation in direct proportion to 1/(translation speed), most neurons did not display a linear dependence. However, across the population of MSTd neurons, the compensation increased with 1/(translation speed) in an approximately linear manner. Shenoy et al. (2002) also found this property of MSTd when examining pursuit speed compensation. In their study, as they increased pursuit speed, individual MSTd neurons also increased their compensation, although the increase was not in direct proportion to pursuit speed and the response was often not linear. However, their entire population did show an approximately linear increase in compensatory shift as a function of pursuit speed. This population level linear dependence for compensation with varying translation speed and pursuit speed simplifies computation for an area that is downstream of MSTd receiving heading information. In our translation speed experiment, we found that amount to be ∼70% when the eyes are making a real pursuit movement and 36% for simulated pursuit regardless of translation and pursuit speed.
Although there is much psychophysical and physiological evidence that an extraretinal signal is important in pursuit compensation, several computational papers have hypothesized that, under the correct conditions, there is enough information in the pattern of retinal motion for the visual system to estimate translation and rotation components without an extraretinal signal (Longuet-Higgins and Prazdny, 1980; Rieger and Lawton, 1985; Heeger and Jepson, 1992; Perrone and Stone, 1994; Lappe et al., 1996). However, experimental human studies have contradicted this idea and shown that human subjects are rather poor at judging their direction of self-motion when shown simulated pursuit heading stimuli (Royden et al., 1992; Royden, 1994; Banks et al., 1996; Crowell et al., 1998a). In fact, studies by Crowell et al. (Crowell et al., 1998b; Crowell and Andersen, 2001) found 0% compensation during the simulated-pursuit condition.
We used similar experimental setups as the studies discussed above, and, on the contrary, we found significant compensation effects in the simulated-pursuit condition. The MSTd neurons compensate about equally for the 1/(translation speed) regression (36% in Fig. 6) and for the pursuit speed regression (33% in Fig. 7). It is possible that differences in the physiological and psychophysical results may be explained by the size of the stimuli. In our study, the stimulus occupied a 20 × 20° window, whereas in the human psychophysical studies, the stimulus occupied a 40 × 40° window. The issue of the stimulus size and its effectiveness in driving retinal-based compensation has been a subject of debate in the literature. Koenderink and van Doorn (1987) showed that simulated-pursuit compensation is possible if the scene has a large field of view and contains depth variation. A single frontoparallel plane stimulus, like the one in this present study, would not provide enough information to compute self-motion accurately. Supporting this claim for compensation requiring large stimuli are psychophysical studies that demonstrate accurate self-motion judgments during simulated pursuit using large displays (90 × 90°), whereas performance was not accurate with smaller displays (Grigo and Lappe, 1999).
However, our physiological results disagree with the idea that larger stimuli are needed for retinal-based compensation. Bradley et al. (1996) found in their neurophysiology experiments that large stimuli (50 × 50° and up) are less effective in driving retinal-based compensation in the simulated-pursuit condition, when compared with smaller stimuli (20 × 20°). Our results support small size stimuli as being more effective in generating compensation in the simulated-pursuit condition. A possible explanation lies in the fact that smaller stimuli windows have more salient borders, which could improve compensation. According to this, the large stimuli would have their borders positioned in the periphery of the visual field, therefore reducing any possible compensatory effects.
In this current study, we find that retinal as well as extraretinal signals are combined to shift FOE tuning curves to represent the heading direction in MSTd. It should be noted that the same FOE shifts may correspond to different translation speed/pursuit speed combinations. The regression analysis showed that many neurons in MSTd are shifting their focus tuning curves according to Equation 2. This is significant in that both translation speed and pursuit speed are represented in Equation 2. However, because we did not observe 100% compensation for either real or simulated pursuit, it is possible that area MSTd is doing a calculation that is similar to but not exactly like Equation 2, or the calculations are being done at a lower gain. It is also possible that heading computation is distributed among other cortical structures such as the ventral intraparietal area (VIP). Zhang et al. (2004) found strong heading direction tuning and pursuit compensation in VIP neurons during pursuit across optic flow stimuli simulating forward translation.
In our experiments, we used two-dimensional (2D) frontoparallel flow fields that simulated approach to a wall. Using these stimuli, we found only partial compensation in both the monkey neurophysiology and human psychophysical experiments. Upadhyay et al. (2000) found that adding three-dimensional (3D) cues such as motion parallax to optic flow displays produced larger responses in MSTd neurons when compared with 2D frontoparallel stimuli. With 3D cues, they found that MSTd neurons not only increased their response, but also displayed stronger heading selectivity. This finding is similar to results found in human psychophysical studies. Crowell and Andersen (2001) found that humans were able to make more accurate heading judgments during pursuit across stimuli that contained 3D cues than those with 2D cues. It is possible that MSTd neurons demonstrate complete compensation using a ground plane with 3D cues. If compensation is improved or complete, this would indicate that 3D motion parallax cues are being used by MSTd neurons.
In summary, the results presented here indicate that many neurons in MSTd shift their focus tuning curves to adjust for changes in translation speed. Previous work in our laboratory has shown that MSTd neurons can shift their tuning curves to adjust for eye movements (Bradley et al., 1996) and changes in pursuit speed (Shenoy et al., 2002). These results combined with our finding that MSTd neurons can compensate for changes in translation speed further support the idea that MSTd is an important cortical area in self-motion processing.
Footnotes
-
This work was supported by the National Eye Institute, J. G. Boswell Professorship, a Career Award in the Biomedical Sciences from the Burroughs Wellcome Fund, and a Howard Hughes Medical Institute Predoctoral Fellowship. We thank Kelsie Pejsa, Nicole Sammons, and Lea Martel for animal care and surgical assistance, Tessa Yao for administrative assistance, Viktor Shcherbatyuk for technical assistance, James A. Crowell for optic flow stimulus assistance, and Marina Brozovic and Boris Breznen for scientific discussions on this manuscript.
- Correspondence should be addressed to Dr. Richard A. Andersen, Mail Code 216-76, Division of Biology, California Institute of Technology, Pasadena, CA 91125. andersen{at}vis.caltech.edu
