Abstract
Most models of spatial vision and visuomotor control reconstruct visual space by adding a vector representing the site of retinal stimulation to another vector representing gaze angle. However, this scheme fails to account for the curvatures in retinal projection produced by rotatory displacements in eye orientation. In particular, our simulations demonstrate that even simple vertical eye rotation changes the curvature of horizontal retinal projections with respect to eye-fixed retinal landmarks. We confirmed the existence of such curvatures by measuring target direction in eye coordinates in which the retinotopic representation of horizontally displaced targets curved obliquely as a function of vertical eye orientation. We then asked subjects to point (open loop) toward briefly flashed targets at various points along these lines of curvature. The vector-addition model predicted errors in pointing trajectory as a function of eye orientation. In contrast, with only minor exceptions, actual subjects showed no such errors, showing a complete neural compensation for the eye position-dependent geometry of retinal curvatures. Rather than bolstering the traditional model with additional corrective mechanisms for these nonlinear effects, we suggest that the complete geometry of retinal projection can be decoded through a single multiplicative comparison with three-dimensional eye orientation. Moreover, because the visuomotor transformation for pointing involves specific parietal and frontal cortical processes, our experiment implicates specific regions of cortex in such nonlinear transformations.
As we navigate through the visual world, our viewpoint is constantly changing, as are the spatial relationships between our eyes, head, and body (Howard, 1982). Thus, the brain must account for these different internal frames of reference to chart visual space (Hallet and Lightstone, 1976; Zee et al., 1976;Mays and Sparks, 1980; Zipser and Anderson, 1988; Flanders et al., 1992; Maunsell, 1995; Miller, 1996). Studies that have addressed this problem have generally considered the angle of gaze direction relative to the head, adding this to retinotopic representations in a vectorial manner to reconstruct visual space in head coordinates (Hallet and Lightstone, 1976; Zee et al., 1976; Mays and Sparks, 1980; Zipser and Anderson, 1988; Flanders et al., 1992; Miller, 1996; Bockisch and Miller, 1999). This model is generally assumed to hold for secondary (vertical and horizontal) eye positions, perhaps supplemented by additional mechanisms that might compensate for tilts of the retina that occur during pure torsional eye rotations (rotation about the line of site) (Howard, 1982; Mittelstaed, 1983; Wade and Curthoys, 1997) and the so-called “false torsion” that occurs at tertiary (oblique) eye positions (von Helmholtz, 1867; Haustein and Mittelstaedt, 1990).
One problem with this implicitly common view is that it fails to account for the complex three-dimensional (3-D) properties of retinal geometry (Liu and Schor, 1998) and its dependence on eye rotation (Crawford and Guitton, 1997). Contrary to intuitions drawn from translational geometry, rotatory displacements in eye orientation have a strong and complex influence on the final static pattern of retinal stimulation. For example, we have recently confirmed that the retinal projections at tertiary eye positions in Listing's plane must be compared with eye orientation (presumably in the brainstem) to generate accurate saccades that obey Listing's law (Klier and Crawford, 1998). However, a similar geometric analysis (Crawford and Guitton, 1997) can be used to demonstrate an even more fundamental principle that is independent of Listing's law; the retinal projections of earth-fixed horizontal lines should curve with respect to a horizontal arc fixed on the retina as a function of vertical eye position (see Theory). Such an effect would be so basic as to impact almost any aspect of spatial vision and visuomotor control.
To our knowledge, curvature of retinal space at secondary eye positions has never been the subject of experimental study. Moreover, the implications of 3-D rotational eye kinematics for arm control have rarely even been considered. To judge by previous experiments (Wolpert et al., 1994; Henriques et al., 1998), correct prehensile compensation for visual distortion is hardly a foregone conclusion. Therefore, the current study had two goals: first, to quantify the degree of curvature in retinotopic projections following vertical eye rotations, and second, to determine how this is accounted for in the visuomotor transformation for pointing. The results demonstrate a specific nonlinear dependence of retinotopic projection on eye position that has been ignored by visual neuroscience but not by the visuomotor transformations of the brain itself.
THEORY
To understand the geometry of retinal projection under vertical eye orientation, we simulated this geometry with the use of math described previously (Crawford and Guitton, 1997) and VRML, a virtual reality modeling language. Figure1A shows several simulated views of stimuli and eye orientations similar to those used in this study. The precise orientation and curvature of these particular lines were chosen for gaze angle invariance and motor invariance of the task described in Materials and Methods. Other patterns will be considered in Discussion. The main point to be taken here is that, even if these lines remain fixed in space, their curvature in retinal coordinates depends on eye position, as follows.
In terms of objective space (Fig. 1A), every point along each stimulus line is located at an equal angle of elevation from the center of the eye. Let us define the horizontal meridian of the eye to be the great circle described by the intersection of a horizontal plane through the eye at primary position (Fig. 1B,horizontal orange line). Thus, when the eye looks straight ahead, the whole central stimulus line, including the current fixation point (○) and a 90° rightward target (■) project onto the horizontal retinal meridian as expected, and the projections of the other lines are (non-great circles) parallel to this, similar to lines of latitude. However, when the eye looks up (or down), the retinal projections of the stimuli curve vertically with respect to the eye-fixed horizontal meridian (Fig. 1C,D). From a space-fixed perspective (C), the retinal projections of the stimulus lines remain horizontal, but the horizontal meridian of the eye (orange) looks curved. Conversely, from the perspective of the eye (D) the horizontal retinal meridian is once again horizontal, but the retinal projections of the stimulus lines look curved. As a result, the retinal projection of a 90° rightward target (■) falls on a point of the retinal projection line (highlighted) that is left and down relative to the central foveal region, signifying that the target is rightward and upward in retinal coordinates.
As a result, the sum of the vertical gaze angle and the retinal target vector would give a misestimate of actual target elevation in space, escalating in a nonlinear manner for points located progressively more peripherally along the line. Similar effects occurred for vertical lines and horizontal eye positions. In other words, even at pure secondary eye positions, target direction could only be computed through vector addition when the target and eye rotation are contained in the same one dimension. Left unaccounted for by some neural mechanism, this would result in mislocalizations of point targets (as well as eye position-dependent misperceptions of objective curvilinear shapes, as shown in Fig. 1).
The latter could be tested in a variety of perceptual and visuomotor systems. We chose pointing because (e.g., in contrast to eye movements) it normally correlates well with perceptual measures of vision (Gauthier et al., 1990; Wolpert et al., 1994). Moreover, the hand–arm system is clearly not organized in eye coordinates during straight-arm pointing but rather a body-fixed “Fick-like” coordinate system (Hore et al., 1992; Miller et al., 1992). Thus, for targets like those shown in Figure 1, an eccentric pointing movement from the central target (○) to the peripheral target (■) on a given horizontal line should show near-motor invariance at each level, with the arm essentially rotating about a body-fixed vertical axis. Therefore, there can be no confusion here about the coordinate frames of the visual input and motor output; a definite internal series of reference frame transformations from eye coordinates to body coordinates is required (Soechting and Flanders, 1989; Flanders et al., 1992; McIntyre et al., 1997; Henriques et al., 1998; Vetter et al., 1999). Furthermore, specific physiological processes in posterior parietal cortex and premotor–motor cortex have now been implicated in such eye-to-head-to-body transformations (Georgopoulos et al., 1982;Mushiake et al., 1997; Snyder et al., 1997). This is of some theoretical importance because, whereas previous models of noncommutative–rotational transformations have been applied to brainstem processes (Tweed and Vilis, 1987; Crawford and Guitton, 1997; Tweed et al., 1999), they can now potentially be applied to higher cortical functions.
MATERIALS AND METHODS
Experiments were performed under informed consent as approved by the York University Human Participants Review Subcommittee. Subjects were six right-handed humans (ages 23–45), with no previous knowledge of the experimental design and no known neuromuscular deficits. Each subject was seated in complete darkness with the head mechanically stabilized at the center of three 2-m-diameter Helmholtz coils. Orientations of the right eye and arm were measured using a 3-D search-coil technique (Hore et al., 1992; Henriques et al., 1998; Klier and Crawford, 1998). In brief, signals from a Skalar (Delft, The Netherlands) 3-D eye coil and a similar homemade 3-D coil secured to the upper arm were sampled at 50 Hz and converted off-line into eye–arm quaternions and pointing directions (Tweed et al., 1990).
The experimental procedure was designed to demonstrate the effect simulated in Figure 1 in the simplest possible way and to isolate its implications for arm movement. For the sake of simplicity, we only used pointing targets displaced horizontally from vertical gaze fixation points. To ensure that subjects could only reconstruct this stimulus pattern with the use of the visuomotor transformation under study, (1) the left eye was patched and no monocular depth cues were available during the experiment, and (2) subjects were not allowed uncontrolled exposure to this stimulus pattern. In other words, one could not tell where the targets were before the experiment began, and we did not provide subjects with visual feedback during the arm movement until the calibration procedures at the end of the experiment. (After experiments, some subjects reported anecdotally that targets did not indeed appear to be horizontally displaced but rather followed the retinocentric “fanning out” pattern illustrated in Fig.1D). Finally, the full 3-D geometry of eye–arm coordination is highly complex, i.e., involving specific linkages between the centers of rotation of the eye, head, and arm segments (Flanders et al., 1992; Sabes and Jordan, 1997), but most of this geometry is not directly relevant to the question at hand. Therefore, we isolated the current effect by directly comparing the input–output relationships of retinal curvature (in eye coordinates) against body-centric errors in final arm angle (relative to appropriate controls).
During experiments, subjects were required to look and point toward light-emitting diodes (LEDs), arranged in a pattern like that illustrated in Figure 1A. Specifically, LEDs were placed along five horizontal semicircles (positioned 30° down, 15° down, 0°, 15° up, and 30° up), forming a vertical hemicylinder of 110 cm radius, centered on the right eye. As in Figure 1, the stimulus targets were arranged in horizontal pairs along these semicircles, with fixation targets (○) along the midline of each circle and pointing targets (■) located to the right (from the subject perspective). A total of 24 such fixation–pointing target pairs were used, placed in the configurations described below and illustrated in Results.
In each pointing trial, subjects began by monocularly fixating an illuminated LED fixation target (F) located centrally along the vertical meridian, at which time a second target (T) LED was briefly flashed to the right and at the same elevation (in space coordinates). Subjects were then required to maintain eye fixation while indicating the position of the rightward target with the use of one of two pointing paradigms (Fig.2A,B). This dissociation between gaze and pointing ensured that the arm did not simply follow the direction chosen by the gaze system, but it can also produce confounding errors related to nonhomogeneities in reading out the retinotopic map. However, our recent controls for this (Henriques et al., 1998; Henriques and Crawford, 2000) suggest that such errors are an order of magnitude smaller than the potential errors to be tested in the current study and mainly relate to misestimates in the magnitude of retinal displacement rather than its direction. For the purpose of analysis, final pointing direction was defined as the last stable arm orientation before the arm reaccelerated toward the next pointing or resting position. Finally, the resting position of the arm was held constant to minimize potentially confounding errors related to variations in initial arm position (Ghilardi et al., 1995;Vindras et al., 1998).
In the “double-point” paradigm (Fig. 2A) subjects were required to first point (arm fully extended) toward Fand then, only when both lights were extinguished, point toward the rightward target. An auditory tone signaled the subject to return the arm to a constant resting position. This paradigm resulted in saccade-like preprogrammed arm trajectories proceeding fromF to T (Fig. 2C), which was useful for obtaining a graphic depiction of the results but was not ideal for quantification because these trajectories were time-consuming, tiring, and somewhat predictable. Therefore, this paradigm was only performed using the five standard horizontal F–T target pairings illustrated in Results (Fig.3A).
In contrast, in the “single-point” paradigm (Fig.2B), the subject did not point at F but rather pointed directly from the resting position toward Tafter it had flashed. With targets presented in random order, this paradigm gave rise to otherwise unpredictable arm trajectories (in contrast to the double-point paradigm). Moreover, because this paradigm required less time and was much less tiring, it was applied to a much larger array of horizontal F–T target pairings (24 pairs in total). Specifically, the spatial coordinates of these pairs were: five midline F targets (30° down, 15° down, 0°, 15° up, 30° up), each paired with one T 60° to its right, and one T 80° to its right; two F targets at 10° right (30° up and 30° down), each paired with one T50° to its right and one T 70° to its right; and five targets at 20° right (30° down, 15° down, 0°, 15° up, 30° up), each paired with one T 40° to its right and oneT 60° to its right. These pairings were selected on the basis of simulations to give a broad, even distribution of predicted errors to test against the actual results. This data provided the primary quantitative test in our experiment.
After placing the Skalar contact lens in the subject's eye, the subject performed the preceding paradigms in the following order. First, subjects did the single-point paradigm (Fig.2B), performing five repetitions for each of the 24 target pairs (see Fig. 5), in random order. Second, they did the double-point paradigm (Fig. 2A), performing a total of 10 repetitions for each of five standard target pairs (Fig.3A). These pairs were performed in counterbalanced order (top-to-bottom-to-top, etc.) so that any resulting fatigue would have no systematic influence on the results. Third, they did the controls for the double-point paradigm. These were the same as the double-point paradigm, but pointing was done with both the arm and target visible (in dim light) for full visual feedback. Fourth, they did the controls for the single-point paradigm. These were the same as the single-point paradigm, but pointing was done with both the arm and target visible for an extended period of time (3 sec), allowing time for corrective movements with full visual feedback.
Several additional controls were then performed. Reference eye and arm positions were recorded while subjects pointed toward the central target with full visual feedback and then with the arm aligned straight ahead (for quantitative analysis of arm rotation in shoulder coordinates). Then, each of the targets (T) were sequentially illuminated for 2 sec, and subjects were instructed to stare at each one in turn. Eye coil signals at the center of each of these fixation “centroids” were later selected for conversion into unit vectors pointing toward the target (Tweed et al., 1990), which were used to express target direction in space coordinates (Klier and Crawford, 1998). These data were later used to compute target directions in retinal coordinates (Fig. 3) and hence predicted arm trajectories based on a linear model (Fig.4). Coil signals were used (as opposed to objective geometric measurements) so that any small errors caused by magnetic field nonhomogeneities would be common to both the predicted and actual arm movement data. We also measured random saccades between nine targets in a ±30° horizontal–vertical grid (illuminated 1 sec each in dark), to establish the orientation of Listing's plane. This was not necessary to compute the following results but was useful for interpretation of certain second-order modulations on the main effect (Klier and Crawford, 1998). Finally, eye coil calibration procedures were followed (Henriques et al., 1998;Klier and Crawford, 1998).
RESULTS
Figure 3A illustrates the stimulus positions used for the double-point paradigm as measured in a space-fixed coordinate system centered on the right eye in which the targets formed horizontal pairs (●, ○). Figure 3B then illustrates the computed directions of the rightward targets (○) in an eye-fixed coordinate system, computed by rotating the rightward positions from Aby the inverse of 3-D eye orientation at the leftward fixation point (●) (Klier and Crawford, 1998). This procedure gives the positions of the targets as they would appear to the right retina when subjects fixated the leftward target of each pair. The main point of Figure 3 is that targets (○) that were displaced purely horizontally from fixation points (●) in space coordinates (Fig. 3A) were displaced in different oblique directions relative to the fovea in oculocentric coordinates (Fig. 3B), tilting as a function of initial eye position. This confirmed the prediction of our simulation (Fig. 1) in which target images fell on lines of projection that curved with respect to the horizontal retinal meridian as a function of eye orientation (Fig. 1). How then would subjects perform when required to point and fixate toward each of the midline targets (●) and then maintain fixation while pointing toward the briefly flashed rightward target (○) in the complete absence of visual feedback?
Figure 4 shows the results of this test in two subjects (top row vs bottom row). A and C show the average initial pointing directions (●) to the midline fixation lights, plotted in a space-fixed coordinate system centered about the shoulder. These arm angles are not identical to the eye-centered target angles in Figure 3 because the arm and eye do not share the same center of rotation, and the arm generally points to align the finger with the line of gaze rather than pointing directly at the target (Flanders et al., 1992). Nevertheless, as confirmed by our control measures of arm kinematics, the task required these subjects to rotate the arm directly to the right. However, if the visuomotor transformation simply generated an arm displacement command in the direction of the retinal target vector (computed as in Fig. 3B) or added these vectors to the current gaze direction vector to compute target direction, it would produce the fanning out pattern of pointing responses (⊗) illustrated in Figure 4, A and C. Note that the resulting position-dependent pattern of errors (depicted by the gray wedges) would arise from a failure to compensate for the nonlinear eye position-dependent pattern of raw retinal signals such as those shown in Figure 3B.
Figure 4B shows the actual arm trajectories (♦) of a subject that showed near-ideal trajectories for this task. Like the data of most of our subjects, these arm trajectories showed little or no tendency to fan out as a function of initial vertical eye position, pointing very accurately relative to visually guided controls (○). Only one subject showed a partial tendency to follow the fanning out pattern, as seen in Figure 4D. In either case, final pointing directions showed little systematic variation across counterbalanced trials, although the variance was slightly higher for the more eccentric targets (SDs between trials, averaged across subjects, were 2.37°, 2.17°, 2.04°, 2.29°, and 2.69° topF–T pair to bottom, respectively). This suggests that the visuomotor transformation was consistent and that subjects were not unduly influenced by fatigue. Averaged across subjects, the slope of the actual vertical errors in pointing relative to controls (Fig.4B,D) as a function of predicted errors (Fig. 4A,D) across target pairs was only −0.26 ± 0.55 (SD). This negative slope signifies a slight overcompensation for eye position. In summary, the double-point test suggested that our subjects compensated for the eye position dependence of their retinal signals, sometimes partially, sometimes completely, and sometimes a bit too much. However, considering that the slopes fit to these data were based on only fiveF–T target pairings, this measure was not as reliable as that shown in the next test.
To confirm these observations quantitatively across a much larger data range, we compared the errors predicted by the vector addition model with actual errors measured relative to control values. This was done using data from the single-point paradigm (Fig. 2B), which, for reasons explained in Materials and Methods, allowed us to explore the broader range of F–T target pairings shown in Figure 5. As mentioned above, these particular target pairings were chosen so as to provide an even distribution of predicted pointing errors (based on simulations). Furthermore, because this paradigm involved unpredictable arm trajectories in various directions toward randomly presented targets, it further controlled for the possibility that subjects might have settled into a rote or cognitively guided pattern of horizontal trajectories in the double-point paradigm.
Each panel in Figure 5 shows ocular fixation directions (●) for one subject, joined (for purpose of display) to corresponding control pointing directions (■). The latter were recorded at the end of the experiment while the subject pointed toward the rightward targets with full visual feedback of both the arm and target. Note that the final pointing directions (♦) followed arm trajectories (data not shown) with variable horizontal and vertical components from the initial resting position in this single-point paradigm. Final pointing responses (♦) consistently undershot the control values (■) vertically, perhaps because of pulse-step mismatch (Fig.2B) or a consistent misperception of initial arm position (Ghilardi et al., 1995; Vindras et al., 1998). However, the main point of the figure is that there was little or no systematic eye position-dependent deviation in pointing responses across the entire pattern of F–T target pairs.
To quantify the latter observation, we computed the average vertical pointing error, relative to controls, for each of the 24 target pairings in the single-point paradigm, and plotted this as a function of the vertical component of retinal target vector (computed for eachF–T target pairing as shown in Fig. 3). The latter corresponds to the error that subjects would make if they failed to account for the effect of vertical eye position on retinal curvature when computing target location. Figure6A shows such a plot for one typical subject. Figure 6B then shows regression lines fit to similar data from all 6 subjects. A slope of one (dotted lines) would suggest that no correction for the position effect had occurred, whereas a slope of zero would suggest perfect correction. Although individuals showed a consistent vertical offset unrelated to eye position (Henriques et al., 1998) and various stochastic errors (Henriques and Crawford, 2000), they showed little or no systematic error as a function of eye position. In fact, only the subject with the largest slope (at −0.209) showed a slope significantly different from zero. The average slope across subjects (hatched lines) was −0.019 ± 0.095 (SD), suggesting that the visuomotor transformation for pointing had made the correct compensation.
DISCUSSION
It has long been known that the brain must account for the vertical and horizontal angles of gaze direction to correctly localize visual stimuli, as well as the torsional angle of eye orientation about the gaze axis (von Helmholtz, 1867; Hallet and Lightstone, 1976; Zee et al., 1976; Mays and Sparks, 1980; Howard, 1982; Mittelstaedt, 1983;Zipser and Andersen, 1988; Haustein and Mittelstaedt, 1990; Flanders et al., 1992; Miller, 1996; Wade and Curthoys, 1997; Bockisch and Miller, 1999). Moreover, we have recently shown that the brainstem saccade generator must compensate for 3-D eye orientations to generate accurate saccades from tertiary eye positions (Klier and Crawford, 1998). However, the current study is the first to experimentally demonstrate that simple rotation of the eye to an upward or downward orientation produces complex curvatures in the correspondence between horizontal lines in visual space and their retinal projections. Furthermore, we have shown that the visuomotor control system for pointing compensates for these eye position-dependent effects. This is noteworthy because, based on the known neurophysiology of frontal (Georgopoulos et al., 1982; Mushiake et al., 1997) and parietal (Snyder et al., 1997) cortex, any such transformation would have to occur at a cortical level before primary motor cortex, most likely within prefrontal (Mushiake et al., 1997) or posterior parietal cortex (Snyder et al., 1997).
Clearly, the locations of visual targets in 3-D space will influence the pattern of retinal stimulation and the predicted motor responses to those stimuli. The particular semicircular pattern of horizontal stimulus lines that we used, giving rise to retinal projections resembling lines of latitude (Fig. 1B), were specifically selected to give horizontal arm trajectories in our double-point paradigm (Fig. 4B). However, how well does this effect generalize? Figure 7shows that if our stimulus array were replaced by a series of horizontal Euclidean lines at different vertical levels on a fronto-parallel plane (Fig. 7A), it would now give rise to retinal projections resembling lines of longitude (except horizontally arranged) (Fig. 7B). With this unique arrangement, the current line of regard would stimulate the horizontal retinal meridian independent of vertical gaze angle (Fig. 7C), but this simply reverses the problem for visuomotor transformations; just as a nonhorizontal retinal code at upward gaze mapped onto horizontal arm displacements in our experiment (Fig. 4B), a horizontal displacement in retinal coordinates (Fig. 7D) would now require a nonhorizontal, oblique displacement of the arm. Moreover, there is obviously nothing special about vertical eye orientations and horizontal lines; the same geometry would hold for any large linear component of a stimulus that is orthogonal to the displacement of eye orientation from center.
Because this basic ocular geometry will affect most aspects of spatial vision and every visuomotor transformation (Klier and Crawford, 1998), each such system would require a compensatory neural mechanism, either implemented at a global level (Bockisch and Miller, 1999) or in parallel for each separate spatial–motor system (Snyder et al., 1997). Those who favor the global representation hypothesis might argue that the brain may normally use visual depth information to reconstruct the curvatures of lines in space such as those used in our study. However, depth information was not available in the current study, forcing subjects to rely on the only possible available information: monocular retinal information and any available internal representation of 3-D eye orientation. This shows that the system has this information and the capacity to use it to reconstruct angular target direction and thus probably does make use of this capability in real life. Furthermore, 3-D eye orientation varies with vergence angle (Mok et al., 1992; Van Rijn and Van den Berg, 1993), and thus accurate binocular depth information also requires some internal knowledge or assumptions of 3-D eye position (Tweed, 1997; Backus et al., 1999).
How then does the cerebral cortex perform such eye position compensations for this and the other previously known eye position dependencies? Based on previous traditions in modeling such transformations (Haustein and Mittelstaedt, 1990), one might conclude that the brain requires a (growing) array of special purpose mechanisms to account for each of these seemingly separate effects. Such corrective mechanisms might be implemented physiologically in a side loop, such as the cerebellum. However, a much more parsimonious model is possible. For example, to compute target direction in retinal coordinates, we rotated the objective target vector by the inverse of 3-D eye orientation (in contrast to the translation-based vector algebra used in most oculomotor studies). Conversely, the brain could construct a head-centric representation of angular target direction by rotating the retinal target vector by an internal representation of 3-D eye orientation. As demonstrated in our previous theoretical oculomotor investigation, this operation will work for any and all retinal target vectors and eye orientations (Crawford and Guitton, 1997), potentially even those encountered in pathological eye movements. Such rotatory processes are not additive but rather multiplicative, with different implications for the organization of realistic neural networks (Smith and Crawford, 1998; Tweed et al., 1999), including specific types of cross talk (Crawford and Guitton, 1997) between dissimilar components of retinal signals and eye position signals.
Two possible arguments against this algorithm might be raised. First, it would be very computationally demanding if performed on input from every point in visual space. However, we have argued elsewhere that such transformations need only apply to targets that have been specifically selected for action or cognition (Henriques et al., 1998;Klier and Crawford, 1998). Second, the preceding scheme would seem to suggest extensive head-centric maps of visual space that are not predominant in actual brain physiology (Georgopoulos et al., 1982;Duhamel et al., 1992; Moschovakis and Highstein, 1994; Mushiake et al., 1997; Snyder et al., 1997; Colby and Goldberg, 1999). However, we have recently confirmed that such multiplicative reference frame transformations can be entirely implicit within a neural net without ever developing an explicit representation of target position in space (Smith and Crawford, 1998). In other words, a cortical retinotopic vector can be converted into a head-centric or body-centric displacement command simply by modulating it as a function of eye or head position. These theoretical arguments are consistent with the general physiology of visuomotor cortex (Georgopoulos et al., 1982;Moschovakis and Highstein, 1994; Maunsell, 1995; Mushiake et al., 1997; Colby and Goldberg, 1999) and are particularly relevant for the recently observed gaze-centered but eye position-dependent behavior in the parietal reach region (Snyder et al., 1997; Andersen et al., 1998; Batista et al., 1999).
If perceptual and visuomotor systems do indeed use parallel, independent streams within the cerebral cortex (Milner and Goodale, 1995; Andersen et al., 1998; Bockisch and Miller, 1999), it would be a mistake to conclude that the current experiment shows that all of these systems account for the effects of eye orientation on retinal geometry. However, if initial information is stored in retinal coordinates, as we have suggested above, it is reasonable to hypothesize that visuomotor transformations downstream would use the correct motor transformation whenever feedforward accuracy is important in their behavioral task. Moreover, cognitive systems would also have to account for the changes of relative curvature in lines as a function of gaze angle to perceive allocentric visual space as constant.
Note that curvature is a relative term, always defined relative to some default notion of straightness. Considering our common sense notions of Euclidean straightness, it seems that higher level systems would probably use the projection pattern illustrated in Figure 7 as a default measure. (This is consistent with anecdotal reports from our subjects that our stimulus array did not form parallel lines). Again, having such a reference measure does not rid visual perception of the nonlinear geometric problems related to eye position. For example, consider the perception of moving objects, as in vertical ocular pursuit of an upward-translating horizontal object. Retinal geometry requires that the retinal projections of nonlinear stimuli would curve with respect to fixed retinal landmarks as it moves, as a function of current eye position (Fig. 1). For the shape and rigidity of such objects to be correctly judged (without previous knowledge), raw retinal representations would have to be rotated by current eye orientation at each point in time.
Such theoretical observations advocate the importance of the subtle eye position-dependent visual responses reported in such diverse areas as occipital cortex (Galletti and Battaglini, 1989; Trotter and Celebrini, 1999), posterior parietal cortex (Zipser and Andersen, 1988; Andersen et al., 1998; Batista et al., 1999), and frontal cortex (Boussaoud et al., 1993). However, the current study suggests that previous analyses of cortical visuomotor responses based solely on the theoretical framework of vector addition may have not yet revealed the full extent of their physiological capacity.
Footnotes
This work was supported by grants from the Canadian Natural Sciences and Engineering Research Council and Canadian Medical Research Council (MRC) to J.D.C. and T.V. D.Y.P.H. is supported by an E. A. Baker Foundation CNIB–MRC Doctoral Research Award. J.D.C. is supported by an MRC Scholarship. We thank Drs. I. Howard, L. Harris, D. Tweed, J. Cullen, K. Humphrey, and P. Medendorp for constructive criticism on this manuscript.
Correspondence should be addressed to Prof. J. D. Crawford, Department of Psychology, York University, 4700 Keele Street, Toronto, Ontario, Canada M3J 1P3. E-mail: jdc{at}yorku.ca.