 |
||||
|
||||
|
||||
|
||||
Previous Article | Next Article
The Journal of Neuroscience, September 1, 2001, 21(17):6917-6932
Eye-Hand Coordination in Object Manipulation
Roland S. Johansson1, Göran Westling1, Anders Bäckström1, and J. Randall Flanagan2 1 Department of Integrative Medical Biology, Section for Physiology, Umeå University, S-90187 Umeå, Sweden, and 2 Department of Psychology, Queen's University, Kingston, Canada K7L 3N6
ABSTRACT
TOP
ABSTRACT
INTRODUCTION
We analyzed the coordination between gaze behavior, fingertip movements, and movements of the manipulated object when subjects reached for and grasped a bar and moved it to press a target-switch. Subjects almost exclusively fixated certain landmarks critical for the control of the task. Landmarks at which contact events took place were obligatory gaze targets. These included the grasp site on the bar, the target, and the support surface where the bar was returned after target contact. Any obstacle in the direct movement path and the tip of the bar were optional landmarks. Subjects never fixated the hand or the moving bar. Gaze and hand/bar movements were linked concerning landmarks, with gaze leading. The instant that gaze exited a given landmark coincided with a kinematic event at that landmark in a manner suggesting that subjects monitored critical kinematic events for phasic verification of task progress and subgoal completion. For both the obstacle and target, subjects directed saccades and fixations to sites that were offset from the physical extension of the objects. Fixations related to an obstacle appeared to specify a location around which the extending tip of the bar should travel. We conclude that gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. The salience of gaze targets arises from the functional sensorimotor requirements of the task. We further suggest that gaze control contributes to the development and maintenance of sensorimotor correlation matrices that support predictive motor control in manipulation.
Key words: eye-hand coordination; object manipulation; grasping; obstacle avoidance; hand movement; saccadic eye movements
INTRODUCTION
Human gaze behavior has been studied in various natural dynamic activities, including driving (Land, 1992
; Land and Lee, 1994
Under the hypothesis that the brain uses gaze fixations to obtain spatial information for controlling manipulatory actions, a central issue is whether there are critical landmarks to which the gaze is drawn and how these landmarks impinge on the action. For example, when subjects direct their gaze to an object to be grasped and subsequently moved, do they fixate specific parts of the object such as the grasp site (to guide finger contact) or protruding edges (to gain shape information to be used in motion planning)? When moving a grasped object around an obstacle, does fixation of the obstacle support motion planning? A related issue concerns the probability with which various landmarks are fixated depending on their role in the task. Still, another issue is the temporal relation between extraction of spatial information by gaze fixations and manual actions that make use of this information. Does this timing vary across phases of the task and are there specific epochs during which gaze and hand events are coupled?
The present account provides novel insights into these questions. We analyzed the coordination between fingertip movements, movements of a manipulated object, and gaze behavior in a task that required grasping and lifting of object and subsequent motion planning with the object in hand. Specifically, subjects were asked to reach for and grasp a bar and then move it to contact a target, either directly or around various obstacles.
MATERIALS AND METHODS
Subjects and general procedure
Four women and five men between 22 and 52 years of age participated in the experiments after providing informed consent, and the experimental protocol was conducted according to the declaration of Helsinki. The subjects were all right-handed, did not require corrective lenses, and had no history of ophthalmological or neurological disease. Figure 1A illustrates the experimental setup. While seated behind a table, subjects used the tips of the right index finger and thumb to grasp and manipulate a bar (2 × 2 × 8 cm) located on a horizontally support surface formed by the top of a wooden stand placed on the table. The color of the bar was gray. All goal-directed bar movements took place in a frontal plane, termed the work plane, located 39 cm in front of the center of the subjects' eyes. We recorded the position of gaze, expressed as the point of intersection between the work plane and the line of sight of the right eye. An electronic shutter located between the eye and the work plane could be used to block the view of the scene at any time; the view of the left eye was always blocked. A black drape positioned 1 m behind the work plane provided a dark background (not illustrated in Fig. 1A). In addition to gaze, we recorded the three-dimensional position and orientation of the bar and the tips of the right index finger and thumb using sensors in the object and attached to the fingernails. Between trials, subjects grasped a "parking bar" between the right index finger and thumb (Fig. 1A). This bar was fixed on the tabletop 29 cm below the support surface. The subjects wore soundproof earphones with white noise to eliminate auditory cues related to changes of the experimental setting. These earphones were also used for verbal instructions.
View larger version (34K):[in this window]
[in a new window]
Figure 1. Apparatus and kinematically defined phases of the target contact task. A, An infrared-based eye-tracker was used to measure line of gaze of the right eye while the subject used the tips of the right index finger and thumb to grasp and move a bar in a vertical plane 39 cm in front of subject's eyes. Miniature electromagnetic sensors recorded the three-dimensional positions and orientations of the fingertips and the bar. For on-line monitoring of subjects' behavior and recording on video tape (SVHS), gaze position was superimposed on an image from a "scene" video camera that gave the subjects' view of the workspace via a one-way reflecting mirror. An electronic shutter was used to block the subjects' view of the scene. Between trials, subjects grasped a "parking bar" between the right index finger and thumb that was fixed on the tabletop; the grasp points were two small "bumps" on either side of the bar. B, Solid black line represents the position of the tip of the index finger or the left tip of the bar. During the (1) pre-reach phase the subject held the parking bar. The (2) reach phase began when the tip of the index finger had moved 2 cm from its parking position. The (3) grasp phase started when the tip of the index finger was <5 cm (dotted-line circle) from the forthcoming grasp site and ended when the bar began to move. In the (4) up phase the bar was moved toward the target. The (5) target phase began when the distance between the left tip of the bar and the target went below 3 cm (dotted-line circle) and lasted while the tip of the bar was within 3 cm of the center of target contact surface (dark plus sign). C, Solid black line represents the position of the centroid of the bar or the tip of the index finger. In the (6) down phase, which began when the bar exited the target zone, the bar was moved toward the table support. The (7) replace phase commenced when the vertical distance between the centroid of the bar and its final position on the support surface became <3 cm and ended when the bar was repositioned on the support surface. During the (8) reset phase, the hand was transported to the parking position. B-C, Arrowheads demarcate the end of each phase.
Apparatus
Gaze recording. An infrared video-based eye-tracking system (RK-726PCI pupil/corneal tracking system; ISCAN, Inc., Burlington, MA) was used to record the position of gaze in the work plane at 120 samples/sec. The eye-imaging camera, the infrared light source (eccentric), and the dichroic mirror were mounted on a wooden frame that was fixed to the table (Fig. 1A). To stabilize the head, subjects bit on a U-shaped stainless steel plate (Protar, KaVo; EWL, Leutkirch, Germany) anchored to the support frame of the apparatus. Both sides of the plate were coated with wax (Alminax; Associated Dental Products Ltd., Wiltshire, UK), and impressions of the dentition in the wax provided high stability of the head. To obtain such impressions, the subjects initially bit on the wax after it had been prewarmed. The rectangular area of the work plane calibrated for tracking the gaze position of the right eye was 14 cm high and 19 cm wide. The line of sight of the right eye was perpendicular to the work plane when the subjects gazed at its horizontal center, 4 cm below its upper limit, and the calibrated area extended 2 cm below the support surface. Recording of hand and object movements. We recorded at 30 samples/sec the three-dimensional positions and orientation (elevation, azimuth, and roll angles) of the bar and of the tips of the index finger and thumb using miniature electromagnetic position-angle sensors (FASTRAK; Polhemus, Colchester, VT). One sensor was fitted inside the bar, and the connecting cable came out at its rear lower edge. Each fingertip sensor (spherical, 11 mm diameter) was mounted on a small Perspex plate shaped to the profile of the fingernail. The plate was attached to the fingernail by double-sided sticky tape, and the connecting cables were taped to the digits. The connecting cables delivered with the position-angle sensors were all substituted with custom-made, light, flexible cables that were painted black. In the experimental work-space, the accuracy of the position measurement was >0.5 mm (resolution: 0.12 mm) and that of the angle measurement >1° (resolution: 0.025°). We represented the positions of the tips of the index finger and thumb by the contact sites that subjects preferred to use when grasping the bar. These sites were estimated while the subjects grasped the parking bar that had the same depth (corresponding to grasp width) and height as the manipulated bar. Symmetrically located on each side of the parking bar were two hemispherical bumps (diameter 3 mm). The subjects were asked to grasp the bar at these bumps located 13.5 cm to the right of the vertical line through the center of the work plane. We used the known locations of the bumps in the calibrated space to offset the fingertip sensors with respect to orientation and position of the preferred contact sites of the digits. Electronic shutter and fixation light. The electronic shutter (Speedglas; Hörnell International AB, Gagnef, Sweden), located ~8 cm in front of the right eye, had opening and closure times of 15 and 10 msec, respectively (Fig. 1A). A fixation light (3 mm diameter red LED) against a black background could be presented to the subjects' left eye through a mirror that was located in front of the left eye behind the plane of the shutter. The viewing distance of the LED was 39 cm. When the subjects fixated the light, the gaze position was in the upper right quadrant of the work plane, located nominally 4 cm to the right of the vertical midline of the work plane (12.2 cm to the right of the target; see below) and 9 cm above the support surface.Tasks
We report data obtained from a target contact task performed by the subjects as a part of a series of bar manipulation tasks studied in the same experimental session. In the target contact task, subjects grasped the bar by its right end and moved it such that its left end contacted a target. The target was a red 1.2 cm cube positioned 12.5 cm above the support surface on the top of a red stand at the left side in the work plane (Fig. 1B,C). The target was mounted on a spring-loaded micro-switch that distinctly yielded when the subjects had displaced its right surface by 2 mm in the left direction. After contacting the target, the subjects replaced the bar on the support surface. The subjects first performed four consecutive trials without an obstacle in the path of the bar and then four trials with each of two obstacles that had to be avoided. The red obstacle was mounted on the same post as the target, below the target. One obstacle had a rectangular shape and one was triangular (Fig. 1B) (see Fig. 4A for all three obstacle conditions). The depth of the obstacles was 4 cm, and the side facing the subjects was aligned to the work plane. The presentation of the quartet of trials with each of the two obstacles was balanced across the subjects. Each trial was initiated by an auditory cue (1 kHz beep for 200 msec) followed by the opening of the shutter allowing vision of the object. When the trial was completed the shutter closed. This was triggered by the digits arriving in the zone of the parking bar (see below). To obtain a reasonably uniform start position of gaze, the subjects gazed the fixation light during the inter-trial periods. The fixation light was turned off when the shutter was opened and the subjects performed the task. However, in one additional test series with the rectangular obstacle (four trials), the fixation light remained on during that task, and eye movements were prevented by requiring subjects to hold gaze on the light. In all test series, between trials we arbitrarily varied the distance between the left tip of the bar and the stand from 0.8 to 4.6 cm (mean 1.8 cm). The time interval between trials was 5-8 sec. Before the test series, the experimenter demonstrated each type of trial, and the subjects were instructed to do the task at their preferred speed. The subjects were asked to hold the parking bar with the tips of the index finger and thumb between trials. All instructions were fed verbally through the earphones; the experimenter toggled a switch that interrupted the noise and connected the sound recorded by an ambient microphone to the earphones.Gaze calibration procedure
We used a two-step calibration procedure to obtain gaze data with satisfactory spatial accuracy. For initial calibration, we used the point-of-regard calibration routine of ISCAN's Line-of-site Plane Intersection Software. The subject was asked to look sequentially at five 3-mm-diameter LEDs that were illuminated one by one. These were mounted on a flat surface aligned with the work plane with one LED was located in the center and one in each corner of the work plane. For the final calibration, we used calibration measurements taken repeatedly during the experiments. Before the first test series and between every third block of trials, the subjects gazed sequentially at nine points on the same surface. These included the same five used during the initial calibration and four additional points located at the midpoint of the four lines that defined the rectangular work plane. Each sampled data point obtained during the experiment was calibrated off-line using data obtained from the nearest calibration measurement before and after the point. A satisfactory gaze recording required that the eyelid did not partly cover the pupil during any phase of the tasks (except during blinks). Therefore, we fixed the subjects' eyebrow in an uplifted position by attaching a tape between the eyebrow and the forehead in a manner that did not prevent the subjects from blinking.Analysis
Data were sampled and analyzed using the SC/ZOOM system (Physiology Section, IMB, Umeå University). All signals (gaze and kinematic data) were time synchronized and stored at 200 Hz using linear interpolation between consecutive measurements. Data were sampled from 1 sec before the opening of the shutter until its closure. To analyze gaze-hand coordination in a common frame of reference, we projected data pertaining to the line of sight and the positions and orientations of the fingertips and bar to the work plane defined in the world-coordinates of the FASTRAK system. Phases of the trials defined kinematically. We divided the target contact task into eight consecutive phases (Fig. 1B,C). (1) Pre-reach phase: the period from the opening of the shutter until the hand began to move, defined as the point when the tip of the index finger had moved 2 cm from its parking position; (2) reach phase: the period from the start of hand movement (as defined above) until the moment the straight distance between the index finger and the forthcoming grasp site became <5 cm. The grasp site was defined as the position of the index finger in object coordinates at start of bar movement. This moment was instrumentally defined as the time when the bar velocity exceeded 2 cm/sec in any direction; (3) grasp phase: the period from the end of the reach phase until the start of bar movement. The time of contact between digits and the bar was defined as the moment the straight distance between the index finger and the grasp site (computed in three-dimensional space) went below 2 mm; (4) up phase: the period from start of bar movement as defined above until the straight distance between the left tip of the bar and the target was 3 cm; (5) target phase: the period during which the left tip of the bar was within 3 cm of the center of target contact surface. The yield of the switch was 2 mm. Therefore, the time of target contact was defined as the moment the horizontal position of the tip was <2 mm from the position recorded when the bar fully depressed the target switch. The switch release time was the moment the horizontal velocity of the tip of the bar first exceeded 2 cm/sec during the retraction from the target; (6) down phase: the period after the target phase when the bar was moved toward the table support. The end of this phase was the moment the vertical distance between the centroid of the bar and its final position on the support surface became <3 cm; (7) replace phase: the period between the end of the down phase and the moment the bar was repositioned on the support surface, defined as the moment bar velocity dropped below 2 cm/sec in any direction and the distance between the lowermost point of the bar and the support surface was to be <2 mm; and (8) reset phase: the transport of the hand to the parking position after the replace phase. The reset was completed when the distance between the tip of the index finger and its bump on the parking bar went below 2 cm, which triggered closure of the shutter. Gaze signals. To determine the gaze position in the work plane, for each coordinate of measurement (horizontal, vertical) we combined two signals provided by the ISCAN Line-of-site Computation and Plane Intersection Software. One signal was the initially calibrated "scene image point-of-regard position." This signal provided a low-pass-filtered representation of gaze position (lagging moving average of 10 samples at 120 Hz). In addition to low bandwidth, this signal suffers from an apparent delay in the representation of gaze position and in contrast to specifications given by the ISCAN Company, this filter could not be changed or removed. The other signal was the "pupil position" signal, which provided the corresponding nonfiltered data but did not benefit from the corneal reflection to correct for slow drifts caused by head movements, etc. To obtain a measure of gaze position (G) with adequate temporal resolution and spatial accuracy, the point-of-regard signal (R) and pupil position signal (P) were combined as follows: Gn = (PnPn
1) + Gn
Before computing G, both signals (R and P) were subjected to the following off-line calibration procedure. To calibrate each data point obtained during the target contact task, we used the nearest nine-point calibration measurement before and after the data point. Separate multiple linear regressions in the horizontal (x) and vertical (y) dimensions were applied to data obtained from both calibration measurements. The terms included in the regression were x, y, x2, y2, xy, x2y, and xy2. The resultant regression coefficients were used to scale the data point obtained in the interval between the two calibration measurements. Figure 2B shows worst-case estimates of the final error in gaze measurements for the x and y coordinates, respectively. We obtained these error distributions by computing the difference between the gaze positions recorded during all nine-point calibration measurements and the corresponding positions predicted from the calibration episode before and after (date pooled from all 9 calibration points obtained in 10 calibration measurements for each of the subjects). The SDs of the error distributions in x and y were ±0.34 and ±0.36 cm in the work plane, respectively. This corresponds to 0.50 and 0.52° angle of gaze. Measurements of gaze. We measured the locations and duration of all gaze fixations. In addition, we measured the frequency of fixations at specific landmark regions in the visual scene and the sequence of fixations of these regions as described in Results. We defined a gaze fixation as the period between the end of a saccade and the start of the subsequent saccade. Unless indicated otherwise, the position of the gaze during a fixation was defined as the mean values of all sampled x and y values during the entire epoch of fixation. We detected the occurrences of saccades based on a filter applied to the gaze position signals. First, gaze velocity in the work plane was assessed from the vectorial sum of the first time differentials of the gaze position signals in x and y using ±6 point numerical differentiation (±30 msec moving window; each sample point had the same weight). We then computed the second time differential of the gaze velocity, again using ±6 point numerical differentiation, and a saccade was scored when the amplitude of the negative peak of this differential exceeded 150 m/sec3 (22,000°/sec3). The peak gaze velocity of the detected saccades was assessed by the vectorial sum of the first time differentials of the gaze position signals in x and y using a ±1 point numerical differentiation. Likewise, the start and end of a saccade were defined by the first and second maximums of the second time derivative of the gaze velocity signal assessed by ±2 point numerical differentiation (see Fig. 2A). The straight distance in the work plane between gaze positions at the start and end of the saccade represented the gaze displacement during a saccade, i.e., the saccade amplitude. Blinks were detected from a transient reduction in the pupil size measurement, provided by the eye tracking system. Gaze shifts >1 cm during a blink were regarded as blink saccades. Blink saccades constituted only 2.6% of the total number of observed saccades. To our knowledge basic saccade parameters have not been reported previously in a natural visuomotor task involving real objects. However, several factors are known to influence the characteristics of saccades. In addition to idiosyncratic factors, the characteristics of saccades depend on the orbital direction of the eye movement (e.g., centrifugal or centripetal, temporal or nasal), the way the target position is designated (visually or by recall from memory, etc.), and the attentional state of the subject (for review see Becker, 1991
View larger version (25K):[in this window]
[in a new window]
Figure 2. Accuracy of gaze measurements and saccade and fixation parameters based on 1316 recorded saccade-fixation episodes. A, Computed gaze position (G) in the horizontal (x) and vertical (y) dimensions together with the recorded point-of-regard signals (R) and pupil position signals (P) for an epoch comprising two saccades, S1 and S2. Gaze velocity in the work plane was assessed from the computed gaze position signals. The first and second maxima of the second time derivative of the gaze velocity signal defined the start and end of a saccade. B, Estimation of the error in gaze position measurements for the horizontal (x) and vertical (y) coordinates of the work plane. C, Histogram showing distribution of saccade durations. D, Histogram showing distribution of saccade amplitudes. E, Scatter plots showing the relationship between saccade amplitude and peak saccade velocity in centimeters per second referenced to the work plane (left ordinate) and in degrees per second (right ordinate). F, The relationship between saccade amplitude and saccade duration. G, Histogram showing distribution of fixation durations. B, D-F, Bottom and top abscissas represent measurements scaled in distance on the work plane and in degrees of eye movement.
RESULTS
The results are divided into eight sections: First we introduce some general features of eye and hand movements based on a single target contact trial (1). We then analyze landmarks in the scene that attract subjects' gaze during our task (2) and assess the pattern of sequential landmark fixations within and across trials and across obstacle conditions (3). The spatiotemporal coordination of gaze and hand actions is then described (4). In the subsequent section we specifically address how the obstacle condition influenced various fixation parameters and establish that certain landmarks are obligatory and others are optional (5). We then analyze the temporal coordination between gaze shifts entering and exiting landmark zones and the specific kinematic events associated with the landmark (6). We also address the spatial accuracy of saccadic gaze shifts in manipulation (7i). Finally, we deal with shortcomings in manipulatory behavior if eye movements are prevented and subjects have to rely entirely on peripheral vision and memory (8).
Gaze-hand coordination in a single target contact trial
Figure 3 shows the pattern of eye and hand movements for a single trial involving the triangular obstacle. Figure 3A shows movements up until the bar contacts the target, and B show movements from that moment until the end of the trial. The dashed line in each panel represents the position of the tip of the index finger, and the solid line represents gaze position. The numbered circles indicate gaze fixations and their sequence. A corresponding number indicates the path of the index finger during each fixation and subsequent saccade. Consecutive fixation, saccade, and hand path units are represented in alternating colors of gray and black.
View larger version (22K):[in this window]
[in a new window]
Figure 3. Gaze and hand movements for a single trial with the triangular obstacle. A, Behavior up until the bar contacts the target. B, Behavior from target contact until the end of the trial. A, B, Dashed line represents the position of the tip of the index finger, and the solid line represents gaze position. The numbered circles indicate successive gaze fixations, and the numbers attached to the path of finger movement indicate the finger movement during the corresponding fixation-saccade period. Consecutive fixation-saccade units and hand path units are represented in alternating colors of gray and black. The hand is outside the represented area of the workspace during fixation-saccades 0, 1, and 16. The small finger movement occurring during the third fixation (A) is masked by the fixation symbol.
Several points can be gleaned from Figure 3. First, gaze and hand movements were linked with respect to key landmarks with gaze leading the hand. Gaze fixated the region of the grasp site (2 and 3) throughout the reach toward the bar (Fig. 3A). When the index finger arrived at the grasp site, gaze shifted to the tip of the bar (4), and when the bar started to move, a saccade was made to the tip of the obstacle (5, 6). Gaze then shifted to the vicinity of the target (7, 8, 9) as the bar rounded the obstacle. When the bar was still in contact with the target, gaze shifted back to the obstacle (Fig. 3B, 10, 11). As the bar and hand rounded the obstacle, the gaze shifted to the support surface (12) and remained there (13, 14) until the bar was replaced. The leftward gaze motion between 13 and 14 represents a pronounced gaze drift that occurred during fixation 14. The amplitude of this drift was at the upper extreme of the drifts that we observed. As illustrated in the Figure, each landmark could be fixated more than once. The number of fixations in this trial was close to the average number of fixations per trial in the triangular obstacle condition (see further below). Note that subjects never fixated or tracked the hand or moving bar during the task.
Fixation landmarks
Subjects thus directed gaze almost exclusively to objects involved in the task. Furthermore, gaze was directed to landmarks on these objects that were important in the task. These included the forthcoming grasp site on the bar, the left tip of the bar used to contact the target, the protruding point(s) on the obstacle, the target, and the support surface.
Figure 4A shows the distribution of all gaze fixations, from all subjects and trials, from the time gaze first left the fixation zone to the moment the bar had been released and the hand moved to the parking position. Separate plots are shown for each obstacle conditions (none, rectangular, and triangular). In addition, for each condition, we show two plots representing fixations that started before (left) and after (right) the tip of the bar entered the target zone. The black circles represent fixations within 3° (2 cm in the work plane) of one of the five landmarks, i.e., fixations within the landmark zones (see Materials and Methods). The gray circles represent fixations outside these landmark zones. The area of each black and gray circle is proportional to the duration of the fixation. The solid bars indicate the mean positions of the bar at the start (left panels) and end (right panels) of the trial; the dashed extensions represent the range of positions.
View larger version (27K):[in this window]
[in a new window]
Figure 4. Gaze fixations in relation to landmarks for all three obstacle conditions. A, The left panels show the distribution of all gaze fixations, from all subjects and trials, from the time gaze left the fixation zone to the start of the target phase. The right panels show the corresponding distribution of gaze fixations after the start of the target phase. Black and gray circles represent fixations inside and outside a 3° (2 cm in the work plane) radius of one of the five landmarks: grasp site, left tip of bar, target, support surface, and protruding point(s) of the obstacle. The area of each circle is proportional to the duration of the fixation; for calibration see the circles inset in the top right corner. In the left and right panels, the solid contours of the bar represent its mean position at the start and end of the trial, respectively, and the dashed extensions depict the range of positions attributable to the inter-trial variation of the bar position. The curve represents the path of the tip of the bar en route to the target zone (left panels) and from the target zone to the support surface (right panels); bar position data were averaged across all trials for each of the obstacle conditions. B, Distribution of gaze fixations within landmark zones normalized for bar movement and movement of grasp site; dots representing fixation locations are not scaled for duration. The landmark zones (3° radius) are represented by dotted circles, and the center of the cross (thin solid lines) indicates the mean position of the grasp site on the bar. Fixations in the landmark zones were combined for all obstacle conditions and phases of the movement, except for fixations in the obstacle zone, which were taken from the triangular obstacle condition. The area enclosed by the smaller solid-line circles within the landmark zones captured 90% of the fixations. C, Location of the grasp site ( ) and the mean gaze position of grasp site fixations (
) for each trial in bar coordinates (bottom panel). The scatter plot plots the horizontal (x) position of the grasp site against that of gaze (r = 0.76; p < 0.001; linear regression) (top panel).
Figure 4A illustrates that the majority of fixations were located close to the landmarks. The 1351 fixations shown in Figure 4A corresponded to an average of 12.5 fixations per trial. Of the total number of fixations, 1109 (82%) were within 3° of one of the landmarks. In addition, the durations of fixations within the landmark zones (median = 0.34 sec) were significantly longer than those outside the zone (median = 0.19 sec) (Mann-Whitney U test; p < 0.001). On average, subjects directed their gaze toward landmarks 90% of the total time spend looking at the scene.
The scatter of fixation points within the grasp site and tip of bar zones is relatively large because the horizontal position of the bar varied from trial to trial. The scatter of fixation points within the grasp site zone was further affected by subjects' choice of grasp site. Figure 4B shows the distribution of gaze fixations within these landmark zones normalized for variations in bar position and grasp site, with the positions of grasp site fixations referenced to the mean grasp site for all trials by all subjects. Figure 4B also shows the distribution of gaze fixations within the landmark zones of the target and the obstacle. The dotted-line circles in Figure 4B represent the landmark zones (3° radius), and the dots that represent fixation location are not scaled for duration of fixation. Gaze fixations were not evenly distributed within the 3° radius that we used to define our landmark zones. Instead they tended to be clustered, and the centers of these clusters did not align perfectly with the nominal landmarks. Moreover, the degree of clustering appeared to vary across landmarks. To quantify this clustering, we first determined the center of gaze as the mean horizontal (x) and vertical (y) position of all fixations within each landmark zone. We then computed, for each landmark, the diameter of a circle about the center of gaze that captured 90% of the fixations (Fig. 4B, solid-line circles). Diameters were obtained for the tip of the bar, the grasp site, and the target (combining data from all three obstacle conditions) as well as for the tip of the triangular obstacle for the periods before and after the tip of the bar entered the target zone. The diameters obtained for the target (3.7°) and tip of the bar (3.3°) were smaller than those obtained for the grasp site (5.0°) and the obstacle for fixations before (5.2°) and after (5.6°) the tip of the bar entered the target zone, respectively.
Inspection of Figure 4, A and B, reveals that there were offsets between the center of gaze and the nominal landmark for the target, tip of bar, and obstacle. Most notably, for the target and the obstacle, gaze was directed, on average, to a point in space displaced from the physical landmark. The centers of the solid-line circles in Figure 4B defining the gaze distributions for the target and obstacle were located 7 mm from the center of target contact surface and 3 mm from the tip of the obstacle, respectively. For fixations related to the obstacle, one intriguing idea is that they served as a virtual target through which the tip of the bar traveled en route to the target. However, this appeared not to be the case, because the closest approach of the tip of the bar was considerably farther from the obstacle (Fig. 4A). Moreover, the location of gaze fixation did not correlate across trials with the position of the tip of bar at its closest approach.
For the grasp site zone, the center of gaze was located ~0.5 cm above the grasp site, but its horizontal position was close to the grasp site. Importantly, the horizontal position of the forthcoming grasp site was directly related to gaze position on a trial-by-trial basis. Figure 4C shows, for all trials, the location of the grasp site (
Sequence of landmarks fixated
Although the sequence of landmarks fixated could vary across subjects and trials within subjects, during any given trial the sequence was clearly linked to the progress of the task. To assess the pattern of sequential landmark fixations, we first determined, for each trial, the sequence of landmarks that were fixated. On the basis of these sequences, we then determined how often the gaze went from a given landmark to each of the other landmarks and expressed this number as a proportion of all shifts between landmarks. Note that the number of landmark shifts was smaller than the total number of saccades because of multiple fixations within landmark zones and the occurrence of fixations outside these zones. Likewise, as will be shown below, not all landmarks were always fixated. Occasionally, subjects could revisit a given landmark during a trial and thus fixate it more than once.
The arrows in Figure 5 illustrate the flow of gaze fixations between landmarks for each obstacle condition based on landmark-shift data from all trials by all subjects. For each obstacle condition, the width of each arrow represents the proportion of all gaze shifts between landmarks during the task. The left panels show gaze shifts between landmarks en route to the target, and the right panels show gaze shifts between landmarks away from the target en route to the support surface. In all obstacle conditions, the grasp site was usually the first landmark fixated as illustrated by the thick arrows from the fixation zone (circle) to the grasp site. In the no-obstacle condition, there were two main paths en route to the target (Fig. 5A, left panel). After fixating the grasp site, subjects either shifted gaze directly to the target or indirectly via the tip of the bar. Gaze was then shifted to the support surface (right panel). A similar pattern was observed in the obstacle conditions, except that the obstacle was frequently fixated (after the grasp site or tip of bar) en route to the target (Fig. 5B,C, left panels) and was fixated again en route between the target and the support surface (right panels). However, with the triangular obstacle in particular, gaze could shift directly from the grasp site to the target and from the target to the support surface, avoiding the obstacle in both cases (Fig. 5C).
View larger version (33K):[in this window]
[in a new window]
Figure 5. Flow of gaze fixations between the defined landmarks for the no-obstacle (A), triangular obstacle (B), and rectangular obstacle (C) conditions. The width of each arrow represents the proportion of all shifts between landmarks, and arrows accounting for 5% or more of the shifts are filled; shifts at proportions <3% are not shown. Left panels show gaze shifts between landmarks en route to the target (circles indicate start position of gaze when the shutter opened). Right panels show gaze shifts between landmarks away from the target en route to the support surface. The widths of arrows in the box of the right panel in A provide calibration information. The number of fixation shifts between landmarks per trial was 3.8 ± 0.8 (mean ± SD) for the no-obstacle condition, and 5.0 ± 0.73 and 5.6 ± 1.2 for the triangular and rectangular obstacle conditions, respectively.
Spatiotemporal coordination of gaze and hand
Figure 6 shows the spatiotemporal coordination of gaze and hand actions based on data pooled across all subjects and trials involving the triangular object. To preserve phase information while combining data from different trials in Figure 6, we normalized the time base by scaling each phase of each trial to the median duration of that phase (Fig. 7E, striped bars). Each of the panels in Figure 6A shows the distance between gaze and one of the landmarks as a function of time. The dots represent gaze position at the start of each fixation, and the horizontal lines connected to each dot represent the duration of the fixation. The solid curve represents the distance between the median gaze position and the landmark. The dashed curves in Figure 6A refer to kinematic data displayed on the same time base as the gaze data. These curves give the median distance between the tip of the index finger (a), the tip of the bar (c, d), or the lowest point of the bar (e) and the indicated landmark. The vertical lines mark the different phases of the task, and the horizontal rectangles represent the 2 cm or 3° landmark zones. Figure 6B illustrates the time-varying probabilities of gaze fixating the different landmarks (computed in 100 msec bins). The thick solid curve in each panel shows the instantaneous probability of gaze fixation being within the 3° landmark zone. The contour of the gray area shows the probability of gaze fixation being within 2° of the landmark. The thin solid curve represents the probability of there having been a fixation within 3° of the landmark at any previous time during the trial.
View larger version (62K):[in this window]
[in a new window]
Figure 6. Spatiotemporal coordination of gaze and manipulatory actions based on data pooled across all subjects and trials with the triangular object. For all plots the common time base has been normalized such that each phase of each trial has been scaled to the median duration of that phase; the trial was initiated at time 0. The vertical lines mark the phase transitions in the task, and the phases are indicated above the top panels. A, Each graph plots the distance between gaze and a landmark (Lm) and the distance between the tip of the index finger (a), tip of the bar (c, d), or the bottom of the bar represented as the lowest point on the bar (e) and the same landmark. The gray dots represent gaze positions at the start of individual fixations, and the horizontal lines represent fixation durations. The solid curve represents the median gaze position as a function of time. The dashed curve represents the median distance between the landmark and either the index finger or the bar. The horizontal rectangles represent the 2 cm (3°) landmark zones. B, Probabilities of fixation of the different landmarks (Lm). The thick curve shows the time-varying instantaneous probability of fixations within 3° of a given landmark (computed for 100 msec bins), and the contour of the gray area shows the corresponding data for fixations within 2° (1.4 cm) of the landmark. The thin curve represents the probability that a fixation within 3° of the landmark had occurred at any previous time during the trial. C, D, Median fixation duration and cumulative median number of fixations per trial as a function of normalized time.
View larger version (36K):[in this window]
[in a new window]
Figure 7. Gaze fixation characteristics and duration of movement phases of the target contact task for each obstacle condition based on data combined from all subjects and all trials. White, gray, and striped bars refer to the no-obstacle, rectangular, and triangular obstacle conditions, respectively (see key at top). A, Bar height indicates the probability of fixation occurring within 3° of the landmark during a trial. B, Total fixation durations per trial for each landmark zone. C, Number of fixations per landmark for trials in which the landmark zone was fixated. D, Durations of individual fixations for each landmark zone. E, Duration of each phase of the trial for the three obstacle conditions. A-D, Fixations of the obstacle before and after the up phase are shown separately. Fixations indicated for the support surface occurred after the up phase. B-E, Box plots illustrate median, quartiles (25th and 75th percentiles), and 5th and 95th percentiles.
During the pre-reach phase, gaze began to shift to the grasp site, and the median gaze fixation position had reached the grasp site zone before the end of this phase (Fig. 6A,a). This occurred ~1 sec before the median position of the fingertip arrived at the grasp zone. However, by the end of the pre-reach phase, the tip of the bar and the obstacle had been fixated in ~20 and 10% of the trials, respectively (Fig. 6B, thin curves in b and c). By the middle of the reach phase, the probability of the grasp site having been fixated was close to 1 (Fig. 6B,a). The instantaneous probability of fixating the grasp site fell during the grasp phase and approached zero by the start of the up phase. During the grasp phase, the median position of fixation started to leave the grasp site while the fingertip was still approaching the grasp site (Fig. 6A,a).
From the grasp site, fixation shifted to one of three landmarks, the tip of the bar, the obstacle, or the target. However, the timing of shifts varied across landmarks. During the grasp phase, fixation shifted from the grasp site to the bar tip in approximately one-quarter of all trials (Fig. 6B,b). The instantaneous probability of fixating the tip of the bar started to increase when the instantaneous probability of fixating the grasp site began to decrease. The stepwise increase in the probability of having previously fixated the tip of the bar indicates that these trials were not those in which the tip of the bar had been fixated during the pre-reach phase. In other words, the subjects rarely returned to the tip of the bar if it was previously fixated in the trial.
Fixations started to shift from the bar to the obstacle and the target ~0.5 sec later (Fig. 6B,c). Thus, up to the end of the grasp phase, fixations were mainly directed to the bar (grasp site or tip) but were directed elsewhere once the bar began to move at the start of the up phase. After leaving the bar, gaze typically shifted first to the obstacle and then to the target, which was fixated in all trials. However, in ~20% of the trials, gaze shifted directly to the target. The early peak, indicated by an arrow, in the probability curves for the target in Figure 6B,d, reflects such gaze shifts. The subsequent increase in the instantaneous probability of fixating the target was closely mirrored by the decrease in the instantaneous probability of fixating the obstacle. The median gaze position entered the target landmark zone ~0.8 sec before the tip of the bar contacted the target, whereas the median gaze position left this zone ~0.2 sec before the tip of the bar moved away from the target (Fig. 6A,d). During the target phase, the instantaneous probability of fixating the landmark-zone representing the tip of the bar increased because the tip entered the target zone (Fig. 6A,b,B,b). However, subjects never tracked the tip of the moving bar, and we never observed gaze shifts between the target and the tip of bar during the target phase.
From the target, gaze shifted to the support surface either directly or via the obstacle. When the obstacle was fixated in either the up or down phases, the fixation tended to be brief and typically began ~0.5 sec before the tip of the bar reached it closest point to the obstacle (Fig. 6A,c).
The instantaneous probability of fixating the support surface when replacing the bar was close to 1 (Fig. 6A,e,B,e). Note that the probability of fixating the support surface zone was also quite high during the early phases of the trial before bar movement. However, this early high probability was attributable to fact that the landmark of the support surface overlapped with the landmark of the grasp site. As shown in Figure 6A,e, when the grasp site was fixated during the early phases of the trial, the median gaze position was ~1 cm above that observed when the support surface was fixated. In a similar vein, modest peaks in the instantaneous probabilities of fixating the grasp site and tip of the bar landmarks were observed when the bar was replaced. Note that in all three cases in which the probability increased because of overlapping landmarks, the probability decreased markedly when the landmark zone was reduced from 3 to 2° (Fig. 6B,a,b,e). This indicates that these landmarks were not the primary gaze targets.
As shown in Figure 6C, fixation duration varied during the course of the trials. Longer fixations were observed when gaze was directed at the grasp site, support surface, and, in particular, the target. These are the three landmarks that were contacted with either the fingertips or the bar. Figure 6D shows that during the course of the trial, a large number of fixations occurred, with a median value of 16.
Fixation parameters across landmarks and conditions
Figure 7A shows, for each obstacle condition, the probability of fixating each of the landmarks within 3° (2 cm in the work plane) during a trial. In all obstacle conditions, the grasp site, target, and support surface were fixated in almost every trial and can be considered obligatory gaze landmarks. The tip of the bar and the protruding point(s) on the obstacle were fixated with lower probability and can therefore be considered optional gaze landmarks. The rectangular obstacle was fixated with greater probability than the triangular obstacle during both the upward and downward movements (
2; p < 0.001 in both cases). The grasp site, target, and support surface were fixated for a substantially longer time than the optional landmarks (Fig. 7B) (Mann-Whitney U; p < 0.001).
We were particularly interested in whether subjects would reduce the total fixation duration at the grasp site and target in the presence of an obstacle. Planned comparisons revealed that the total fixation duration at the target was reliably greater in the no-obstacle condition than in the two obstacle conditions combined (Mann-Whitney U; p < 0.003). However, the obstacle condition did not significantly influence the total fixation duration at the grasp site (Mann-Whitney U; p = 0.50).
The number of fixations per landmark primarily influenced the variation in total fixation duration across landmarks (Fig. 7C), but the duration of the individual fixations also contributed (Fig. 7D). The number of fixations at the obligatory landmarks (grasp site, target, and support surface) was significantly greater (Mann-Whitney U; p < 0.001) than the number at the optional landmarks (tip of bar and the obstacle in both the up and down phases of the task). Likewise, the durations of individual fixations were reliably longer for the obligatory landmarks (Mann-Whitney U; p < 0.001). The effect of obstacle on the total fixation duration at the target was attributable to the number of fixations and not the duration per fixation (Fig. 7, compare D, C). In Figure 7A-D, we have combined data from all four successive trials in each obstacle condition; neither the probabilities of fixating each landmark nor the fixation duration changed across trials.
Figure 7E shows the duration of each phase for all three obstacle conditions. The overall duration of the trial was not significantly affected by obstacle condition (Kruskal-Wallis test; p = 0.33). However, there were reliable differences across conditions in the durations of the up, target, and down phases (Kruskal-Wallis; p < 0.001 in all cases). As expected, the durations of the up and down phases were shorter in the no-obstacle condition. In contrast, the duration of the target phase was longer when no obstacle was present, and this matches the greater total fixation of the target under this condition.
Time relations between gaze shifts and kinematic events
Figure 8 analyzes the timing of gaze shifts entering and exiting landmark zones referenced in time to kinematic events associated with the landmark. Gaze landmark zones were defined using the 3° distance of the landmark (see Materials and Methods).
View larger version (23K):[in this window]
[in a new window]
Figure 8. Time relation between kinematic events associated with various landmarks and gaze entering and exiting the corresponding landmark zones (Lm) represented as cumulative frequency distributions. A, Gaze entering and exiting the grasp site zone relative to index finger contacting the bar. B, Gaze entering and exiting the obstacle zone with reference to the time at which the tip of the bar passed closest to the obstacle during the up phase. C, Gaze entering and exiting the target zone referenced to when the target switch was released. D, Gaze entering and exiting the obstacle zone referenced to the time at which the tip of the bar passed closest to the obstacle during the down phase. E, Gaze entering and exiting the landmark zone of the support surface referenced to when the bar contacted the support surface. A-E, Dashed and solid line curves refer to gaze entering and exiting the landmark zone, respectively. Thin curves represent no-obstacle condition, and the thick curves represent data for the two obstacle conditions combined. Negative time values represent gaze lead before the kinematic event. Gaze landmark zones were defined as gaze fixations being within 3° of the landmark.
Considering the grasp site zone (Fig. 8A), on average, gaze entered this zone almost 2 sec before the index finger contacted the bar. This occurred about the time of reach onset (Fig. 6A). However, the exit times were distributed about the point in time of grasp contact. On average, gaze exited the grasp site zone just before contact (median = 163 msec), but left after contact in one-quarter of the trials. In virtually all trials gaze had already exited the grasp site zone by the time the bar started to move. Gaze arrived at the grasp site slightly earlier when there was no obstacle (Mann-Whitney U; p < 0.04). This apparent obstacle effect was likely caused by the greater pre-reach, reach, and grasp phase durations observed in the first test series, which was run without an obstacle (Fig. 7E). However, the presence of an obstacle did not influence the distribution of gaze exit times from the grasp site (p = 0.90), despite the fact that the obstacle was usually fixated. Thus, subjects did not sacrifice the visual control of fingertip contact to fixate the obstacle when present. Note that the distribution of gaze exit times, although far sharper than that of the entry times, was nevertheless quite variable. The SD of the time between gaze exit and grasp contact onset was 241 msec (collapsing across all obstacle conditions), and the corresponding variability for gaze entry was 624 msec.
Figure 8B shows when gaze entered and exited the obstacle zone with reference to the time at which the tip of the bar passed closest to the obstacle during the up phase. Despite the fact that the obstacle was an optional fixation landmark, the gaze entry and especially the gaze exit time distributions were relatively tight, indicating a strong coupling between gaze and bar movement. The SD for the exit time was 188 msec, whereas that of the entry time was 392 msec. On average, gaze arrived at the obstacle ~0.5 sec (median = 0.54 sec) before the tip of the bar made its closest approach and departed almost at the same time as the closest approach (median time difference = 1 msec).
Considering the target landmark zone, neither gaze entry nor exit time was well aligned to the moment the tip of the bar initially contacted the target. On average, the gaze entered the target zone ~1 sec before target contact (median = 1.13 sec) and exited the zone well after initial target contact (median = 0.71 sec). Moreover, the coupling between these gaze and contact events was rather loose; the SDs of the time difference distributions were 0.54 and 0.47 sec, respectively, and were similar for the different obstacle conditions. However, gaze was tightly coupled to the moment the target switch was released, which represents the goal completion of the target contact phase. Figure 8C shows when gaze entered and exited the target zone referenced to switch release. Note that gaze exit times were distributed evenly about the moment of switch release (median time difference =
Figure 8D shows when gaze entered and exited the obstacle zone with reference to the time at which the tip of the bar passed closest to the obstacle during the down phase. As during the up phase, the entry and exit time distributions were relatively narrow, indicating a strong coupling between gaze and bar movement. The SDs for the entry and exit distributions were 253 and 245 msec, respectively. As during the up phase, gaze arrived at the obstacle ~0.5 sec before the tip of the bar made its closest approach (median = 0.44 sec) and departed at the same time as the closest approach (median time difference =
Considering finally the landmark zone of the support surface, the gaze arrived at this zone ~1 sec (median = 1.04 sec) before the bar contacted the surface, and for all trials, gaze stayed there until after contact (Fig. 8E). Compared with the gaze exit times for the other landmark zones represented in Figure 8, the gaze exit times from the support surface zone were loosely coupled to the kinematic event (SD = 730 msec). However, at the time the bar contacted the support surface, the manipulatory task was completed and there was no forthcoming landmark that attracted gaze. As a result, the gaze either stayed in the region of the support surface until the shutter closed and the fixation light was activated, or it could shift to a point in the vicinity of the bar (see long-duration fixations in the right middle panel of Fig. 4A), and occasionally it could shift back to the target during the reset phase (Fig. 6B,d).
In sum, for all landmarks, gaze arrived in the landmark zone well before the hand or tip of the bar. The time at which gaze exited each landmark zone (with the exception of the support surface fixated at the end of the trial) was closely aligned with a contact event. The contact event could be making (grasp site) or breaking (target) of a contact or a potential contact (obstacle).
Accuracy of saccades to landmarks
Previous work on saccade generation using point light targets has shown that initial shift in gaze typically undershoots the target and is followed by one or more corrective saccades (Becker, 1991
View larger version (33K):[in this window]
[in a new window]
Figure 9. Estimates of error of saccadic gaze shifts directed to the target based on saccades from the grasp site, tip of the bar, obstacle, and target landmarks. Data from all gaze shifts to within 3 cm (4.4°) of the centroid of all fixations within the 2 cm (3°) target landmark zone that is included. A, Resultant distance between the fixation at the end of the saccade and the target fixation centroid as a function of saccade amplitude. Saccades ~10 cm originated from the grasp site or tip of the bar, saccades ~5 cm from the obstacle, and the small amplitude saccades are refixations within the target landmark zone. C, D, The x and y components of these distance as a function of saccade amplitude. Positive x-values indicate that the fixation was to the right of the centroid, and negative y-values indicate that the fixation was below the centroid (compare Fig. 4). D, Fixation duration as a function of distance from the fixation to the centroid. Note the log scale of the ordinate. A-D, Data from all three obstacle conditions combined. Bottom and top abscissas represent distance in the work plane and in degrees of eye movement, respectively. For each graph, the slope of the linear regression line was different from zero (p < 0.001 in all four instances).
We also observed that that the duration of the fixation after a saccade decreased as the distance from the fixation to the centroid increased (Fig. 9D). Fixations that were farther than ~1 cm from the centroid were generally very brief (between 50 and 200 msec), whereas closer fixations were an order of magnitude longer (note the vertical log scale in Fig. 9D). This strongly suggests that the more distant fixations were followed rapidly by corrective saccades. Indeed, it is established that the latency of the first corrective saccade decreases with the magnitude of the error that necessitates the correction but that this function approaches an asymptotic minimum (Deubel et al., 1982
That most fixations of long duration were gathered within ~1 cm from the target fixation centroid (Fig. 9D) implied that the preferred gaze location was within a diameter of ~2 cm or ~3° of vision (Fig. 4). This suggests that the extent of the "functional fovea" for the target-related fixations corresponded to ~3° angle of vision. In view of the errors in our measurements (see Materials and Methods), this angle appears consistent with previous estimates of the size of the fovea as the central 2° of vision (Rayner, 1998
It is well documented that when saccades are made to two localized targets in reasonably close proximity (e.g., <10 or 20° in eccentricity), the first saccade can go to some intermediate location. This is referred to as the global or center of gravity effect (Findlay, 1982
Performance without eye movements
We have observed that subjects chose to generate saccadic eye movements that brought important landmarks into central vision in a manner related to the phase of the task. Essentially, gaze appeared to lead the hand throughout the task. Presumably, these eye movements provided retinal and extra-retinal information that was useful to control the hand. However, people can manipulate objects without always gazing the objects involved, as when we grasp our morning coffee while reading the newspaper. This suggests that peripheral vision and/or memory can be adequate for guiding manipulatory hand movements in some contexts. Because the task that we examined was stable across repeated trials and varied only slightly across conditions, subjects should have been able to make effective use of peripheral vision and memory.
To study the importance of the saccadic eye fixations in our task, we asked our subjects to repeat the task with the rectangular obstacle while fixating the fixation light throughout the trial (see Materials and Methods). We then compared the performance during gaze locking with that observed during free gaze movements by analyzing the durations of the phases of the task and the efficiency of grasping, target attainment, and obstacle
