Abstract
There are reciprocal connections between visual and motor areas of the cerebral cortex. Although recent studies have provided intriguing new insights, in comparison with volume of research on the visual control of movement, relatively little is known about how movement influences vision. The motor system is perfectly suited to learn about environmental forces. Does environmental force information, learned by the motor system, influence visual processing? Here, we show that learning to compensate for a force applied to the hand influenced how participants predicted target motion for interception. Ss trained in one of three constant force fields by making reaching movements while holding a robotic manipulandum. The robot applied forces in a null [null force field (NFF)], leftward [leftward force field (LFF)], or [rightward force field (RFF)] direction. Training was followed immediately with an interception task. The target accelerated from left to right and Ss's task was to stab it. When viewing time was optimal for prediction, the RFF group initiated their responses earlier and hit more targets, and the LFF group initiated their responses later and hit fewer targets, than the NFF group. In follow-up experiments, we show that motor learning is necessary, and we rule out the possibility that explicit force direction information drives how Ss altered their predictions of visual motion. Environmental force information, acquired by motor learning, influenced how the motion of nearby visual targets was predicted.
Introduction
Connectivity studies show that visual areas in the occipital and parietal lobes are reciprocally connected with motor areas of the frontal lobe (Wise et al., 1997; Van Essen, 2005). Beyond efference-copy-based updating (Von Holst and Mittelstaedt, 1950; von Helmholtz, 1962; Duhamel et al., 1992), the functional roles of pathways relaying information from the motor system to the visual system remain relatively unclear, because we know far more about how vision affects action than we do about how action affects vision.
The motor system may contribute to predictions of visual motion. Studies of biological motion perception with point-light displays have shown that observers more easily identify (Loula et al., 2005) and show greater functional activation for (Calvo-Merino et al., 2005) their own movements, and that learning a novel arm-swing pattern while blindfolded can improve later visual recognition of the accompanying gait (Casile and Giese, 2006). These results suggest that the visual system is informed about movement dynamics (the coordination of limb segments through the precise generation of muscle forces and joint torques) in a way that goes beyond efference-copy-based predictions of visual reafference (Von Holst and Mittelstaedt, 1950; von Helmholtz, 1962) and that stored information about movement dynamics can be used to predict the visual characteristics of movement.
The motor system may also contribute to predictions of visual motion by informing the visual system about environmental forces. Zago et al. (2004) reported that participants' manual interception of falling targets was better timed if the target accelerated with gravity than if the target fell with constant velocity. This result suggests that stored gravity information is used to predict target motion. Is stored gravity information visual or motor in nature? The visual system may store information about gravity: everything that falls does so with acceleration caused by gravity. But the motor system must store information about gravity: all land-dwelling creatures must plan movements that compensate for gravity. The developmental emergence of motor compensation for gravity (Von Hofsten and Spelke, 1985) occurs several months earlier than visual sensitivity to acceleration due to gravity (Kim and Spelke, 1992). At least, this order suggests that visual and motor information about gravity are acquired separately, and at most, this order supports the possibility that motor compensation for environmental forces may contribute to visual sensitivity to environmental forces.
To determine whether motor experience with forces biases predictions of visual motion in a way that is consistent with how gravity biases predictions of downward acceleration, we designed a novel experiment that avoids confounding motor and visual information about force. We examined whether horizontal force information acquired by the motor system would bias predictions of horizontal target acceleration. We trained participants to produce movements that compensate for constant, unidirectional forces that pushed the hand to the left or to the right, and then we tested whether this training influenced participants' performance in an interception task. If we detect a bias in interception timing or success that depends on force direction, this would provide evidence that environmental force information acquired by the motor system can influence how the visual motion of nearby objects is predicted.
Materials and Methods
Participants
Eighty healthy undergraduates participated in this study. There were 10 participants in each force field (FF) direction group in both experiments 1 and 2. There were 10 participants in each of experiments 3 and 4. All were naive to its purpose. All participants were right-handed as assessed by the Dutch Handedness Questionnaire (Van Strien, 1992) and had normal or corrected-to-normal vision. The study was conducted in full compliance with the regulations of the University of Western Ontario Ethics Review Board.
Apparatus
Each participant was seated at a table such that its horizontal surface was at chest height. A custom-made air sled supported the participant's right arm and allowed the arm and hand to move over the surface of the horizontal table with little friction. The participant grasped the handle of an InMotion2 robotic manipulandum (InMotion Technologies, Cambridge, MA) and used the robot to reach to visually displayed targets. The robot was programmed to apply a constant, unidirectional 10 N force to the hand. Robot position (x, y) data were sampled at 500 Hz and stored on a personal computer for later analysis.
Computer-generated displays were projected by a light-emitting diode projector (VPL-CS1; Sony, Tokyo, Japan) onto an angled mirror suspended above the workspace. The angled mirror reflected the display onto a horizontal back-projection screen (105 × 60 cm) and horizontal mirror (107 × 75 cm) that were supported above the table surface. The robotic arm was positioned below the surface of the mirror and moved parallel to the table surface. This arrangement gave participants the impression that the visual items were displayed in the same horizontal plane in which the movements were performed (Fig. 1A,C).
Experimental tasks and scoring routines. A, The stimuli and response apparatus used for the force training task. The participant grasped the robotic manipulandum and used it to move a small black cursor to targets presented on a horizontal display. The blue rectangle represents the rest position in which there were no forces applied to the hand. The green square represents one of three start positions. Force magnitude increased as a function of the cursor position between the home position and the start position. The red dots reflect the arrangement of the three potential targets around each start position. B, Maximum perpendicular displacement for one trial. C, The stimulus and response apparatus used for the interception task. The participant moved the cursor into the green square start position on the right side of the screen while resisting the force in which they had trained. The target moved from left to right and participants either made a quick, jabbing movement to intercept the target directly in front of the hand starting position (interactive interception task) or they pressed a button with their left hand when they judged the target to be directly in front of their hand. D, The IP and interception zone used to determine our measure of timing error and interception success, respectively. ITD was defined as the difference in time when the center of the target passed x-coordinate of the IP and when the hand cursor passed the y-coordinate of the IP. Finally, an interception response was deemed successful if (any part of) the target and the hand cursor were in the interception zone concurrently.
Design
The interactive and remote interception experiments had the same basic design. There was a motor learning phase, in which participants in the experimental groups learned to produce movements that compensated for a constant, unidirectional 10 N leftward force field (LFF) or a 10 N rightward force field (RFF), and participants in the control group learned a null 0 N force field (NFF). This motor learning phase was followed by an interception test phase. Importantly, the force field that participants experienced during the learning phase was also applied during the interception test. Each participant was pseudorandomly assigned to one of the three force direction groups such that there were an equal number of participants in each group.
Force training task
To allow the participant to become accustomed to the robotic manipulandum, each participant first performed 40 targeted-reaching movements in the null field. Then, each participant performed 198 targeted reaching movements while resisting the assigned constant, unidirectional force. The three start locations were arranged at body center and 20 cm to the left and to the right of body center. Three static targets were placed 8 cm away from each start position and were arranged in the 45, 90, and 135° directions (Fig. 1A). The start and target locations were arranged this way to provide the participant with the experience that the force was uniform in strength and direction over a relatively large spatial field. A black circular cursor (0.5 cm) corresponded to hand position and was updated in real time as the participant moved the robot handle through the workspace. Each of the 9 possible reaching movements was repeated 22 times in random order for a total of 198 reaching movements.
To avoid possible fatigue, the participant began each trial by placing the cursor in the home position. No force was applied to the hand while in the home position. After 500 ms, one of three start positions appeared and the participant moved the cursor from the home position into the start position (8 cm). To avoid abrupt application of the force field, force strength was programmed to increase as a linear function of the vertical distance between the home position and the start position. At the start position and at every point beyond it, 100% of the 10 N force was applied.
Once the hand was in the start position, after a short, variable delay (400, 500, or 600 ms) one of the three targets appeared. Each participant was instructed to make a fast, accurate movement to the target. To encourage consistent movement times, participants were informed that they moved too fast if movement time was <250 ms, and too slow if movement time was >350 ms. These criteria were used only to provide participants with feedback about the consistency of their movements and were not used to exclude data from the analysis. We measured maximum perpendicular distance, the maximum deviation of the hand path from the straight line directly connecting the start position with the target position (Fig. 1B). We also measured movement time, where movement initiation was defined as the time at which tangential velocity first exceeded 0.05 ms−1 and movement termination was defined as the first time after peak velocity that tangential velocity fell below 0.05 ms−1.
Experiment 1: interactive interception task
To test whether force learning influenced visual motion prediction, we asked participants to intercept accelerating targets. In the interception task, each participant used a ballistic arm movement to “stab” a cartoon fish target (40 × 20 mm) that accelerated from left to right across the display (Fig. 1C). The force field in which each participant trained was also applied during the interception task. As during training, the participant initiated each trial by moving the cursor into the home position, and force magnitude increased as the participant moved the hand from the home to the start position. The interception task start position corresponded to the right-most start position in the training task and was chosen to maximize the available distance over which the target moved (52 cm). Once the participant was in the start position, the target appeared behind a cartoon seaweed shield to the far left. After a variable wait time (400, 500, or 600 ms), the target moved across the display, always from left to right. On 80% of trials, the target accelerated at a rate of 1.0 ms−2 from one of three initial speeds (0.25, 0.50, 0.75 ms−1). This rate of acceleration and these speeds were chosen to produce target viewing times (472, 556, and 672 ms, respectively) in the range used by Zago et al. (2004). The remaining 20% of trials were constant speed (1.10 ms−1) catch trials. The speed used for catch trials was chosen to produce a viewing time as close as possible to the fastest accelerating target (473 ms). We included the catch trials as to check how consistently the force fields affected the interception of both difficult (accelerating) and easy (nonaccelerating) targets. This gave us the opportunity to assess the extent to which participants treat the force field like a new environment.
The participant was instructed to intercept the target by performing a quick, jabbing movement, perpendicular to the motion of the target, through a virtual interception point (IP) 8 cm directly in front of the start position of the right hand. The interception distance (8 cm) was equated with reaching distance in the training task. After each trial, the participant was informed if they hit the target, missed the target by arriving too early, or missed the target by arriving too late. Each participant performed 120 interception trials.
Each participant was instructed to hit as many targets as possible. He was instructed to fix his gaze on the interception point, and to avoid trying to pursue the target across the display. He was instructed to continue making straight movements, and he was told that he could not hit the target successfully by waiting at the interception point or chasing the target.
Experiment 2: remote interception
To examine whether force information effects on motion prediction depend on the nature of the interception response, we asked each participant to indicate the interception response with a remote button press response. The button press interception task was identical to the interactive interception task except that the participant did not make a ballistic movement to intercept it. Instead, the participant pressed a button at the time he judged the target was directly in front of the hand.
After training in one of three assigned force fields, each of 30 new participants grasped the robotic manipulandum with their right hand and initiated each trial by moving the cursor into the home position. The start position appeared on the right side of the display, and as the participant moved the hand from the home to the start position, force field magnitude increased. This force was at full strength at the start position. Once the participant was in the start position, the visual target presentation and motion parameters were identical with that of the interactive interception experiment. The participant held a small, custom-made button box in their left hand and pressed the button to register their interception response. Thus, participants in both the interactive and button press interception tasks resisted the force field with their right hand while responding, and so in both tasks, information about force and visual motion were available concurrently. As in the interactive interception experiment, participants were instructed to hit as many targets as possible. After each trial, participants were informed if they hit the target, missed the target by arriving too early, or missed the target by arriving too late. Each participant performed 120 interception trials.
Experiments 3 and 4: opportunity for motor learning removed
Participants in experiments 1 and 2 had an opportunity to train in the applied force fields, but they also actively resisted the force while performing the interception task. Is motor training necessary or do participants simply need to simultaneously resist a force while performing the interception task? To disentangle these two possibilities, we eliminated participants' opportunity to engage in motor learning while maintaining concurrent proprioceptive information about force and visual information about motion on each interception task trial. Ten undergraduates participated in each of these experiments. Each participant performed 40 targeted reaches in the null field so that they could become accustomed to the robotic manipulandum. They did not perform the force training phase of the experiment, but they performed a modified version of the button press interception task used in experiment 2. To prevent participants from adapting to one force direction over the course of the interception task, we manipulated force direction within subjects such that each participant experienced both the LFF and RFF (Tong et al., 2002; Osu et al., 2004; Gupta and Ashe, 2007). In experiment 3, force direction alternated every 5 trials, and in experiment 4, force direction alternated every 10 trials. Participants initiated each trial by moving the cursor into the home position in which they experienced only the NFF. After a short delay, the start position appeared to the right, and participants moved the cursor into the start position. To prevent participants from having experience moving through the force field, in experiments 3 and 4, there was no application of force while they hand moved to the start position. Instead, once the cursor was in the start position, we applied the force to the statically positioned hand by gradually increasing applied force strength from 0 to 10 N over a period of 2 s. Thus, participants did not have the opportunity to move their limb through the force field. Shortly after the force application was complete, the interception target was presented and participants indicated their interception response by pressing the button. The target motion parameters were identical with that of the interactive interception experiment. Thus, participants in experiments 3 and 4 resisted the force field with their right hand while responding, and so static proprioceptive information about force and visual information about target motion were available concurrently.
Interception-dependent measures
Interception timing and spatial error.
We measured interception timing performance by measuring the hand–target interception time difference (ITD), the signed difference in time between when the hand crossed the y-coordinate value of the interception point and when the center of the target crossed the x-coordinate value of the interception point (Fig. 1D). When a button press was registered, ITD was defined as the difference in time between when the button was pressed and the target crossed the x-coordinate value of the interception point. In both cases, ITD was negative when the hand arrived at the interception point too early and was positive when the hand arrived at the IP too late. Interception movement time was defined as the time at which movement tangential velocity first exceeded 0.05 ms−1 to the time the hand crossed the y-coordinate value of the interception point. Finally, interception spatial error was defined as the two-dimensional Cartesian coordinates of the hand when the center of the target crossed the interception point.
Interception success.
A virtual interception zone (40 × 10 mm) was used to determine whether the target was intercepted successfully. As shown in Figure 1D, a successful hit was counted if the hand cursor was in the interception zone at any time that any part of the target was in the interception zone. This means that, on some occasions, a hit was counted because the hand and the target were in the interception zone at the same time but did not physically overlap one another. A miss was counted if the hand moved through the interception zone before the target arrived in the interception zone or if the hand moved through the interception zone after the target had left the interception zone.
Statistical analyses
For the force training task, both maximum perpendicular distance and movement time were submitted to a force direction (LFF, NFF, RFF) by trial (22 bins of nine trials) mixed ANOVA. Significant interactions between force direction and trial number were analyzed further by testing for differences between force direction groups during the first nine trials of training (early) and the last nine trials of training (late). Significant main effects of force direction were further analyzed by the Bonferroni test for multiple comparisons between means.
The first five interception trials were discarded as practice trials. For the interception task, ITD for accelerating targets was submitted both to a force direction (LFF, NFF, RFF) by initial speed (0.25, 0.5, 0.75 ms−1) two-way mixed ANOVA and a force direction (LFF, NFF, RFF) by trial (1–115) mixed ANOVA. Success rates were submitted to a force direction (LFF, NFF, RFF) by initial speed (0.25, 0.5, 0.75 ms−1) two-way mixed ANOVA. We submitted each spatial error measure to a force direction (right, null, left) by initial speed (0.25, 0.50, 0.75 ms−1) mixed ANOVA. Interactions indicated by ANOVA were decomposed by conducting simple main effects analyses, and significant main effects were further analyzed by the Bonferroni test for multiple comparisons between means.
In experiments 3 and 4, ITD was submitted to a force direction (LFF, RFF) by initial speed (0.25, 0.5, 0.75 ms−1) repeated-measures ANOVA. Effects over trials were analyzed in two ways. First, we analyzed possible changes over the course of the experiment with a force direction (LFF, RFF) by trial (1–115) repeated-measures ANOVA. The second way was to analyze possible cumulative effects within blocks of 5 (experiment 3) or 10 (experiment 4) trials. This was accomplished with a force direction (LFF, RFF) by trial-number-within-block (1–5 or 1–10) repeated-measures ANOVAs.
Results
Force field training
During training, participants adapted their reaching performance to compensate for the constant, unidirectional forces applied to the hand. Figure 2A shows mean perpendicular distance as a function of trial for the first nine trials, and then as a function of trial, but binned over successive windows of nine trials, for the remainder of training. Although there are apparent differences between the three force field groups over the first few trials, constant unidirectional force fields are easily learned, and by trial 9, the three groups are not different for the remainder of training. Figure 2B shows that the influence of the force field on participants' hand paths depended on the amount of training they had received (p = 0.006). Early in training (trials 1–9), participants' hand paths were deviated by the force field applied to the hand (p < 0.001). Movements made by participants in the LFF group were deviated significantly more to the left (−4 ± 2 mm) than movements made by participants in the NFF (4 ± 2 mm; p = 0.027) and the RFF (9 ± 2 mm; p < 0.001) groups. Late in training (trials 190–198), participants' hand path deviation no longer depended on the force field applied to the hand (p = 0.993). Analyses of training movement times as a function of FF direction and trial number revealed that there was no significant difference between FF groups (LFF, 442 ± 15 ms; NFF, 413 ± 15 ms; RFF, 442 ± 15; p = 0.121), and there were no significant changes in movement time between early training and late training (early, 427 ± 15; late, 438 ± 15; p = 0.565). Thus, participants learned to plan and execute straight reaching movements with consistent movement times, indicating that they learned to compensate for the force applied to the hand.
Results of force field training. A, Maximum perpendicular displacement (PD) (in millimeters) is shown as a function of force field direction (left, null, and right) and time in training. Over the first nine trials, force direction significantly influenced the direction and magnitude of PD. By trial 10, however, PD no longer depended on force field direction. B, Bars represent mean perpendicular displacement over the first nine trials (early) and the last nine trials (late) of training. Early in training, force direction significantly influenced the direction and magnitude of PD. By the end of training, however, PD no longer depended on force field direction. In each panel, error bars represent the SEM, and asterisks depict significant differences between means.
Interactive interception of accelerating targets
Immediately after force field training, we measured participants' ability to intercept a rightward-accelerating target. We predicted that if motor force direction information contributes to how the future positions a moving target are predicted, then interception timing and success rates would be influenced by the congruence between force direction applied to the hand and target motion direction. Humans find horizontal acceleration difficult because past motion does not predict future motion in a linear manner. When faced with intercepting a horizontally accelerating target, humans predict constant velocity motion, and as a result, they initiate their interception responses too late (Gottsdanker et al., 1961; Collewijn, 1972) (but see Rosenbaum, 1975; Tresilian, 1999; Brouwer et al., 2002). Therefore, we expected that participants who learned the RFF (congruent with the direction of target motion) would respond earlier and therefore be more successful than the NFF control group, and participants who learned the LFF (opposite to the direction of target motion) would respond later and therefore be less successful than the NFF control group.
Figure 3A shows that force direction significantly influenced rates of interception success (p < 0.001). Participants exposed to the RFF, the force direction congruent with target motion, intercepted a greater percentage of targets than participants exposed to the NFF (p < 0.001), and participants exposed to the incongruent LFF intercepted a smaller percentage of targets than those who were exposed to the NFF (p = 0.026). Thus, force direction applied to the hand influenced interception success rates.
Interception success, timing error, and spatial error for the interactive interception task. A, Interception success expressed as the percentage of targets hit as a function of FF direction. Participants who trained in the left FF hit significantly fewer targets than participants in the null FF, and participants who trained in the right FF hit significantly more targets than participants in the null FF. B, ITD as a function of FF direction. Participants who trained in the right FF missed the target by less time than participants who trained in the null or left FFs. C, ITD for misses as a function of viewing time and force direction. Under short viewing time conditions (472 ms), force direction does not influence ITD. In our longest viewing time condition (672 ms), ITD for the right FF group is significantly earlier than that for the null FF group, and ITD for the left FF group is significantly later than that for the null FF group. D, Spatial error for both hits and misses. For hits, hand spatial error was small and did not differ between force direction groups in either the X or Y dimensions. On trials counted as misses, there were significant differences between the FF groups along the Y (depth) dimension but not the X (horizontal) dimension, such that the left FF group missed the target by a greater distance than the null FF group, and the right FF group missed the target by a smaller distance than the null FF group. Error bars represent the SEM, and asterisks represent significant differences between means.
Response timing was captured by an ITD measure, the signed difference in time between when the hand crossed the upper boundary of the interception zone and when the center of the target crossed the x-dimension value of the interception point. ITD was positive if the hand arrived too late. Time to contact (TTC) was also used to capture interception timing, and it is defined as the difference in time between when the interception hand movement was initiated and when the target crossed the interception point. TTC was also positive if the interception movement was initiated after the target crossed the interception point. Analyses of both ITD and TTC revealed the same pattern of results. Figure 3B shows ITD as a function of force direction. The timing of participants' responses depended on the direction of applied force (p < 0.001). The mean ITD of the RFF group was significantly smaller than that of both the NFF (p < 0.001) and the LFF group (p < 0.001). This effect did not change as a function of trial number (p = 0.198; TTC, p = 0.102).
Given that the accelerating target traveled a fixed distance with constant acceleration, we varied target viewing time by manipulating the target's initial speed. Varying the initial speed of the target significantly interacted with how force affected ITD (p = 0.030; TTC, p = 0.019). As shown in Figure 3C, when initial target speed was high (0.75 ms−1) and viewing time was short (472 ms), force direction did not affect ITD (p = 0.065; TTC, p = 0.396). But when initial target speed was low (0.25 ms−1) and viewing time was relatively long (672 ms), force direction had a significant impact on ITD (p < 0.001; TTC, p < 0.001) such that the RFF group made significantly smaller timing errors than the NFF group (p < 0.001; TTC, p < 0.001) and the LFF group made significantly larger timing errors than the NFF group (p = 0.024; TTC, p = 0.064). Thus, force information affected interception timing robustly when initial speed was low and viewing time was relatively long.
Finally, spatial errors made by participants in the three groups revealed an effect of force direction that is consistent with the differences in timing reported above. Figure 3D shows the mean spatial error, defined as the two-dimensional Cartesian coordinates of the hand when the center of the target crossed the interception point, for each force direction group. Analyses of spatial error along the depth dimension (on the vertical axis) show that, for hits, there was no significant effect of force direction (p = 0.070). On missed attempts, however, the mean hand position of the LFF group was significantly farther from the interception point than that of the NFF group (p = 0.045), and the mean hand position of the RFF group was significantly closer to the interception point than that of the NFF group (p = 0.008). Spatial error along the horizontal dimension (the dimension of force application) was by comparison very small, and for both hits and misses, it did not reveal the direction of force applied to the hand (p = 0.542).
To make claims about changes in visual processing, it is important to demonstrate that differences in interception timing or success cannot be attributed to force field effects on the kinematics of the interception response itself. Figure 4A shows that the mean path of the interception movement did not differ for the LFF and RFF groups. To capture the degree to which the force field affected the movement path executed by participants in the LFF and RFF groups, we analyzed the hand's horizontal spatial position at the time when it crossed the upper boundary of the interception zone. Note that, unlike the data in Figure 3B, this measure is not locked to target location. This analysis showed that, although both groups' hand position was biased to the left (p < 0.001), this position did not differ between the LFF and RFF groups (p = 0.536). This overall leftward bias can be attributed to the fact that the target approached from the left. Figure 4B presents maximum perpendicular distance (in millimeters) during interception as a function of the force direction applied. Analysis of this measure revealed that there was no significant effect of force direction (p = 0.451), no significant effect of trial (p = 0.657), and no interaction between force direction and trial (p = 0.940). Figure 4C shows that there was a significant effect of force field direction on interception movement time (p < 0.001) but that there were no significant differences between the LFF and RFF directions (p = 0.36). Together, these movement path and movement time analyses show that the interception movements performed by the LFF and RFF groups were statistically indistinguishable and indicate that force information acquired during training generalized to movements performed during interactive interception.
Movement paths and movement times for interceptive movements made in experiment 1. A, Movement paths for interception movements made between the start position and the upper limit of the interception zone by both the left (black) and right (gray) FF groups. There is no significant difference in the horizontal position achieved by the two groups at the end of the movement. B, Interception movement path curvature or displacement (in millimeters) for the left (black), null (white), and right (gray) FF groups. C, Interception movement time (MT) (in milliseconds) as a function of force field direction. Like during training, MT for the null group (white) is significantly lower than that for the left (black) or right (gray) FF groups. There is no difference between the left and right FF groups. In each panel, error bars represent the SEM, and asterisks depict significant differences between means.
Interactive interception of constant-velocity catch trials
If participants treat a constant, unidirectional force applied to the hand as an environmental force, then constant-velocity motion may be affected by force direction in the same way as accelerating motion. During the interception task, we randomly interleaved constant velocity (catch) targets with accelerating targets. We equated the viewing time of catch trials (473 ms) to the viewing time of the fastest accelerating target (472 ms). Force direction influenced ITD for interception attempts on both constant-velocity and accelerating motion trials in the same direction (p < 0.001). The RFF group timed their response significantly earlier (12 ± 2 ms) than both the NFF (23 ± 2 ms; p < 0.001) and the LFF groups (23 ± 2 ms; p < 0.001). Overall, interception timing was more accurate for constant-velocity-target trials (5 ± 2 ms) than it was for accelerating-target trials (34 ± 1 ms; p < 0.001). Not surprisingly, participants intercepted constant-velocity targets more successfully (57.7 ± 2.0%) than they intercepted comparable accelerating targets (41.4 ± 1.7%; p < 0.001). Catch-trial success rates were not affected by force direction (p = 0.50).
Remote interception of accelerating targets
Does the extent to which motor force information influences visual prediction of motion depend on the nature of the interception response? To answer this question, we examined the effect of force field learning on interception when the interception response was a remote judgment indicated by a button press. We trained three new groups of participants in the RFF, LFF, and NFF, and then repeated the previously described interception task with the exception that participants indicated interception with a remote button press response. As in the interactive interception experiment, we measured the ITD by calculating the absolute difference in time between when the participant pressed the button and when the center of the target crossed the x-coordinate of the interception point, and we measured interception success rates.
As for interactive interception, participants' remote interception responses were late on average and they were significantly influenced by force direction (p < 0.001) (Fig. 5A). The mean ITD of the RFF group was significantly earlier than that of both the NFF (p = 0.001) and the LFF groups (p < 0.001). This effect did not change over the course of the interception test and, unlike interactive interception, did not depend on the initial speed of the target (p = 0.723). Figure 5B shows that success rates were not affected significantly by force direction (p = 0.080). Thus, when intercepting remotely, the effect of force direction on interception timing was significant, but the effect of force direction on interception success was not.
Timing error and success rates for the button press interception task used in experiment 2. A, ITD as a function of FF direction. Participants who trained in the right FF timed interception earlier than participants who trained in the null or left FFs. B, Interception success expressed as the percentage of targets hit as a function of FF direction. In each panel, error bars represent the SEM, and asterisks depict significant differences between means.
Is static force information sufficient?
These results are consistent with the hypothesis that environmental force information acquired during motor learning can influence how the motion of nearby targets is predicted. Under the “motor learning hypothesis,” the motor system must learn to compensate for an environmental force (i.e., establish a predictive model of force direction effects on limb dynamics) before effects on visual prediction will be observed. An equally compelling alternative, however, is that motor learning is not required but that concurrent presentation of static proprioceptive force information to the hand and visual motion information to the eye is sufficient to drive effects of force direction on response timing and interception success rates. Under the “multisensory integration hypothesis,” motor learning is not necessary because when motor and visual information about the environment are presented concurrently, this information is integrated in a direct manner.
We tried to disentangle these two hypotheses in our third and fourth experiments by eliminating participants' opportunity to engage in motor learning while maintaining concurrent proprioceptive information about force and visual information about motion on each interception task trial. We took three steps to create these conditions. First, we eliminated the training phase of the experiment. Second, to prevent participants from adapting to a single force direction over the course of the remote interception task, each participant experienced both the LFF and RFF, and we alternated the force field applied to the hand every five trials (Brashers-Krug et al., 1996; Karniel and Mussa-Ivaldi, 2002; Tong et al., 2002; Osu et al., 2004; Gupta and Ashe, 2007). Finally, to prevent participants from acquiring experience moving through the force field, we applied the force to the stable hand at the beginning of each trial by gradually increasing applied force strength over a period of 2 s. Otherwise, the details of target motion during the interception task were identical with those of the previously described interception tasks. Our dependent measures were interception time difference and interception success rates.
The results for the third experiment revealed that force direction had no influence on either interception timing of missed trials (mean LFF, 21 ± 1 ms; mean RFF, 20 ± 1 ms; p = 0.612) or interception success rates (mean LFF, 27.8 ± 1.5%; mean RFF, 27.4 ± 1.5%; p = 0.872). To examine the possible cumulative effects of force exposure, we ran a fourth experiment that was identical with the third with the exception that force direction alternated every 10 trials. The results of the fourth experiment were not different from those of the third. In both experiments, this result did not change as a function of trial number (experiment 3, p = 0.647; experiment 4, p = 0.542), and there was no evidence that force effects accumulated over repeated trials within a single force field (experiment 3, p = 0.664; experiment 4, p = 0.304). Finally, at the end of each of these two experiments, we asked participants to report all of the directions they felt the robot pushing their hand by marking directions on a circle. The results of this self-report measure (Fig. 6) show that participants could discriminate and remember force direction adequately, yet this explicit force information was not sufficient to influence their interception performance.
The histogram (dotted line) and mean (solid line) responses of participants' explicit postexperiment estimates of the directions in which the robot pushed their hand. The histogram was bimodal with modes centering on the 0° (right) and 180° (left) positions. The means of all responses on the left and right sides of the vertical meridian are depicted by the solid black and gray lines, respectively.
Discussion
Here, we show that learning to compensate for the effect of a constant horizontal force on movement, applied to the hand using a robotic manipulandum, influenced the timing and success rates of participants' attempts to intercept a nearby accelerating target. We believe these results show that force information acquired, stored, and used by the motor system to produce movements that compensate for that environmental force can also contribute to how the motion of nearby objects is predicted. Although our findings show that force information influences how visual motion is predicted, they do not speak to whether force information influences how visual motion is overtly perceived.
The results differ in several ways from other studies demonstrating influences of the motor system on the visual system. First, our study is unlike demonstrated effects of efference copy, because whereas efference copy is used to predict the visual consequences of one's own movement performance (von Holst and Mittelstaedt, 1950; von Helmholtz, 1962; Duhamel et al., 1992), here we show that motor force information influences predictions of the motion of an object that shares the environment of the body but whose motion is independent of the body. Second, although others have demonstrated effects of action on perception, the attribution of these effects to information provided by the motor system is confounded by information provided by other sensory modalities, like touch (Blake et al., 2004; Kunde and Kiesel, 2006), or by explanations that can be framed in terms of high-level, explicit cognitive links between response selection and perception (Musseler and Hommel, 1997; Wohlschlager, 2000; Hommel et al., 2001), or by conceptual information like effort or one's own performance evaluation (Proffitt et al., 2003; Witt and Proffitt, 2006).
Here, we demonstrate that the effect of force information on motion prediction depends on participants' opportunity to engage in motor learning (i.e., to establish a predictive model of how the motor system must compensate for the force). Our third and fourth experiments rule out explanations framed in terms of integration of concurrently available kinesthetic and haptic information about force with visual motion (i.e., the multisensory integration hypothesis). The results are also inconsistent with a mechanism that relies on the congruence and/or incongruence of visual and motor event codes of stimulus direction (Musseler and Hommel, 1997; Wohlschlager, 2000; Hommel et al., 2001), because despite the presence of both congruent and incongruent pairings of visual motion and force direction in our third and fourth experiments, force direction did not influence interception timing. Moreover, the results of our third and fourth experiments rule out a mechanism that relies on explicit information about force direction. Participants were able to report the directions in which the hand was pushed, yet this explicit force direction information did not influence interception timing.
We considered two possible explanations and the results were consistent with the motor learning hypothesis. The motor learning hypothesis states that, once force information has been learned by the motor system, it can influence predictions of visual motion. Motor learning research suggests that the motor system learns to deal with novel forces applied to the hand by modeling how the force will perturb limb movements. In this way, the motor system models the effect of environmental forces on the limbs, and uses this information to predict and compensate for perturbations (Shadmehr and Mussa-Ivaldi, 1994; Shadmehr and Holcomb, 1997; Flanagan and Beltzner, 2000; Haruno et al., 2001; Flanagan et al., 2003). Here, we provide evidence that the predictive model of the environment acquired during motor training influences visual processing, in particular, predictions of visual motion. There is little debate that interception demands prediction of visual motion, although the parameter that best captures prediction is the focus of some debate (Lee et al., 1983; Port et al., 1997; Brouwer et al., 2002; Zago et al., 2004). The notion that motor information influences visual prediction is supported by our observation that the size of force effects on interception timing depended on target viewing time. In their study of interception of horizontally accelerating targets, Port et al. (1997) found that participants used a reactive interception strategy when viewing times were short, but that when viewing times were long, they were more likely to use a predictive strategy. Here, we show that force direction influenced interception timing most effectively under our longest viewing time condition, that is, in the condition that best supported the use of a predictive strategy. Our central claim is that this visual prediction can be biased by relevant information acquired by the motor system.
We believe that the parietal cortex may play a critical role in how motor force information influences predictions of target motion. Neurophysiological recordings of monkey motor cortical and parietal cell firing patterns during interception show that both the parietal cortex [BA7a (Brodmann's area 7a)] and the primary motor cortex respond to real moving visual stimuli as they approach the interception point, regardless of the monkey's intention to make a response (Lee et al., 2001; Port et al., 2001; Merchant et al., 2004). Moreover, there are detectable effects of hand force on the firing rates of cells in parietal cortex and detectable effects of stimulus position on the firing rates of cells in motor cortex (Lee et al., 2001; Port et al., 2001; Merchant et al., 2004). Given that information used to compensate for the effects of novel forces on upper limb movements is represented in the motor cortex at least temporarily (Shadmehr and Holcomb, 1997; Muellbacher et al., 2002; Chouinard et al., 2005; Cothros et al., 2006), the opportunity exists for novel force information to influence target motion processing in the parietal and motor cortices. One possibility is that the parietal cortex receives visual motion information from middle temporal cortex (MT), an area known to process target motion and direction of motion (Zeki, 1974), and then parietal and motor cortices interact both to predict the motion of the stimulus and determine the appropriate time to trigger an interception response.
Our finding that force effects on interception timing and success rates were more robust for interactive interception than for remote interception is consistent with this proposed link to frontoparietal networks because these networks are more likely to be engaged by visuomotor interactions than by remote visual judgments (Milner and Goodale, 1995). These findings also suggest the possible involvement of frontotemporal networks (Seltzer and Pandya, 1989; Milner and Goodale, 1995). The connections of the motor system to dorsal visual areas in the posterior parietal cortex, the visual areas that are engaged during on-line visuomotor control, are more robust (more direct and more numerous) than the connections of the motor system to ventral visual areas in the inferotemporal cortex (Seltzer and Pandya, 1989; Wise et al., 1997), the visual areas that are activated by tasks that involve making visual judgments (Milner and Goodale, 1995).
The results of our study highlight the relative facility with which human participants make use of new horizontal force information during interception. This flexibility contrasts with the results of adaptation studies with downward visual motion stimuli and in environments that are inconsistent with gravity. Even after repeated exposure to downward-going constant velocity targets, adaptation remains incomplete; humans continue to anticipate downward acceleration due to gravity, both in laboratories on earth (Zago and Lacquaniti, 2005; Zago et al., 2005) and in space (McIntyre et al., 2001), where vestibular, cutaneous, and proprioceptive signals about gravity are absent. However, the disparity between our study and studies of gravity effects on motion processing is explainable. Gravity is ever present, one of the few constants in our existence. Other environmental forces, however, act on us in ways that require adaptation and compensation, but in ways that can be irregular. For example, the wind is a horizontal force that we must deal with that changes direction and magnitude.
Our results suggest that, once the motor system adapts to a new environmental force, this force information is available to other perceptual systems. In the experiments presented here, however, participants (in experiments 1 and 2) experienced a single force context during both training and interception. In future studies, we will address questions about whether force information continues to influence vision even after force information is removed and whether force information continues to influence vision in other situations that test the extent of generalization. It is possible that the system adopts a flexible “forget and learn” strategy (Fusi et al., 2007) for these irregular encounters with environmental forces.
Overall, we believe that motor system influences on the visual system are adaptive. In nature, nearby objects, even those that can generate their own motion, are influenced by environmental forces in the same way that we would be influenced. These experiments demonstrate that when we are exposed to new environmental forces, like the wind or a river current, our motor system acts like a sensory system. It stores information about forces in the world, uses that information to adapt motor behavior, and shares this information with other sensory systems, like vision, so that we can adaptively predict what is happening nearby.
Footnotes
-
This work was supported by a Canadian Institutes for Health Research (CIHR) Fellowship (L.E.B.), National Sciences and Engineering Research Council (NSERC) Summer Studentship (E.T.W.), Canada Research Chairs program (M.A.G.), NSERC (M.A.G., P.L.G.), CIHR (M.A.G., P.L.G.), CIHR New Investigator Award (P.L.G.), and the National Institutes of Health (P.L.G.). We thank D. Pulham for equipment construction. Portions of this research were presented at the 2006 meeting of the Vision Sciences Society (Sarasota, FL), and the 2005 meeting of the Society for Neuroscience (Washington, DC).
- Correspondence should be addressed to Dr. Liana E. Brown, Department of Psychology, Trent University, Peterborough, Ontario, Canada K9L 1Z8. lianabrown{at}trentu.ca
