Abstract
Our present understanding of the neural mechanisms and sensorimotor transformations that govern the planning of arm and eye movements predominantly come from invasive parieto-frontal neural recordings in nonhuman primates. While functional MRI (fMRI) has motivated investigations on much of these same issues in humans, the highly distributed and multiplexed organization of parieto-frontal neurons necessarily constrain the types of intention-related signals that can be detected with traditional fMRI analysis techniques. Here we employed multivoxel pattern analysis (MVPA), a multivariate technique sensitive to spatially distributed fMRI patterns, to provide a more detailed understanding of how hand and eye movement plans are coded in human parieto-frontal cortex. Subjects performed an event-related delayed movement task requiring that a reach or saccade be planned and executed toward one of two spatial target positions. We show with MVPA that, even in the absence of signal amplitude differences, the fMRI spatial activity patterns preceding movement onset are predictive of upcoming reaches and saccades and their intended directions. Within certain parieto-frontal regions we show that these predictive activity patterns reflect a similar spatial target representation for the hand and eye. Within some of the same regions, we further demonstrate that these preparatory spatial signals can be discriminated from nonspatial, effector-specific signals. In contrast to the largely graded effector- and direction-related planning responses found with fMRI subtraction methods, these results reveal considerable consensus with the parieto-frontal network organization suggested from primate neurophysiology and specifically show how predictive spatial and nonspatial movement information coexists within single human parieto-frontal areas.
Introduction
How the human brain plans and effortlessly orchestrates movements of different body effectors (e.g., hands and eyes) remains a poorly understood problem in visual–motor neuroscience. An influential view, based on neural recording studies in monkeys, proposes that movement intentions are organized into effector-specific subregions (e.g., eyes vs hand), with each region predominantly involved in the visual–motor planning and control of its own effector (e.g., saccades vs reaches; Snyder et al., 1997). For instance, interconnected regions like the medial intraparietal area (MIP), V6A, and dorsal premotor cortex (PMd), are preferentially engaged when reach movements are being planned, whereas interconnected areas like the lateral intraparietal area (LIP) and frontal eye-fields (FEFs) are preferentially engaged in eye movement planning (Andersen and Buneo, 2002; Andersen and Cui, 2009), even when in both cases the spatial target of the movement remains unknown (Hoshi and Tanji, 2000; Calton et al., 2002; Dickinson et al., 2003). Furthermore, planning processes in some of these areas, while still intimately linked to the preferred effector, abstractly reflect the spatial goals/intentions of an action rather than the specific muscle activations required. For example, neural recordings from MIP show that reach targets are predominantly encoded in gaze-centered coordinates, as is the case in LIP (Andersen et al., 1985; Batista et al., 1999; although see Chang and Snyder, 2010). This use of a common spatial target representation can facilitate the first stages of the coordinate transformations required to convert sensory inputs (e.g., visual) into motor (e.g., limb) outputs and assist eye–hand coordination (Andersen and Buneo, 2002). But how do these neural findings in monkeys compare to our current understanding of how movement intentions are coded in the human brain?
The poorer spatial sensitivity inherent in noninvasive methodologies like functional magnetic resonance imaging (fMRI) presents clear difficulties in comparing brain activity between species (Kagan et al., 2010). For instance, even if there were a region with highly specialized and intermingled populations of neurons that separately encode various effectors (e.g., eye vs hand) or their movement parameters (e.g., left vs right), the coarse spatial resolution of fMRI may fail to reveal overall signal amplitude differences. Despite these inherent challenges, multiple efforts have been made to topographically describe these attributes of visual–motor integration. To date, the emerging viewpoint, in contrast to findings in the monkey, suggests that effector selectivity and spatial planning processes are poorly localized, varying only gradually across the human parieto-frontal network (for review, see Culham et al., 2006; Filimon, 2010). It is becoming increasingly clear however, that this conventional fMRI approach of comparing signal response amplitudes (and examining each voxel independently) may lack the sensitivity to detect certain types of neural information (Haxby et al., 2001; Kamitani and Tong, 2005; Kriegeskorte and Bandettini, 2007).
Here we implemented multivoxel pattern analysis (MVPA), an fMRI technique that discriminates between conditions based on the evoked spatial voxel activity patterns, to reexamine the effector-specific and spatially specific nature of intention-related signals in the human brain. Using a delayed eye and hand movement task we found —similar to that previously demonstrated in nonhuman primates—that during planning we can predict distinctly effector-specific and spatially specific movements within several parieto-frontal brain areas.
Materials and Methods
In brief, our task required subjects to perform either a reach or saccade movement toward a target in one of two different locations on each trial; subjects were first cued to the action to be carried out and then, following a delay period, performed the action (see Fig. 1B,C). On all trials both objects were presented throughout the sequence, such that the visual presentation remained constant and only the instructions differed. This delayed timing paradigm allowed us to isolate the sustained plan-related neural activity that evolves before movement (Plan phase) from the transient visual response (Preview phase) and movement execution responses (Execute phase) (Fig. 1D).
Experimental methods and example brain activity. A, Subject setup from side view. B, Top, Experimental apparatus and objects shown from the subject's point of view. The objects (white blocks) never changed position from trial-to-trial. Green star with dark shadow represents the fixation LED and its location in depth. Dashed line represents the arc of reachability with respect to the participant. The hand is positioned at its starting location. Bottom, Executed saccade and reach movements. Dashed lines represent eye position. C, Timing of one event-related trial. Trials began with the 3D objects being simultaneously illuminated while the subject maintained fixation (Preview phase; 6 s). Subjects were then instructed via headphones to perform one of four movements: touch the left object (“Touch Left”), touch the right object (“Touch Right”), saccade to the left object (“Look Left”), or saccade to the right object (“Look Right”). This cue initiated the Plan phase portion of the trial. Following a fixed delay interval (12 s), subjects were cued (“beep”) to perform the instructed hand movement (initiating the Execute phase). Two seconds after the Go cue, vision of the workspace was extinguished and participants waited for the following trial to begin (14 s, ITI). D, Averaged neural activity from left posterior IPS, pIPS, over the length of a single trial. Events in D are time-locked to correspond to events in C. MVPA was performed on single trials based on the windowed average of the percentage signal change corresponding to the three different time points denoted by each of the gray-shaded bars (each corresponding to activity elicited from the three distinct trial phases: Preview, Plan and Execute). The time points corresponding to the Plan phase (bordered in red) were of critical interest and provide the focus of our analyses.
The focus of this study was to examine whether MVPA implemented in specific regions of interest (ROIs) during movement planning (Plan phase) could predict the following: (1) upcoming reaches versus saccades; and (2) upcoming leftward versus rightward movements. In the former case, we examined the dependence of effector decoding on the spatial location of the object to be acted upon, the Effector-within-Space. This included two decoding tests: (1) Hand-Left (HandL) versus Eye-Left (EyeL); and (2) Hand-Right (HandR) versus Eye-Right (EyeR). In the latter case, we examined the dependence of spatial decoding on the effector to be used, the Space-within-Effector. This included two decoding tests: (1) HandL versus HandR; and (2) EyeL versus EyeR.
Furthermore, we also examined whether the intention-related signals in an area were governed by similar sensorimotor mechanisms. To do this, we performed cross trial-type MVPA by using one set of trials to train the pattern classifier and a different set of trials to test the classifier (Dinstein et al., 2008). For instance, to test Space-across-Effector, we trained the classifier to discriminate location for one effector (e.g., EyeL vs EyeR trials) and tested it on the other (e.g., HandL vs HandR). By similar logic, we tested Effector-across-Space by training the classifier to discriminate effector for one location (EyeL vs HandL) and testing it on the other (EyeR vs HandR). In sum, this additional analysis allows us to test whether the patterns of activity discriminating two movements are similar to the patterns of activity discriminating a different set of movements. For instance, although classifiers accurately discriminating both HandL versus EyeL and HandR versus EyeR pairwise comparisons indicate that actions are being planned with respect to an effector, accurate cross trial-type decoding would furthermore show that the effector specificity is represented to some degree independent of target location. Likewise, in terms of spatial specificity, by crossing classifier training and testing for EyeL versus EyeR and HandL versus HandR trials, we can assess whether eye and arm movements are being prepared by using a similar spatial target representation to some degree independent of the effector to be used (as reported in the monkey; Andersen et al., 1985; Batista et al., 1999; Andersen and Buneo, 2002). Critically, similar to many natural environmental interactions—where the spatial locations of targets remain constant and it is only our planned movements that change—in our task the positions of the two target objects never changed from trial to trial.
To localize a set of common ROIs that could then be selected in each individual subject for use in the MVPA analysis, we performed a whole-brain, group level, voxelwise search to find areas where the activity during movement generation [i.e., movement planning (Plan) and execution (Execute)] was higher than the activity during simple visual object presentation (Preview; when subjects had vision of both targets and yet were unaware of which action to perform). This [Plan & Execute > Preview] contrast revealed activity throughout a well documented parieto-frontal network of areas (Fig. 2; see Table 1 for coordinates) frequently characterized in both human and monkey studies. Within this network, we focused MVPA on eight commonly described neuroanatomical ROIs in both the left and right hemispheres (16 ROIs total), each previously implicated in visual–motor processing in both the human and monkey: (1) superior parieto-occipital cortex (SPOC), a general area involved in reach preparation and execution (Galletti et al., 1997; Prado et al., 2005; Cavina-Pratesi et al., 2010); (2) posterior intraparietal sulcus (pIPS), a common area involved in attention and eye and hand movement-related processes (Calton et al., 2002; Beurze et al., 2009; Chang and Snyder, 2010; Szczepanski et al., 2010); (3) middle IPS (midIPS), an area often implicated in both eye and reach movement planning (Andersen et al., 1985; Snyder et al., 1997; Batista et al., 1999; Culham et al., 2006); (4) dorsal premotor cortex/frontal eye fields (PMd/FEFs), a general neuroanatomical region primarily involved in reach-related processes (Caminiti et al., 1990a, 1990b; Pesaran et al., 2006; Cavina-Pratesi et al., 2010) and eye movements (Bruce and Goldberg, 1985; Kastner et al., 2007; Pertzov et al., 2011) [note that due to the particular difficulty of dissociating the activations from PMd and FEF with fMRI (e.g., Connolly et al., 2007; Filimon, 2010; although see Amiez et al., 2006), combined with the fact that activation from the same anatomical brain region (i.e., junction of superior frontal and precentral sulci) will be interchangeably named PMd or FEF depending on whether the task involves arm or eye movements (e.g., Beurze et al., 2007, 2009; Kastner et al., 2007; Pertzov et al., 2011), in this study we have chosen to denote this one general functional region as PMd/FEF]; (5) precentral gyrus, an area often implicated in general eye and hand motor-related processes (Beurze et al., 2009; Filimon, 2010); (6) ventral premotor (PMv) cortex, an ROI most often implicated in hand-related actions (Graziano et al., 1994; Rizzolatti and Luppino, 2001; Cavina-Pratesi et al., 2010; Filimon, 2010); 7) dorsomedial frontal cortex (DMFC), a general motor area involved in internally generated eye and arm actions (Kermadi et al., 1997; Luna et al., 1998; Deiber et al., 1999; Pierrot-Deseilligny et al., 2004); and (8) dorsolateral prefrontal cortex (DLPFC), a ROI most often implicated in maintaining items of interest in working memory (Funahashi et al., 1989; Goldman-Rakic, 1996). One additional area, left somatosensory cortex (SS-cortex), was selected as a sensory control region and was not expected to accurately decode movements until stimulation of the hand's mechanoreceptors at movement onset (i.e., at Execute phase). The critical question of interest here was whether we would be able to predict upcoming reaches and/or saccades and their movement directions from preparatory activity in each of these prespecified areas.
Parieto-frontal brain areas selected for MVPA. Cortical areas that exhibited larger responses during movement generation than the preceding visual phase [(Plan + Execute) > 2*(Preview)] are shown in orange/yellow activation. Results calculated across all subjects (Random Effects GLM) are displayed on one representative subject's inflated hemispheres. The general locations of the selected ROIs are outlined in circles (actual ROIs were anatomically defined separately in each subject). L, Left; R, Right (ROI acronyms are spelled out in main text). Sulcal landmarks are denoted by white lines (stylized according to the corresponding legend). LH, Left hemisphere; RH, right hemisphere; corr., corrected; df, degrees of freedom.
Subjects
Eight right-handed volunteers participated in this study (four females; mean age, 24.3 years) and were recruited from the University of Western Ontario (London, Ontario, Canada). Informed consent was obtained in accordance with procedures approved by the University's Health Sciences Research Ethics Board.
Setup and apparatus
Each subject's workspace consisted of a black platform placed over the waist and tilted away from the horizontal at an angle (∼10–15 degrees) to maximize comfort and target visibility. To facilitate direct viewing of the workspace, we also tilted the head coil (∼20 degrees) and used foam cushions to give an approximate overall head tilt of 30° (see Fig. 1A). Participants performed individual movements with the eyes or the right hand toward one of two object locations when required. To minimize limb-related artifacts, participants had the right upper arm braced, limiting movement to the elbow and creating an arc of reachability (Fig. 1B). The target objects were made of LEGO pieces [7 × 3 × 3 cm (length × depth × height)] and were secured to the workspace at a location along the arc of reachability for the right hand. The target stimuli were painted white to increase their contrast with the black background of the platform. The exact placement of the objects on the platform was adjusted to match each participant's arm length, such that all required movements were comfortable. The left target object was placed on the left side of the platform within reach by the participant's right hand, and the right target object was placed on the right side of the platform equidistant with respect to the subject's sagittal plane and at a further distance nearing the maximal extent of the participant's reach. Once specified, the target objects were secured to the platform at these locations (see Fig. 1B). During the experiment, the two objects were illuminated simultaneously from the front by a bright white light-emitting diode (LED) attached to flexible plastic stalks (Loc-Line, Lockwood Products). Each trial was preceded by a period where participants were in complete darkness. During participant setup, the illuminator LED was positioned so as to equally illuminate the objects in both locations. Experimental timing and lighting were controlled with in-house software created with Matlab (The Mathworks). See Figure 1, A and B, for an overview of the experimental setup and task. To control for eye movements, a small green fixation LED was placed above and immediately between the two target objects, and subjects were required to always foveate the fixation LED unless a saccade was executed. Eye fixation, saccades, and arm movements were examined off-line from videos recorded using an MR-compatible, infrared-sensitive camera positioned underneath the fixation LED and directed toward the subject's eyes and hand.
For each trial, the subjects were required to perform one of four actions upon the target objects following a delay period: (1) manually touch the top of the left object with the knuckles (“Touch Left” auditory command; HandL trial), which required transporting the hand to the object without hand preshaping; (2) touch the right object (“Touch Right” auditory command; HandR trial), which required the exact same hand posture; (3) saccade to the left object (“Look Left” auditory command; EyeL trial); or (4) saccade to the right object (“Look Right” auditory command, EyeR trial). To specifically control for the amplitude of the saccades and allow equivalency across trials for both of the eye movement conditions, small black dot stickers were placed on the centers of the target stimuli (on the object surface which faced the subject) and, when cued, participants were required to saccade to these specific locations. Importantly, for each trial the target objects never changed their peripherally located positions, thus eliminating retinal differences across the experiment. Critically, from trial to trial it was only the subject's movement intentions that changed.
Experiment design and timing
To parse the visual–motor planning response from the simple visual and motor execution responses, we used a slow event-related planning paradigm with 34 s trials, each consisting of three distinct phases: “Preview,” “Plan,” and “Execute” (see Fig. 1C). We adapted this paradigm from previous work with eye and arm movements that have successfully isolated delay activity from the transient responses following the onset of visual input and movement execution (Curtis et al., 2004; Beurze et al., 2007, 2009; Gallivan et al., 2011; Pertzov et al., 2011).
Each trial began with the Preview phase where the subject's workspace was illuminated, revealing the two peripherally located target objects. After 6 s of the Preview phase, subjects were given an auditory cue (0.5 s), either “Touch Left,” “Touch Right,” “Look Left,” or “Look Right,” informing them of the upcoming movement required; this cue marked the onset of the Plan phase. Although there were no visual differences between the Preview and Plan phase portions of the trial (i.e., both objects were always visually present), only in the Plan phase did subjects have all the information necessary (i.e., conjunction of effector and target location) to prepare the upcoming movement. After 12 s of the Plan phase, a 0.5 s auditory beep cued participants to immediately execute the planned action (for a duration of ∼2 s), initiating the Execute phase of the trial. Two seconds following the beginning of this Go cue, the illuminator was turned off. For reach movements, subjects were instructed to touch the top of the target object and return the hand to its central starting position when the illuminator was extinguished. For eye movements, subjects were instructed to saccade to the target object (and foveate the black dot) and then return the eyes to the fixation LED when the illuminator was extinguished. Other than the execution phase of each arm action, throughout the other phases of the trial (Preview, Plan and intertrial interval (ITI)] the hand was to remain still and in a relaxed “home” position on the platform between the two target objects. To provide a tactile landmark at the home position for the hand to return to following completion of an arm movement, a small elevated plastic nib was secured to the platform at this location before initiating the experiment. After the illuminator was turned off, subjects then waited in the dark while maintaining fixation for 14 s, allowing the BOLD response to return to baseline before the next trial (ITI phase). The four trial types, with five repetitions per condition (20 trials total), were randomized within a run and balanced across all runs so that each trial type was preceded and followed equally often by every other trial type across the entire experiment.
Practice sessions were carried out to familiarize participants with the paradigm, namely the delay timing that required the cued action to be performed only at the beep (Go) cue. These sessions were conducted before the subjects entered the scanner as well as during the anatomical scan (collected at the beginning of every experiment). A testing session for one participant included set-up time (∼45 min), eight or nine functional runs, and one anatomical scan, and lasted ∼2.5–3 h. Throughout the experiment, the subject's eye movements (as well as hand movements) were monitored using an MR-compatible, infrared-sensitive camera optimally positioned directly below the fixation point (MRC Systems). The videos captured during the experiment were analyzed off-line to verify that the subjects were indeed performing the task as instructed. A more rigorous tracking of the eyes was not performed because our eye-tracking software does not work while the head is tilted due to a partial occlusion from the eyelids.
MRI acquisition and preprocessing
Imaging was performed on a 3 tesla Siemens TIM MAGNETOM Trio MRI scanner. The T1-weighted anatomical image was collected using an ADNI MPRAGE sequence [time to repetition (TR), 2300 ms; time to echo (TE), 2.98 ms; field of view, 192 × 240 × 256 mm; matrix size 192 × 240 × 256 mm; flip angle, 9°, 1 mm isotropic voxels]. Functional MRI volumes were collected using a T2*-weighted, single-shot, gradient-echo echo-planar imaging acquisition sequence [TR, 2000 ms; slice thickness, 3 mm; in-plane resolution, 3 × 3 mm; TE, 30 ms; field of view, 240 × 240 mm; matrix size, 80 × 80 mm; flip angle, 90°; and acceleration factor (integrated parallel acquisition technologies, iPAT), 2 with generalized, auto-calibrating, partially parallel acquisition reconstruction]. Each volume comprised 34 contiguous (no gap) oblique slices acquired at a ∼30° caudal tilt with respect to the plane of the anterior and posterior commissure (ACPC), providing near whole brain coverage. We used a combination of imaging coils to achieve a good signal:noise ratio and to enable direct viewing without mirrors or occlusion. Specifically, we tilted (∼20° degrees) the posterior half of the 12-channel receive-only head coil (six channels) and suspended a four-channel receive-only flex coil over the anterior–superior part of the head. The cortical surface from one subject was reconstructed from a high-resolution anatomical image, a procedure that included segmenting the gray and white matter and inflating the boundary surface between them. This inflated cortical surface was used to overlay group activation for figure presentation (also note that voxel activity was spatially interpolated from 3 mm functional isovoxels to 1 mm functional isovoxels for all group data figures). All preprocessing and univariate analyses were performed using Brain Voyager QX version 2.12 (Brain Innovation).
Following slice scan time correction, 3D motion correction (such that each volume was aligned to the volume of the functional scan closest to the anatomical scan), high-pass temporal filtering (4 cycles/run), and functional-to-anatomical coregistration, functional and anatomical images were rotated such that the axial plane passed through the ACPC space and then transformed into Talairach space. Other than the sinc interpolation inherent in all transformations, no additional spatial smoothing was performed. Talairach data were only used for group voxelwise analyses to define a set of a priori action-related ROIs that were common across all subjects. We then defined these same areas anatomically within each subject's ACPC data (native space). Given that MVPA discriminates spatial patterns across voxels, we found it beneficial to select ROIs at the single subject level and to use ACPC data in lieu of the Talairach data (thus avoiding further voxel smoothing due to Talairach interpolation). We explicitly tested this in two of our subjects and observed an average decrease in decoding accuracies of ∼1% for Talairach data versus ACPC data. Using the ACPC data also had the advantage that each region of interest could be reliably identified in single subjects regardless of variations in slice planes, a particular problem given the sulcal variability of parietal cortex.
For each participant, functional data from each session were screened for motion and/or magnet artifacts by examining the time course movies and the motion plots created with the motion correction algorithms. None of the runs revealed head motion that exceeded 1 mm translation or 1° rotation. Error trials—trials where the participant fumbled with the object, performed the incorrect instruction, or contaminated the plan phase data by slightly moving their limb or eyes or by performing the action before the “Go” cue—were identified off-line from the videos recorded during the session; only two trials from two subjects (four trials total) contained such movement errors. This very low error rate likely reflects the fact that by the time subjects actually performed the required eye and hand movements in the scanner, they were highly practiced and well trained in the delay task.
Regions of interest
To first localize the specific action-related areas common among all individuals in which to implement MVPA, we used a random effects (RFX) general linear model (GLM) group voxelwise analysis (on the Talairach-transformed data). Predictors were created from boxcar functions convolved with the Boynton hemodynamic response function. For each trial, a boxcar function was aligned to the onset of each phase, with a height of 1 and a duration dependent upon the phase as follows: (1) 3 volumes for the Preview phase; (2) 6 volumes for the Plan phase; and (3) 1 volume for the execute phase. The ITI was excluded from the model, and therefore all regression coefficients (βs) were defined relative to the baseline activity during the ITI. In addition, the time course for each voxel was converted to percentage signal change before applying the RFX-GLM.
To specify our ROIs at the group level (allowing further investigation of common ROIs at the single-subject level), we searched for brain areas involved in movement generation by contrasting activity for movement planning and execution (collapsed over effector and spatial target location) versus the simple visual response to object presentation before instruction: [Plan(EyeL + EyeR + HandL + HandR) + Execute(EyeL + EyeR + HandL + HandR) > 2 * Preview(EyeL + EyeR + HandL + HandR)]. The resulting statistical map of all positively active voxels (RFX, t(7) = 3, p < 0.01, cluster threshold corrected to 278 mm3) was then used to define 17 different ROIs (eight ROIs on both the left and right and one ROI, somatosensory cortex on the left; for activity from this contrast, see Fig. 2). Eight of these ROIs (across parietal and premotor cortex) were selected based on their well documented involvement in movement planning/execution, and the final ROI, left somatosensory cortex, was selected as a sensory control region [i.e., known to respond to transient stimuli (i.e., sensory events), but not expected to participate in sustained movement planning/intention-related processes]. Importantly, each of these ROIs could then be easily anatomically localized in each individual's ACPC-aligned data (see below, ROI selection).
The specific voxels submitted for MVPA were then selected from the (Plan & Execute > Preview) GLM contrast on single subject ACPC-aligned data and based on all significant activity within a 3375 mm3 cube centered on the predefined anatomical landmarks for each of the 17 ROIs [t = 3, p < 0.005, each subject's activation map was cluster threshold corrected (corrected, p < 0.05) so that only voxels passing a minimum cluster size were selected; average minimum cluster size across subjects was 112.5 mm3; for details see below, ROI selection]. These ROI sizes were chosen because they not only allowed the inclusion of several functional voxels for pattern classification (an important consideration), but also ensured that adjacent ROIs did not substantially overlap (for the average number of functional voxels selected across the eight subjects, see Table 1). Critically, given the orthogonal contrast employed to select these 17 areas (i.e., Plan & Execute > Preview), their activity is not directionally biased to show any preview-, plan-, or execute-related pattern differences between any of the experimental conditions (for verification of this fact, see the univariate analyses in Fig. 6). All univariate statistical tests are Greenhouse–Geisser corrected, and for post hoc tests (two-tailed paired t tests) we applied a threshold of p < 0.05. Only significant results are reported (see Fig. 6).
ROI selection
Left and right superior parieto-occipital cortex.
The superior parieto-occipital cortex or SPOC is defined by selecting voxels located medially and directly anterior to the parieto-occipital sulcus (POS) (Gallivan et al., 2009).
Left and right posterior IPS.
The posterior IPS or pIPS is defined by selecting activity at the caudal end of the IPS (Sakata et al., 1998; Beurze et al., 2009).
Left and right middle IPS.
The middle IPS or midIPS is defined by selecting voxels half-way up the length of the IPS, centered on the medial bank (Calton et al., 2002), and near a characteristic “knob” landmark observed consistently within each subject.
Left and right dorsal premotor cortex/frontal eye fields.
The dorsal premotor cortex, PMd/frontal eye field, FEF, region is defined by selecting voxels at the junction of the precentral sulcus (PreCS) and superior frontal sulcus (SFS) (Picard and Strick, 2001; Kastner et al., 2007; Pertzov et al., 2011).
Left and right precentral gyrus.
The precentral gyrus is defined by selecting voxels lateral to the junction of the PreCS and SFS, encompassing the precentral gyrus and posterior edge of the PreCS (Gallivan et al., 2011).
Left and right ventral premotor cortex.
The ventral premotor cortex, PMv, is defined by selecting voxels slightly inferior and posterior to the junction of the inferior frontal sulcus (IFS) and PreCS (Tomassini et al., 2007).
Left and right dorsomedial frontal cortex.
The dorsomedial frontal cortex or DFMC is defined by selecting voxels near the interhemispheric fissure adjacent to the paracentral sulcal branch of the cingulate sulcus (Kastner et al., 2007; Pertzov et al., 2011).
Left and right dorsolateral prefrontal cortex.
The dorsolateral prefrontal cortex, DLPFC, is defined by selecting voxels anteriorly located along the intermediate frontal sulcus (Curtis and D'Esposito, 2003).
Left somatosensory cortex.
The left somatosensory cortex, L-SS, is defined by selecting voxels encompassing the postcentral gyrus and postcentral sulcus (PCS) medial and anterior to the anterior IPS.
See Table 1 for details about ROI coordinates and sizes and Figure 2 for representative anatomical locations on one subject's brain.
ROI locations and size across subjects
To provide a type I error control, we also tested the performance of our classifiers in ROIs outside of our action-related network where no statistically significant classification should be possible. To select these ROIs, we further reduced our statistical threshold (after specifying the [Plan & Execute > Preview] network within each subject) down to t = 0, p = 1 and selected all positive activation within 3375 mm3 centered on a consistent point: (1) within each subject's right ventricle; and (2) at a location situated just outside the skull of the brain, near the right visual cortex in the ACPC plane.
Multivoxel pattern analysis
Whereas conventional univariate fMRI analyses examine each voxel separately—and typically smooth and average activity across multiple adjacent voxels to detect differences in signal amplitude—multivoxel pattern analysis or MVPA instead uses classification algorithms to differentiate the fine-grained spatial voxel patterns elicited by different classes of stimuli. In effect, voxel pattern classification is able to reveal distributed neural representations contained in spatial activity patterns that might be ignored or missed by traditional analysis approaches (Kriegeskorte and Bandettini, 2007; Mur et al., 2009; Pereira et al., 2009; Raizada and Kriegeskorte, 2010; Kriegeskorte, 2011). It was this fine-grained nature of MVPA that prompted us to examine not only if movement plans could be decoded from preparatory human brain activity (where little or no signal amplitude differences typically exist) but also examine both the effector and spatial specificity of the intended movements across multiple parieto-frontal areas. The additional benefit of the MVPA technique is that it allowed us to investigate the underlying mechanisms of the predictive neural representations. For this, we used cross trial-type classification to directly examine whether the differences in the voxel activity patterns elicited by two different planned movements were similar to the differences in the voxel activity patterns elicited by two other planned movements within the same area (for further details see below, Cross trial-type decoding).
Support vector machine classifiers
MVPA was performed with a combination of in-house software (using Matlab) and the Princeton MVPA Toolbox for Matlab (http://code.google.com/p/princeton-mvpa-toolbox/) using a Support Vector Machines (SVM) binary classifier (libSVM, http://www.csie.ntu.edu.tw/∼cjlin/libsvm/). The SVM model used a linear kernel function and a constant cost parameter, C = 1 (congruent with many other fMRI studies; LaConte et al., 2003; Mitchell et al., 2003; Mourao-Miranda et al., 2005; Haynes et al., 2007; Pessoa and Padmala, 2007), to compute the hyperplane that best separated the trial responses.
In brief, MVPA with linear SVM classifiers (Kamitani and Tong, 2005; Harrison and Tong, 2009; Meyer et al., 2010; Chen et al., 2011) requires a model to be “trained” with a subset of the data and then “tested” with an independent set (a more detailed explanation can be found elsewhere, e.g., Pereira et al., 2009). To verify the generalizability of the set of trials into two separate stimulus classes, an iterative cross-validation procedure in which several independent subsets of trials are used to train and test the classifier is employed. The separability of the sets of trials into the correct classes is then assessed by comparing the average accuracy of the classifier over N iterations to the chance level (Duda et al., 2001).
Voxel pattern preparation
To prepare the data for spatial pattern classification, the percentage signal change was computed from a windowed average of the time course at a time point of interest (e.g., Preview, Plan, or Execute) with respect to a windowed average of the time course at a common baseline for each voxel in the ROI (a procedure similar to that used for analyzing event-related average time courses). The baseline window was defined as the average of volumes −1 and 0 with respect to the start of the trial (before initiation of the trial). For the Preview phase time points, we extracted the mean of volumes 3–4; time points corresponding to the peak of the visual transient response (see Fig. 3) [note that although volumes 3–4 encompass time points both 1 volume before and 1 volume after the auditory instruction, the activity during this time window–given the sluggishness of the BOLD response–can only be attributable to a simple visual response and cannot reflect any plan-related activity initiated by the auditory cue]. For the Execute phase time points, we extracted the average of volumes 12–13, time points corresponding to the peak of the transient movement response following the subject's action (see Fig. 3). Lastly, for the Plan phase we extracted the average of volumes 8–9 (the final two volumes of the Plan phase), corresponding to the sustained activity of a planning response (see Fig. 3). Following the extraction of each trial's percentage signal change, these values were z-scored across the run for each voxel within an ROI.
Trial-related percentage signal change neural activity in the parieto-frontal regions used for MVPA. Activity in each plot is averaged across voxels within each ROI and across subjects. Vertical dashed lines correspond to the onset of the Preview, Plan, and Execute phases of each trial (from left to right). Shaded gray bars highlight the 2-volume (4 s) windows that were averaged and extracted for MVPA (a conventional univariate analysis of signal amplitude differences within these same time-windows is provided in Fig. 6). Note that time corresponds to imaging volumes (TR = 2) and not seconds.
Of all the Plan phase time points we could have used to capture the activity associated with movement planning, we used the average of volumes 8–9 (the final two volumes of the Plan phase) given the common observation that planning (i.e., intending to perform a movement) often involves sustained neural processes (although it is worth noting that portions of motor planning are also supported by transient neural responses; for example, see Weinrich and Wise, 1982; Riehle et al., 1997; Ohbayashi et al., 2003). While simple stimulus (i.e., visual) and movement execution responses ubiquitously show transient neural activity (Andersen et al., 1997; Andersen and Buneo, 2002)—where the fMRI hemodynamic response peaks approximately at 6 s after the event and then falls—planning responses typically remain high for the duration of the intended movement (Curtis and D'Esposito, 2003; Curtis et al., 2004; Chapman et al., 2011; Gallivan et al., 2011). As such, we reasoned that if pattern differences were to arise during movement planning, they would be more likely to occur during the sustained planning response. For these reasons, we selected the final two volumes of the Plan phase to serve as our data points of critical interest, a 2 volume window where the BOLD response had already reached its peak and, rather importantly, a time point before the subject had initiated any movement. This time-dependent approach, in addition to revealing which types of movements could be decoded, also examined when predictive information pertaining to specific actions arose (i.e., within the Preview, Plan, or Execute phase).
Pairwise discriminations
SVMs are designed for classifying differences between two stimuli and LibSVM (the SVM package implemented here) uses the so-called “One-against-One Method” for each pairwise discrimination. Although it is often the case that multiple pairwise results are combined to produce multiclass discriminations (Hsu and Lin, 2002) (i.e., distinguish among more than two stimuli), for the purposes of this particular experiment (i.e., to characterize brain regions according to the types of upcoming movements they could predict: eye vs hand, left vs right), we found it imperative to examine the individual pairwise discriminations separately (e.g., decoding accuracies for HandL vs HandR). For instance, a brain region that could predict HandL versus HandR trials and EyeL versus EyeR trials, but not HandL versus EyeL or HandR versus EyeR trials—an interesting theoretical finding here—would not be readily apparent with a traditional multiclass discrimination approach, and indeed a further detailed investigation of this relationship would require the individual pairwise discriminations to be assessed independently in any case.
Single-trial classification
For each subject and for each of the 17 action-related ROIs, 12 separate binary support vector machine classifiers were estimated for MVPA (i.e., for each of the Preview, Plan, and Execute phases and each pairwise comparison: HandL vs EyeL, HandR vs EyeR, HandL vs HandR, and EyeL vs EyeR). We used a “leave-five-trials-out” cross-validation to test the accuracy of the binary SVM classifiers, meaning that five trials from each of the conditions being compared (i.e., 10 trials total) were reserved for testing the classifier and the remaining trials were used for classifier training (i.e., 35 or 40 remaining trials per condition, depending on whether the subject participated in eight or nine experimental runs, respectively). Single trials in the independent test dataset (10 total), before classifier testing, were averaged according to condition (i.e., creating two averaged data points to be separated in multidimensional voxel space), thus improving voxel pattern signal-to-noise in the test dataset (see also Smith and Muckli, 2010). Because a full cross validation is not entirely reasonable with a “leave-five-trials-out” design due to the ∼105 possible iterations, to provide a highly reliable estimate of decoding accuracies we performed 1008 train-and-test iterations for each pairwise discrimination (the precise reason for this number of iterations is explained below).
To ensure unbiased classification results, a necessary consideration for single-trial classification analysis is that each individual trial and condition type, in addition to being randomly selected for each iteration, be equally represented across the total number of iterations for classifier training and testing. We achieved this by running 1008 iterations in each subject (where in subjects with eight runs, each trial was used exactly 126 times to train the classifier, and in subjects with nine runs, each trial was used exactly 112 times to train the classifier). This large number of train-and-test iterations produces a highly representative sample and a precise estimate of true classification accuracies. For instance, this approach provided a test–retest reliability within ±0.5% based on multiple simulations of 1008 iterations conducted in two subjects. We statistically assessed decoding significance with a two-tailed t test versus 50% chance decoding. To control for the problem of multiple comparisons, a false discovery rate (FDR) correction of q ≤ 0.05 was applied based on all the t tests performed (Benjamini and Yekutieli, 2001).
Permutation tests
In addition to the t test, we separately assessed statistical significance with nonparametric randomization tests (Golland and Fischl, 2003; Etzel et al., 2008; Smith and Muckli, 2010; Chen et al., 2011), which also determined that the chance distribution of decoding accuracies was approximately normal and had a mean around 50%. Following classifier training (and testing) with the true trial identities, for each subject, ROI, and pairwise comparison, we permuted the correspondence between the Test trial identities and data 100 different times before testing the classifier and then computed classifier performance the same as before (average of 1008 train-and-test iterations) for each individual permutation of the Test labels. This produced 100 mean accuracies (the one “true” mean accuracy containing the correct test labeling was appended to this permuted distribution). We then generated a randomized population of 1000 mean accuracies based on 1000 combinations of randomly drawn accuracies from each subject's permuted distribution (of 101 accuracies) and then found the true group mean accuracy's empirical probability based on its place in a rank ordering of this randomized distribution. The peak percentiles of significance (p < 0.001) are limited by the number of samples producing the randomized probability distribution at the group level. The findings from this nonparametric randomization test produced significant results with much higher significance than those found with a standard parametric t test (a finding also noted by Smith and Muckli, 2010; Chen et al., 2011). For instance, decoding accuracies showing statistical significance at p < 0.05 with the standard t test showed significance at p < 0.001 with the permutation tests. This indicates that the t test group analysis (the one performed in Fig. 4) provides a highly conservative estimate of the statistical significance of the decoding accuracies. The important finding highlighted from these permutation tests is that the brain areas showing significant decoding with the one sample parametric t tests (vs 50%) also show significant decoding (albeit higher) with the empirical nonparametric permutation tests.
Cross-trial type decoding
To test whether a SVM pattern classifier trained to discriminate between two trial types could then be used to accurately decode pattern differences when tested on a different sets of trials (e.g., train set, EyeL vs EyeR; test set, HandL vs HandR) instead of using a “leave-five-trials-out” cross-validation analysis (implemented above), we used all the available single trial data for both classifier training and testing (i.e., one train-and-test iteration; see Smith and Muckli, 2010). Mean decoding accuracies for each subject were computed by averaging together the two accuracies generated by using each pair of trial types for classifier training and testing (e.g., Eye trials were used to train the classifier in one analysis when Hand trials were used for testing, and then they were used to test the classifier in the other analysis when the Hand trials were used for classifier training). The means across subjects of this cross trial-type decoding approach are reported in Figure 5. For additional verification, we also performed a leave-five-trials-out cross trial-type cross-validation analysis using the same number of train-and-test iterations as employed in our standard decoding analysis (i.e., 1008), which produced nearly identical results. We statistically assessed cross-decoding significance with a two-tailed t test versus 50% chance decoding. A FDR correction of q ≤ 0.05 was applied based on all the t tests performed (Benjamini and Yekutieli, 2001).
Results
Decoding movement plans for reaches and saccades
During movement planning we found a wide range of decoding profiles across the network of specified parieto-frontal brain regions; activity patterns in several of the areas predicted the effector to be used (eye vs hand) or the target location to be acted upon (left vs right), and in several regions we could accurately predict both types of movement information (Fig. 4). For instance, in parietal cortex, L-SPOC and R-SPOC—a human and monkey region often implicated in reaching (Galletti et al., 1997; Prado et al., 2005; Fattori et al., 2009; Gallivan et al., 2009; Cavina-Pratesi et al., 2010)—not only showed effector-specific preparatory responses but also predicted the direction of an upcoming reach (but not the direction of an impending eye movement). In both L-pIPS and R-midIPS, general areas in the human and monkey implicated in eye and hand movement processes (Snyder et al., 1997; Chang et al., 2008; Beurze et al., 2009; Chang and Snyder, 2010), a combined effector-specific and spatially specific planning response was observed; both areas predicted all conjunctions of the effector to be used (hand vs eye) and the location to be acted upon (left vs right). Interestingly, this contrasts with the decoding profile found in L-midIPS, which instead showed only effector-independent movement intentions; the spatial location to be acted upon was encoded regardless of the effector required to perform the movement. In premotor cortex we found a similar array of decoding profiles: (1) the reach- and saccade-related L-PMd/FEF (Bruce and Goldberg, 1985; Caminiti et al., 1990a, 1990b; Pesaran et al., 2006; Kastner et al., 2007; Andersen and Cui, 2009; Pertzov et al., 2011) predicted all conjunctions of spatial and effector movements; (2) the hand-related L-PMv (Graziano et al., 1994; Rizzolatti and Luppino, 2001; Cavina-Pratesi et al., 2010) encoded effector-independent (spatially specific) movement intentions; and (3) the R-PMd/FEF, L- and R-precentral gyrus, L- and R-DMFC, and L-DLPFC regions encoded mostly reach-specific movement plans (i.e., decoding the effector to be used and reach direction). Importantly, consistent with expectations, our control region, L-SS cortex, failed to decode planned movements (and only the executed actions; see Fig. 4), reinforcing the notion that predictive movement information is constrained within plan-related parieto-frontal areas. As an additional control for our decoding accuracies, we ran the same classification analyses in two nonbrain ROIs where decoding is unlikely: the right ventricle and outside the brain. Indeed, MVPA in these two areas showed no accurate decoding for any trial phase (see Fig. 7).
Decoding movement intentions across the parieto-frontal network. Decoding accuracies are shown for each time phase (Preview, Plan, and Execute) in each ROI. Classifier training was done on single trials and tested on the average activity patterns of the single trials for each condition in the independent test dataset. Importantly, accurate classification can only be attributed to the spatial response patterns of different planned movement types and not to the overall signal response amplitudes within each ROI (see Fig. 6). Note that decoding accuracies are color coded according to pairwise discriminations and not trial types. Error bars represent standard error of the mean (SEM) across subjects. Solid black lines are chance accuracy level (50%). Black asterisks assess statistical significance with t tests across subjects with respect to 50%. Red asterisks assess statistical significance based on a FDR correction of q ≤ 0.05 (critical p value of 0.012), based on all t tests performed.
Effector-dependent and Effector-independent movement plans revealed with cross-decoding
Expanding on the within-trial MVPA analysis, we also examined whether certain movement intentions were coded using similar distributed neural mechanisms by applying cross trial-type MVPA (see Dinstein et al., 2008; Formisano et al., 2008; Harrison and Tong, 2009). To test whether the spatially selective coding of planned reach and saccade movements found in several of the parieto-frontal areas could be partially explained on the basis of an effector-independent target representation (i.e., similar spatial target encoding across the hand and eye), we trained pattern classifiers using EyeL and EyeR trials and tested the accuracy of the classifiers using HandL and HandR trials (the opposite train-and-test process—train set, HandL vs HandR; test set, EyeL and EyeR—was also performed and we averaged the accuracies from both approaches). This Space-across-Effector decoding approach was statistically significant in L-pIPS, R-pIPS, L-midIPS, R-midIPS, and L-PMd/FEF (Fig. 5, light gray bars). Significant cross trial-type decoding seems sensible in many of the areas where the within-trial activity patterns are already able to accurately discriminate both reach and saccade direction (i.e., HandL vs HandR and EyeL vs EyeR trials) like in L-pIPS or L-midIPS (see Fig. 4) and may further suggest that the spatial specificity of the response in these areas may be partially reliant on a common target encoding for the hand and eye (given that the spatial locations of the targets are constant across the planning of reaches and saccades). In fact, even nonsignificant cross trial-type decoding may be informative in areas able to individually discriminate reach and saccade direction (e.g., L-PMv), as it could suggest that the spatial target representations underlying eye and hand movements may be different. There is difficulty, however, in interpreting the Space-across-Effector results from the plan-related activity in R-pIPS and the execute-related activity in L-midIPS, as both these regions fail to show accurate decoding for EyeL versus EyeR trials in the respective trial phases when we used within-trial MVPA (see Fig. 4).
Cross trial-type decoding accuracies examining the degree of effector specificity and spatial specificity of the intended movements. Decoding accuracies are shown for each time phase (Preview, Plan, and Execute) in each ROI. Effector-across-space accuracies were computed from training classifiers on HandL versus EyeL trials and testing on HandR versus EyeR trials and then averaging these values with the opposite train-and-test ordering within each subject. Space-across-effector accuracies were computed from training classifiers on EyeL versus EyeR trials and testing on HandL versus HandR trials (again, averaging these values with the opposite train-and-test ordering within each subject). Error bars represent SEM across subjects. Solid black lines are chance accuracy level (50%). Black asterisks assess statistical significance with t tests across subjects with respect to 50%. Larger dark asterisks assess statistical significance based on a FDR correction of q ≤ 0.05 (critical p value of 0.010), based on all t tests performed.
We next examined whether the effector specificity of the planned movements in several parieto-frontal regions could be further discriminated from resident spatial processes in the same areas by training classifiers using HandL and EyeL trials and testing the classifiers using HandR and EyeR trials (again, the opposite train-and-test process was also performed and the accuracies were averaged). This Effector-across-Space decoding analysis was statistically significant during planning in several regions: L-SPOC, L-pIPS, L-PMd/FEF, L- and R-precentral gyrus, and L- and R-DMFC (Fig. 5, dark gray bars), indicating that the neural coding for the effector to be used (hand vs eye) was motor-specific and not necessarily dependent on the spatial target location to be acted upon. Recall that, rather interestingly, several of these same areas also showed spatial tuning with respect to the planned target location using within-trial MVPA (Fig. 4). We comment further on this important finding in the below in the discussion below.
Critically, our results show that decoding revealed during the Plan phase can only be attributed to the intention to perform a specific movement given the absence of decoding observed during the preceding Preview phase, when movement planning information was unavailable (Figs. 4, 5).
Additional multivariate analyses
In addition to performing the aforementioned multivariate analyses, we also, to aid the interpretation of our cross trial-type decoding effects (i.e., examining how similar the patterns of activity were in the accurate cross-classification cases), performed considerable additional analyses. First, we examined the voxel weights assigned by the trained SVM classifier so as to create both voxel “importance maps” (for example, see Polyn et al., 2005; Ethofer et al., 2009; Harrison and Tong, 2009; Meyer et al., 2011) and “fingerprints” (see Formisano et al., 2008) in four representative single subjects, thereby delineating the voxels that receive a higher weighting depending on the planned movements trying to be decoded (this analysis was performed in six of our ROIs: L-SPOC, L-pIPS, L-midIPS, L-SS cortex, L-PMd/FEF, and L-PMv; data not shown). The direct aim of this voxel weight approach was to further explore the underlying nature of the accurate cross-classification effects by visually inspecting similarities in the weight mappings between the pairwise comparisons that make up each of the cross-decoding cases (i.e., Effector-across-Space and Space-across-Effector). We found that while the weighting of some voxels appeared to be highly consistent across pairwise comparisons within a subject, others appeared to change. In an attempt to further quantify the cross-decoding effects (i.e., determine why some effects do or do not cross classify), we ran a linear correlation between the weighting given to each single voxel for each constituent pairwise comparison of the cross-decoding cases. For instance, in the Effector-across-Space cross-decoding cases, we examined how the weighting in each voxel correlated between the HandL versus EyeL comparisons and the HandR versus EyeR comparisons (note that we also examined how the weighting in each voxel correlated in the Space-across-Effector cross-decoding cases by correlating the voxel weights assigned in the HandL versus HandR and EyeL versus EyeR pairwise comparisons). We intuitively reasoned that in the cases where there was above 50% chance cross-decoding, there might be some positive linear relationship between the voxel weights assigned for the two pairwise comparisons that make up a single cross-decoding case. We found that this hypothesized positive relationship was not reliable across subjects even in cases where above 50% cross-classification exists. We anticipate that this finding may reflect the fact that the SVM classifiers were trained to optimally discriminate the nuances of the spatial activity patterns evoked by a pair of trial types. Furthermore, if accurate cross decoding was driven by commonalities in a small subset of the voxels (each with high weightings), then a simple correlation—which weights all voxels equally—might not effectively capture this relationship. Indeed, it is also possible that a more complex relationship (i.e., nonlinear) describes the correspondence between the weighting given to each voxel in the successful cross-decoding cases, and this issue will surely be a matter for future methodological investigations.
We additionally performed a representational similarity analysis (RSA) (Kriegeskorte et al., 2008a, 2008b) in the same ROIs and the same four subjects to inspect and visualize the response pattern similarities for all pairs of movement plans. Interestingly, we found that the SVM classifier was able to reveal many pattern effects not noticed by the RSA; that is, in brain regions where the SVMs were able to predict and discriminate different upcoming movements, the RSA failed to reveal corresponding levels of discrimination (based on correlation values) in the voxel spatial patterns. For example, with similarity analyses we generally failed to show higher voxel pattern correlations for trials of the same class type than for trials of a different class type, a necessary finding for the successful implementation of an RSA approach (e.g., see face-selective correlation patches and animate vs inanimate correlation distinctions in human inferior temporal cortex; Kriegeskorte et al., 2008b). This result likely means that the parieto-frontal activity patterns that we are trying to decode (i.e., high-level movement plans and intentions) are somewhat noisy signals [for verification of this, note the relatively low-levels of SVM classifier performance we found across the different brain areas and movement plans, ∼55–65%, compared to the typical accuracies found in stimuli-viewing visual cortex decoding experiments using similar methods, ∼80% (e.g., Kamitani and Tong, 2005; Harrison and Tong, 2009)] and that their detectability may benefit from a decoding technique that differentially weights the voxels in an ROI (in contrast, voxel pattern correlation methods weight each voxel equally). There were a few areas, however, where RSA was quite successful in revealing significant pattern correlations within individual subjects (e.g., SS-cortex and L-PMd during the execution phases of the movement). We of course acknowledge that we may not have found interpretable RSA effects during movement planning because we did not use high-resolution fMRI imaging (e.g., 1.95 × 1.95 × 2 mm voxels as used in Kriegeskorte et al., 2008b) and used only the standard 3 × 3 × 3 mm functional imaging voxels typically used in sensorimotor experiments from our laboratory to get near full brain coverage.
Conventional univariate analysis of signal response amplitudes
Importantly, the accurate decoding shown here can only reflect the voxel spatial activity patterns for different movement plans and not the overall response amplitudes within each ROI. When we averaged trial responses across all voxels in each ROI (i.e., as done in conventional univariate fMRI analyses), we found only a few significant differences for the four planned movements (see Fig. 6).
Few signal amplitude differences found within the parieto-frontal regions and time windows used for MVPA. Responses are averaged across voxels within each ROI and across subjects for the 2-volume averaged windows corresponding to Preview, Plan, and Execute phases. Note that very few statistically significant univariate differences are found throughout the parieto-frontal network. Errors bars represent SEM across subjects.
Classifier decoding accuracies in nonbrain control regions. A, Nonbrain control ROIs defined in each subject (denoted in light orange; example subject shown). B, Classifier accuracies for the right ventricle (left) and outside the brain ROI (right). Error bars represent SEM across subjects. Solid lines show chance classification accuracy (50%). Note that no significant differences were found with t tests across subjects with respect to 50% chance. C, Percentage signal change activity from each selected region, averaged across subjects.
In addition to this analysis of signal response amplitudes, we also wanted to topographically characterize both effector-specific and spatially specific (effector independent) movement planning and execution processes at the group level in a more similar manner to that done in previous studies. The aim of these additional analyses was to replicate past fMRI findings using conventional subtraction methods and similar experimental tasks and further highlight and differentiate the types of predictive sensorimotor neural information that can be extracted from a multivariate analysis of the activity patterns (for a good example, see also Kriegeskorte and Bandettini, 2007). To do this, we performed a number of group RFX GLM subtraction contrasts.
First, to characterize the cortical topography of effector-specific movement planning processes (independent of spatial specificity), we searched for areas that showed higher activity for eye movement planning than the activity elicited by simple visual presentation of the targets before instruction of the eye movement (RFX GLM, t(7) = 3, p < 0.01 [Plan(EyeL + EyeR) > Preview(EyeL + EyeR)], cluster threshold correction, 291 mm3; Fig. 8, left, shown in yellow). To similarly characterize the areas engaged in the planning of hand movements, we performed the equivalent contrast ([Plan(HandL + HandR) > Preview(HandL + HandR)]; same statistical threshold, cluster threshold correction: 284 mm3; Fig. 8, left, shown in red, overlap for eye and hand shown in orange). We also further characterized movement execution topography for the eye versus hand (and their overlap) using similar contrasts and the same logic as above ([Execute(EyeL + EyeR) > Preview(EyeL + EyeR)]; [Execute(HandL + HandR) > Preview(HandL + HandR)], same statistical thresholds, same cluster thresholds; Fig. 8, right).
Activation topography of effector selectivity (eye vs hand) during movement planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t(7) = 3, p < 0.01, cluster threshold corrected) during movement planning for the eye or hand independent of the spatial target location: [Plan(EyeL + EyeR) > Preview(EyeL + EyeR)] in yellow; [Plan(HandL + HandR) > Preview(HandL + HandR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for movement execution of the eye or hand: [Execute(EyeL + EyeR) > Preview(EyeL + EyeR)] in yellow; [Execute(HandL + HandR) > Preview(HandL + HandR)] in red. The overlap of eye and hand movement planning or execution is shown in orange.
Second, we also examined the brain areas recruited for reach planning with respect to the upcoming reach direction (i.e., spatial-specificity for reach movements). We searched for areas with higher activation for planning reaches to the left or right targets versus visual presentation of the object before the reach instruction (RFX GLM, t(7) = 3, p < 0.01, [Plan(HandL) > Preview(HandL)], cluster threshold correction: 287 mm3; [Plan(HandR) > Preview(HandR)], cluster threshold correction: 269 mm3; Fig. 9, left, leftward reaches in yellow, rightward reaches in red, and their overlap shown in orange). The same type of contrasts were also applied to similarly characterize reach execution processes ([Execute(HandL) > Preview(HandL)]; Execute(HandR) > Preview(HandR)], same statistical thresholds, same cluster thresholds; Fig. 9, right).
Activation topography of spatial selectivity (left vs right targets) for reach planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t(7) = 3, p < 0.01, cluster threshold corrected) during reach planning depending on the spatial target location [Plan(HandL) > Preview(HandL)] in yellow; [Plan(HandR) > Preview(HandR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for reach execution: [Execute(HandL) > Preview(HandL)] in yellow; [Execute(HandR) > Preview(HandR)] in red. The overlap of left and right reach planning or execution is shown in orange.
Lastly, we wanted to characterize the regions of the brain recruited for saccade planning with respect to the direction of the saccade (i.e., spatial-specificity for eye movements). Consistent with the rationale above, we searched for areas that showed higher activation for the planning of saccades to the left or right versus visual presentation of the target objects before the eye movement instruction (RFX GLM, t(7) = 3, p < 0.01, [Plan(EyeL) > Preview(EyeL)], cluster threshold correction: 277 mm3; [Plan(EyeR) > Preview(EyeR)], cluster threshold correction: 284 mm3; Fig. 10, left; leftward saccades in yellow, rightward saccades in red, and the overlap shown in orange). We then also characterized the directionality of the executed saccade movements according to the same logic ([Execute(EyeL) > Preview(EyeL)]; [Execute(EyeR) > Preview(EyeR)], same statistical thresholds, same cluster thresholds; Fig. 10 right).
Activation topography of spatial selectivity (left vs right targets) for saccade planning and execution defined with conventional subtraction analyses. Left, Brain areas that showed significant activation (RFX GLM, t(7) = 3, p < 0.01, cluster threshold corrected) during saccade planning depending on the spatial target location: [Plan(EyeL) > Preview(EyeL)] in yellow; [Plan(EyeR) > Preview(EyeR)] in red. Right, Brain areas that showed significant activation (at the same statistical threshold) for saccade execution: [Execute(EyeL) > Preview(EyeL)] in yellow; [Execute(EyeR) > Preview(EyeR)] in red. The overlap of left and right saccade planning or execution is shown in orange.
With respect to the activation topography of hand movements, eye movements, their overlap, and the spatial specificity of these effector-specific responses, the group findings reported here correspond remarkably well with the parieto-frontal network of areas engaged for planned and executed reaches and/or saccades (Sereno et al., 2001; Curtis et al., 2004; Medendorp et al., 2006; Schluppeck et al., 2006; Beurze et al., 2007, 2009; Kastner et al., 2007; Kagan et al., 2010; Lindner et al., 2010), planned pointing and saccades (Connolly et al., 2000, 2003; Astafiev et al., 2003; Hagler et al., 2007), and reaching and saccade execution (Levy et al., 2007; Filimon et al., 2009), as well as other work from our laboratory using actual reaching (Culham et al., 2003, 2006; Culham and Valyear, 2006; see also Filimon, 2010 for review). These previous studies have also found overlapping and topographically mixed saccade and reach responses in the posterior and middle IPS and superior parietal cortex, as well as parts of PMd/FEF, precentral gyrus, PMv, DMFC, and DLPFC. One notable departure from these previous findings however, is that we failed to observe larger response amplitudes for reaching versus saccade execution in several anterior and superior parts of the PPC, as well as PMv and PMd. In fact, our univariate analysis of signal amplitude responses (for the same time points as those extracted for pattern classification) found no statistical differences between the execution of reaches and saccades except for the following: (1) L- and R-DMFC, areas thought to encompass the supplementary eye fields in humans (Kastner et al., 2007; Pertzov et al., 2011), showed higher activity for executed saccades versus reaches; and (2) SS-cortex, which showed higher activity for executed reaches versus saccades, entirely consistent with expectations (see Fig. 6). The pragmatic explanation of this present discrepancy with past findings (i.e., an overwhelming lack of significant signal amplitude differences between the execution of reaches and saccades across the parieto-frontal network) relates to the contrasts we employed to localize the action-related ROIs to be used for MVPA. To localize the ROIs in Figure 2, we searched for areas that simply showed higher activity for movement planning and execution compared to the simple transient responses that accompany visual presentation of the object before instruction (Plan + Execute > Preview). This type of general contrast was in fact necessary, because in addition to specifying ROIs preferentially involved in sensorimotor planning and control, it importantly specifies that the voxels submitted for MVPA are unbiased toward showing univariate differences between reaches and saccades and the movement directions (orthogonality of ROI localization for pattern classification is a critical consideration when interpreting the resultant decoding accuracies; Kriegeskorte et al., 2009). This general ROI selection procedure significantly contrasts with previous approaches that explicitly search for reach and saccade univariate differences (e.g., Astafiev et al., 2003; Hagler et al., 2007; Levy et al., 2007; Filimon et al., 2009) to characterize their cortical topographical relationships and neural organization. In fact, when we directly performed a group subtraction analysis of a similar nature to localize our same ROIs (RFX GLM, t(7) = 3, p < 0.01, [Execute(HandL + HandR) > Execute(EyeL + EyeR)], cluster threshold correction: 276 mm3), we found very little activation throughout the brain (of which no voxels survived cluster threshold correction) [Note that this particular null result might initially seem to contradict the findings presented in Figure 8, where several regions involved in hand movements (in red) do not overlap with regions involved in eye movements (in yellow); however, it is important to reemphasize that the topographical activation maps in Figure 8 only show significant reach- and saccade-related activation with respect to the Preview phase and not with respect to each other (i.e., hand vs eye movements)]. For the sake of completeness, we also looked for spatially specific responses for reach and saccade execution at the group level using the subtraction approach (RFX GLM, t(7) = 3, p < 0.01, [Execute(HandL) > Execute(HandR)], cluster threshold correction: 269 mm3; [Execute(EyeL) > Execute(EyeR)], cluster threshold correction: 287 mm3). Both contrasts expectedly revealed symmetrical contralateral early visual cortex activity consistent with the visual stimulation created by displacement of the limb (i.e., HandL actions produced an increase in right visual cortex activity) or eye (i.e., EyeL actions produced an increase in left visual cortex activity, presumably due to the rightward visual field shift created by the leftward eye movement). Again, however, similar to the effector-specific contrast performed above (i.e., reaches vs saccades), we found no parieto-frontal cortex activation that survived cluster correction for either contrast. These null findings may of course reflect the reduced statistical power to be expected from a RFX group voxelwise analysis with only eight subjects (minimum group size generally recommended is 12; Desmond and Glover, 2002), a point likely further compounded by the fact that all the single-subject data used for group analysis was left unsmoothed (note that although data smoothing is generally considered beneficial for group voxelwise analysis, its potentially disadvantageous effect on detecting fine-grained multivariate activity patterns with MVPA seems to be a matter of ongoing debate, see Kriegeskorte and Bandettini, 2007; Mur et al., 2009; Kamitani and Sawahata, 2010; Op de Beeck, 2010; for this reason and to allow for a direct comparison of the two analysis methods using the exact same dataset, we have chosen to perform our univariate analyses using the unsmoothed data). Nevertheless, despite the potential sources of small discrepancies with past fMRI findings, we find it important to reemphasize the critical findings from this experiment: despite the absence of signal amplitude differences between different effector- and spatial-movement plans, we are still able to predict in several parieto-frontal areas the chosen effector and spatial location of an upcoming action.
Discussion
Whereas previous fMRI studies have predominantly used multivariate methods for decoding the perceptual neural responses that accompany (and follow from) the presentation of sensory stimuli (Haxby et al., 2001; Kamitani and Tong, 2005; Formisano et al., 2008; Kriegeskorte et al., 2008b; Harrison and Tong, 2009), here we apply MVPA to decode planned target-directed actions, events that have yet to occur. We show in several parieto-frontal brain areas that we cannot only decode the intention to perform a saccade or reach action, but that we can also reveal the underlying spatial and nonspatial nature of the movement plan. Importantly, we demonstrate that accurate decoding does not merely reflect an attention-driven modulation. Consistent with intention versus attention distinctions made from monkey neural activity (Snyder et al., 1997), in most of the areas examined the preparatory signals discriminate whether an eye or arm movement is being planned toward a single spatial target location. Furthermore, we show that this predictive movement information is not revealed from a conventional univariate analysis of signal response amplitudes, a finding that may indicate why previous attempts to identify effector-specific and spatially specific movement plans with fMRI in the human have largely revealed graded or indistinguishable processes.
Beyond applying standard within-trial MVPA analyses to determine which types of planned movements can be decoded in each brain area, we also used cross trial-type decoding analyses that allowed us to further investigate whether: (1) planned reach and eye movements were coded using similar spatial target representations (Space-across-Effector decoding); and/or (2) whether the planned movements were coded using nonspatial, motor-specific signals (Effector-across-Space decoding). Ultimately, the combination of both MVPA approaches (within-trial and cross-trial classification analyses) allowed for the fine-grained differentiation of effector-specific and effector-independent (spatially specific) signals within single parieto-frontal regions, a distinction often limited to the activity profiles of individual parieto-frontal neurons in nonhuman primates. For instance, in the monkey IPS the neural arrangement is highly multiplexed, with many plan-related neurons preferentially responsive to both an effector (eye vs hand) and target direction (e.g., left vs right), whereas others may only carry spatial information (Snyder et al., 1997; Calton et al., 2002; Dickinson et al., 2003). This idiosyncratic arrangement provides a sensible neurophysiological basis for the complex profiles of the effector-specific and effector-independent plan-related activations observed here at the coarse level of distributed parieto-frontal fMRI patterns. Importantly, the pattern analysis approach taken here also allows for a significant theoretical departure from previous human fMRI (and also magnetoencephalography) studies. Nearly all prior investigations have relied on the implicit assumption that movement plans for reaches and saccades are topographically organized with respect to the planned target location (e.g., a target for an eye movement on the right will be encoded by the left hemisphere). This notion emerges from our understanding of the contralateral architecture that governs lower-level cortical structures (e.g., visual cortex, superior colliculus) and has found varying degrees of support in parietal cortex (Sereno et al., 2001; Medendorp et al., 2003; Hagler et al., 2007; Levy et al., 2007; Kagan et al., 2010; Van Der Werf et al., 2010). For several regions, however, there seems to be no good reason to suspect that simply because the resident neurons may encode stimuli in a particular reference frame (e.g., gaze-centered coordinates), the entire region will be topographically organized (e.g., retinotopically) in an ordered configuration along similar dimensions (see also Filimon, 2010). In fact, this assumption critically overlooks several important parieto-frontal properties such as its highly distributed/unstructured organization (Andersen and Buneo, 2002), neurons containing ipsilateral or bilateral response fields (Barash et al., 1991; Ben Hamed et al., 2001), and moreover, the fact that movement planning processes are asymmetrically organized (lateralized) in the human compared to the monkey brain (Kagan et al., 2010; Patel et al., 2010). To further underscore these points, we explicitly examined the topography of planned and executed reaches and saccades using conventional fMRI analyses (see Figs. 8⇑–10) and found, in agreement with previous fMRI investigations containing analyses of a similar nature (Hagler et al., 2007; Levy et al., 2007; Beurze et al., 2009; Filimon et al., 2009), that the cortical topography for reaches, saccades, and spatial locations is highly mixed. This result highlights another main benefit of analyzing distributed activity patterns in human fMRI sensorimotor investigations: assumptions of structurally organized cortical topographies are unnecessary.
Posterior parietal cortex decoding
Although studies in both humans and monkeys have commonly reported a role for superior parietal cortex in reaching (Andersen and Cui, 2009; Vesia et al., 2010), here we show that L- and R-SPOC preparatory activity also predicts upcoming reach directions, a finding that has only been previously reported with neural recordings in monkeys (e.g., Fattori et al., 2009). Whereas we might only loosely speculate on the underlying organization governing the spatial selectivity of reaches in SPOC (see, for instance, Bernier and Grafton, 2010), the decoding profiles found in pIPS and midIPS appear to offer more substantive and well grounded interpretations. In the human, pIPS has been implicated in a vast range of sensorimotor processes including visual–spatial attention (e.g., Szczepanski et al., 2010) and spatial target and effector integration (Beurze et al., 2009). The middle portion of the IPS, in contrast, has been more thoroughly characterized in nonhuman primates (Andersen and Buneo, 2002) of which a prominent feature is its poorly defined functional properties; neurons in the area can encode either reaches or saccades. For instance, reach-related neurons can be found not only in both V6A and MIP but also within aspects of cIPS and LIP, areas proximally positioned on the lateral bank of the IPS (Calton et al., 2002; Chang and Snyder, 2010) more prominently implicated in coding action-relevant 3D visual object features (Sakata et al., 1998) and the targets for saccades (Andersen et al., 1985), respectively. A few human studies to date have attempted to describe the homologous functional locations of these regions, and the pIPS and midIPS decoding results provided here also offer good approximations; L-pIPS, L-midIPS, and R-midIPS all predict the direction of both eye and arm movements and, interestingly, this spatial selectivity is similar enough across both effectors to allow for accurate cross-classification. Although future investigations will require different initial positions of the eye, hand, and target locations to discriminate whether these effector-independent, spatially selective movement plans are with reference to the eye, limb, head/body, or world (e.g., Snyder et al., 1998; Batista et al., 1999; Buneo et al., 2002; Pertzov et al., 2011), the critical finding here is that we are able to localize and decode from human preparatory activity similar spatial target representations for the hand and eye—known to be present in monkeys for quite some time (Andersen et al., 1985; Batista et al., 1999) but difficult to confirm in humans with conventional fMRI analyses.
Frontal cortex decoding
If PPC can be predominantly described to encode reach targets with spatial reference to the eye (Batista et al., 1999; although see Chang and Snyder, 2010), then by comparison, the spatial representations that characterize reach planning in premotor/frontal areas appear more closely linked to the final motor output. A recently emerging view of premotor cortex and of PMd in particular suggests that neurons prepare reaching movements by using a relative position code; single neurons can encode the target relative to the eye (eye-centered), the target relative to the hand (hand-centered), and the eye relative to the hand, as well as combinations of two or all three (Pesaran et al., 2006). Furthermore, it has more recently been shown that the same relative position code can also characterize the planning of saccades in PMd (Pesaran et al., 2010), presumably facilitating eye–hand coordination given that both saccade and reach movements can be planned in a common spatial reference frame (Pesaran et al., 2010). These recent findings in monkeys may be particularly relevant for interpreting the L-PMd/FEF decoding results here; not only are both reach and saccade movement plans spatially tuned to target locations, but we also show that this spatial tuning is similar across effectors (i.e., similar enough to allow significant cross-classification), perhaps indicating a shared relative position code (Pesaran et al., 2006, 2010). An alternative explanation for the successful Space-across-Effector decoding is that we might instead be measuring the coding of a gaze-centered reference frame common to both PMd and FEF neurons (Pesaran et al., 2006; Bruce and Goldberg, 1985; Beurze et al., 2010; Kastner et al., 2007). Future investigations will be required to disentangle these different possibilities. Interestingly, in L-PMv we were able to predict upcoming reach and saccade movement directions, but as an important departure from our PMd/FEF results, this spatial-selectivity was not found to be similar/shared across effectors. This finding may resonate with previous neurophysiological findings suggesting that PMv encodes target locations for saccades and reaches in different spatial reference frames, with eye movements possibly in head- or eye-centered coordinates (Fujii et al., 1998) and hand movements possibly in arm/limb-centered coordinates (Graziano et al., 1994).
Implications
These findings offer new insights into how saccades and reaches are planned in the human brain. We show that it is possible to decode planned reaches and saccades, their intended spatial directions, and, in several cases, characterize both the spatial and nonspatial nature of the movement plan. Importantly, whereas motor cortex signals are increasingly used to decode motor intentions (Andersen et al., 2010), here we highlight several candidate parieto-frontal brain areas where high-level, intention-related activity can be harnessed to operate and control neural prosthetics in movement-impaired human patient populations.
Footnotes
This work was supported by an operating grant from the Canadian Institutes of Health Research (MOP84293) to J.C.C. We are grateful to Larry Snyder, Mark Daley, Lore Thaler and Melvyn Goodale for helpful discussions.
- Correspondence should be addressed to Jason P. Gallivan, Centre for Neuroscience Studies, Queen's University, Kingston, Ontario, Canada K7L 3N6. jasongallivan{at}gmail.com
