Neural Correlates of Online Action Preparation

Participants

A total of 11 individuals (four females; mean age, 26 ± 4) participated in Experiment 1, and 22 individuals (12 females; mean age, 24 ± 4) participated in Experiment 2. Four individuals participated in both experiments. Inclusion criteria required right-handedness and no history of psychiatric or neurological disorders. Participants provided written informed consent to all procedures and data usage before the study started, and all the experimental procedures were approved by the Human Research Ethics Board at Western University.

Apparatus

Finger presses were produced on a right-hand MRI–compatible keyboard with five 10.5 × 2 cm keys. Each key had an indentation to guide fingertip placement. Finger presses were isometric. Forces were measured by transducers (FSG-15N1A; Sensing and Control Honeywell; the dynamic range of 0–25 N; update rate, 5 ms) located beneath the fingertip indentation of each key. To register a key press, the applied force had to exceed the 0.8 N threshold, indicated by a horizontal red line on the top of the screen (Fig. 1B). Five white lines were displayed on a computer screen such that the vertical position of each line was proportional to the force exerted by each finger on the respective key (Fig. 1C). In Experiment 2, the visual stimuli (symbols) were presented in the two vertically aligned boxes with a fixation cross between them (Fig. 2A). The distance between the symbols and fixation was ∼1°.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Task design for Experiment 2. A, Visual display. Participants were instructed to produce the action indicated by the symbol in the bottom box. The content of the top box indicated the next action. Participants were instructed to fixate the cross between the two boxes, such that each symbol (∼1° from fixation) could be identified without shifting gaze. B, Time course of the continuous paradigm. One low-pitch tone (gray note) alternated with two high-pitch tones (black note). Participants had to produce actions indicated in the bottom box synchronously with high-pitch tones with ±200 ms (red intervals). The symbol displayed in the top and bottom box over time is indicated by the top and bottom yellow strip, respectively. In the overlap condition, the second symbol (&) was visible in the top box during the preparation phase of the first response. C, In the nonoverlap SS condition, the second response cue appeared together with the first tone, preventing online preparation. D, In the nonoverlap LS condition, the first response had a long preparation time.

Paced response task (general procedures)

In both experiments, the task required participants to produce responses at a fixed pace. The responses were either single-finger or simultaneous three-finger presses (chord). For each of these, we assigned five arbitrary symbols to the five responses (Fig. 1B). The five chords consisted of different combinations of pressing three of five fingers. The chords (135, 145, 235, 245, and 345) were used with 1 indicating the thumb and 5 the pinkie finger. After learning the symbol-to-action mapping, participants were trained to produce the three-finger presses simultaneously, so the chord could be produced as a single unit. During scanning, the average absolute asynchrony between the time when each finger crossed the force threshold was 16 ms (±2 ms, ± indicating the SEM across participants).

To help participants keep a regular pace, we presented a sequence of high- and low-pitch tones. Similar to the “forced-response” paradigm (Haith et al., 2016), participants had to respond synchronously with the high-pitch tones. The response time was defined as the time point when the summed force across fingers reached its peak.

To provide feedback on the temporal accuracy of the response, we displayed a bar in the lower part of the screen (Fig. 1C). If the response was too early, the bar pointed to the left, and if the response was too late, the bar pointed to the right. The length of the bar indicated the size of the deviation. The acceptable deviation (200 ms) was specified by the box boundaries. If the executed response matched the instructed cue, the bar appeared in green; otherwise, it appeared in red, indicating an error.

Points were awarded for each press and time accuracy according to the following scheme: −2 points in case of timing error (deviation >200 ms); 0 points for pressing any wrong key without timing error; and 1 point in case of a correct response without timing error.

Experiment 1

The purpose of this experiment was to quantify the time necessary to prepare a single finger or chord movement, respectively. At the beginning of the experiment, participants were trained in symbol-to-action mapping. In separate runs for single-finger and chord, we presented a symbol on the screen and asked them to produce the corresponding response. We encouraged them to delay their response until they felt confident that the response was correct. Each run consisted of 60 responses with symbols presented in random order, except for the fact that symbols were never repeated. We continued training until participants achieved an accuracy of above 95% for both single-finger and chord runs.

After training, participants produced a sequence of chords or single-finger presses at a continuous pace of one response per 2 s. A tone was presented each second, alternating between high-pitch and low-pitch sounds (gray and black notes in Fig. 1C, respectively). Participants were instructed to respond simultaneously with high-pitch tones, and the low-pitch tones made the response times temporally more predictable. At a random preparation time of 240–1,750 ms before each high-pitch tone, a symbol appeared on the screen that instructed the required response.

Five single-finger runs were interleaved with five chord runs. Each run consisted of 60 responses. Symbols could occur with equal probability, but no repetitions were allowed. The assignment between symbols and responses was varied across participants (Fig. 1B).

Experiment 2

In Experiment 2, participants again produced a sequence of responses synchronized to a sequence of regularly paced tones. These occurred in triplets, at a pace of one tone per 750 ms. The first tone was low-pitch, and the next two were high-pitch, and participants were instructed to synchronize their responses with the two high-pitch tones. On the screen, symbols were presented in two boxes (Fig. 2A). The lower box contained the cue for the immediately upcoming response, while the upper box informed participants about future responses. Participants had to produce the two responses in three different conditions: in the overlap condition (Fig. 2B), the preparation for the second response overlapped with the execution of the first response. The stimulus for the first symbol appeared 750 ms before the high-pitch tone in the upper box and shifted to the lower box 550 ms before the tone. Simultaneously, the stimulus for the second response appeared in the upper box, allowing the participants a 1,300 ms preparation time for this response. In the nonoverlap conditions, the preparation for the second response did not overlap with the execution of the first response—the stimulus for the second response appeared simultaneously with the tone for the first response. We introduce two variants of the nonoverlap condition: in the nonoverlap short–short (SS, Fig. 2C) condition, both responses had a 750 ms preparation time. This condition only differed from the overlap condition by the earlier presentation of the second stimulus. By comparing the behavioral performance for the second response across conditions, we could therefore test whether the participants used the overlap for online planning.

In the nonoverlap long–short (LS, Fig. 2D) condition, the first response had a 1,300 ms preparation time. This condition was designed to have the same set of preparation times as the overlap condition, averaged across the two responses. Given that the limited temporal resolution for fMRI forced us to average activity across each pair of responses, this condition provided a strong comparison to the overlap condition. If the activity in these two conditions only differed by the relative timing of the preparation and execution processes, the temporally averaged activity should be identical.

To train the participants in the task, we conducted a behavioral session before the scanning session. At the beginning of this training session, participants were familiarized with the symbol-to-action mapping, as done in Experiment 1. We then trained participants on a simpler version of the task, giving them a 1,300 ms preparation time for both responses (the first response was like nonoverlap LS, and the second response was like overlap). Each pair of responses is repeated eight times (with random symbol pairs each time), resulting in the 16 responses forming a single 18.75 s cycle (2.250 × 8 s plus an extra 750 ms at the beginning). Before each cycle started, a 5 s instruction screen displayed either “Single” or “Chord” to indicate the press type to participants. Each training run comprised three cycles of single-finger presses and three cycles of chords. Participants completed four runs for this phase of training.

Subsequently, participants were trained in the six conditions of the experiment (overlap, nonoverlap SS, nonoverlap LS) × (single-finger, chord). Similarly to training, these were presented in cycles of 16 responses. Before each cycle, a 5 s instruction screen was presented, indicating the upcoming press type (but not the overlap condition). Each run included two cycles of each condition (randomly interleaved), totaling 192 responses. Participants completed six runs during training.

On the day after, participants underwent an fMRI session consisting of 10 runs and one anatomical scan. Similar to training, each run consisted of two cycles of each condition, with each cycle preceded by a 5 s instruction screen. The order of conditions was random. Additionally, two periods of 15 s rest, each preceded by a 5 s “fixate” screen, were placed randomly between conditions. Also, periods of 10 s rest were added at the beginning and the end of each functional run. The rest periods allowed for a better estimation of the baseline activation. Each of the 10 functional runs took ∼6 min, and the entire scanning session (including setup and anatomical scan) lasted for ∼100 min.

Behavioral data analysis

We evaluated the state of preparation using two different metrics. To assess whether the subjects had selected the correct finger or chord, we calculated the average accuracy. An action was considered accurate if at the peak force time—when the combined force of all five fingers was at its maximum—the force exerted by each active finger was greater than that exerted by each passive finger.

To focus on action-related processes, we determined how well the selected chord was executed. This analysis was restricted to correct trials only. We quantified the execution quality by evaluating the force trajectory in five-dimensional finger space from response initiation to peak force. To determine the initiation time, we calculated the first derivative of the summed force (across all fingers) and then determined the moment that the rate of force change exceeded 5% of the maximum rate within that trial. If the active fingers were pressed simultaneously and the passive fingers remained stationary, the produced force trajectory would be aligned with the ideal straight-line trajectory (Waters-Metenier et al., 2014). For each action, the ideal force trajectory begins at the finger forces at initiation time and ends at the “mean configuration,” which represents the average force profile of that action at the subject's peak force time within correct trials. Sequential presses of active fingers, involuntary coactivation of passive fingers, pressing the wrong fingers, and initial errors that were later corrected all cause deviation from the ideal straight-line trajectory (Fig. 1D). We therefore quantified the accuracy as the Euclidian norm between the produced force and the projection of the produced force onto the straight-line trajectory. This distance was averaged over all time points from the initiation until the peak time to produce the “mean deviation.”

To provide an estimation of the necessary preparation time in Experiment 1, we calculated the accuracy as a function of the preparation time separately for each participant and for single-finger and chord. We then fitted the data separately for each subject and each press type with the following function:y^n=d−c1+exp(−axn+b)+c, where yn is the predicted accuracy when the preparation time is xn and a, b, c, and d are free parameters determining, respectively, the slope, the shift, minimum accuracy, and maximum accuracy of the function. Parameters were then fitted to the data of the press type separately using MATLAB's fminsearch routine to minimize the mean squared error loss function. The required preparation time was estimated as the x value for which the predicted accuracy reached 80%.

Statistical analyses on the required preparation time, accuracy, and mean deviation were performed using two-tailed paired–sample t tests and a within-subject repeated–measure ANOVA with factor conditions and press types.

Imaging data acquisition

High-field fMRI data were acquired on a 7 T Siemens Magnetom MRI scanner with a 32-channel head coil at Western University. The anatomical T1-weighted scan of each participant was acquired halfway through the scanning session (after the first five functional runs) using a magnetization-prepared rapid gradient echo sequence with an isotropic voxel size of 0.75 mm [field of view, 208 × 157 × 110 mm (A–P, R–L, F–H), encoding direction coronal]. To measure the BOLD responses in human participants, each functional scan (352 volumes) used the following sequence parameters: GRAPPA 3; multiband acceleration factor 2; repetition time (TR), 1.0 s; echo time (TE), 20 ms; flip angle (FA), 30°; 46 slices; and 2.3 mm isotropic voxel size. To estimate and correct for magnetic field inhomogeneities, we also acquired a gradient echo field map (transversal orientation, field of view, 210 × 210 × 160 mm; 64 slices; 2.5 mm thickness; TR, 475 ms; TE, 4.08 ms; FA, 35°).

Preprocessing and first-level analysis

Data analysis was performed using SPM12 (http://www.fil.ion.ucl.ac.uk/spm/) and custom-written MATLAB (MathWorks) routines. Images were corrected for field inhomogeneities and head motion (Hutton et al., 2002). Due to the short TR (1 s), we did not adjust images for the sequence of slice acquisition. The data were high-pass filtered to remove slowly varying trends with a cutoff frequency of 1/128 Hz and coregistered to the individual anatomical scan. No smoothing or normalization to a group template was implemented during preprocessing.

The preprocessed images were analyzed with two different general linear models (GLMs): the first was designed to estimate how much each of the six task conditions (overlap, nonoverlap SS, nonoverlap LS) × (single-finger, chord) activated each voxel in each of the 10 functional runs. In the design matrix, the regressor for each condition consisted of two boxcar functions (1 for each cycle of 16 responses; length, 18.75 s). We also added a single regressor for instruction periods that happened before each cycle or fixation period (2 s length boxcar functions). The estimate of this regressor was not used in further analysis.

We used a second GLM to estimate the pattern corresponding to each of the 10 actions (five single-finger presses and five chords, Fig. 1A) within each run by modeling the first and the second response separately for each action but averaged across all responses and conditions. Each regressor therefore modeled ∼20 events of a particular action occurring (16 responses per sequence × 2 repetitions of each sequence × 3 conditions per press type/five action types). The length of boxcar functions varied depending on the available preparation time. We used 1.8 s for the second response in the overlap condition and the first response in the nonoverlap LS condition (Fig. 2B,D, respectively) and 1.25 s for the rest. The instruction regressor was treated the same as in the first GLM.

The boxcar functions were convolved with an individual-specific hemodynamic response function (HRF). For each participant, we tested which of the 20 HRF functions drawn from the GLMsingle library (https://github.com/cvnlab/GLMsingle/) maximized the proportion of the variance that the model could explain of the time series of voxels in the left primary motor and dorsal premotor cortex (PMd). The selected HRF was then applied to the whole brain. For HRF selection, we treated all conditions as one condition; therefore, this procedure did not bias any subsequent analysis that concerned differences between conditions. Ultimately, the first-level analysis resulted in one activation image (β values) per condition per run. We then calculated the percent signal change for each condition relative to the baseline activation for each voxel for each functional run and averaged it across runs.

Surface-based analysis

Individual subject's cortical surfaces were reconstructed using FreeSurfer (Dale et al., 1999). Individual white–gray matter and pial surfaces were extracted and spherically morphed to match a group template atlas based on the sulcal depth and local surface curvature information (Fischl et al., 1999). Subsequently, surfaces were resampled to a left–right symmetric template (fs_LR32k; Van Essen et al., 2012) included in the connectome workbench distribution (Marcus et al., 2011) using the surfAnalysis toolbox (surf_resliceFS2WB.m, https://github.com/DiedrichsenLab/surfAnalysis). Individual data were then projected onto the group map via the individual surface using the surf_vol2surf.m function in the surfAnalysis toolbox.

Regions of interest (ROIs)

We identified 10 ROIs (Fig. 5A) based on the cortical areas defined in Glasser et al. (2016) to cover the main anatomical areas that exhibited task-related activations in general (Fig. 5B). These ROIs included the supplementary motor area (SMA), the PMd, the ventral premotor cortex (PMv), the primary motor cortex (M1), the primary somatosensory cortex (S1), the anterior and posterior superior parietal lobules (SPLa/SPLp), the MT+ complex and neighboring visual areas (MT+), the ventral stream visual cortex (VSVC), and the early auditory cortex (EAC).

The SMA was defined as the medial aspect of Brodmann area (BA) 6, covering supplementary and cingulate eye field, 6ma, and 6mp. The PMd was located at the junction between the superior frontal and precentral sulci in the lateral aspect of BA 6, covering 6a, 6d, and frontal eye field. The PMv covered 6v, PEF, and 55b. The M1 ROI covered BA 4; cut 2 cm above and below the hand knob area (Yousry et al., 1997) to restrict it to the cortical hand area. The S1 was defined similarly as the hand-related aspect of BA 1, 2, and 3. The superior parietal cortex was divided into an anterior region (SPLa) covering AIP, 7PC, LIPv, and LIPd and a posterior region (SPLp) covering MIP, VIP, and 7PL. The MT+ covered areas in the lateral occipital and posterior temporal cortex, including LO1, LO2, LO3, V3CD, V4t, FST, MT, MST, and PH. The VSVC lies around the ventral aspect of the left hemisphere, covering areas anterior to early visual areas such as FFC, VVC, V8, VMV1, VMV2, VMV3, and PIT. The EAC includes A1, LBelt, MBelt, PBelt, and RI. The ROIs were defined on the group surface and then projected into the individual space via the cortical surface reconstruction of that individual using the Region toolbox (https://github.com/DiedrichsenLab/region). We selected all voxels that lay between the individual pial and white matter surfaces as part of the ROI.

Additionally to the ROI analysis, we also performed a continuous searchlight analysis (Oosterhof et al., 2011). A searchlight was defined for each surface node, encompassing a circular neighborhood region containing 100 voxels. The voxels for each searchlight were found in exactly the same way as for the ROI definition. As a slightly coarser alternative to searchlights, we also defined a regular tessellation of the cortical surface separated into small hexagons (cortical patches) and extracted the functional data in the same way.

Analysis of activation

We calculated the percent signal change for each condition relative to the baseline value for each voxel for each functional run and averaged it across runs. For the ROI analysis, the percent signal change was averaged across all voxels in the predefined regions in the native volume space of each subject. Additionally, for visualization, the volume maps were projected to the surface for each subject and averaged across the group in the Workbench space.

Statistical analyses to assess differences in percent signal change were conducted using two-tailed paired–sample t tests and within-subject repeated–measure ANOVA with factors conditions (overlap, nonoverlap LS) and press types. Statistical tests on the surface were conducted using an uncorrected threshold of p = 0.001, and the familywise error was controlled by calculating the size of the largest superthreshold cluster across the entire cortical surface with estimated smoothness of FWHM 11.4 mm that would be expected by chance (p = 0.05) using Gaussian field theory as implemented in the fmristat package (Worsley et al., 1996).

Dissimilarities between activity patterns for responses

To evaluate which regions displayed cue- or action-specific representation, we calculated cross-validated Mahalanobis dissimilarities between the patterns of estimated activities (β values). We first prewhitened the β values: we estimated the voxel noise covariance matrix from the residuals of the GLM and used optimal shrinkage toward a diagonal noise matrix (Ledoit and Wolf, 2003). We then divided the patterns by the matrix square root of this estimate. Multivariate prewhitening has been found to increase the reliability of dissimilarity estimates (Walther et al., 2016). Next, we calculated the cross-validated Mahalanobis dissimilarities (i.e., the crossnobis dissimilarities; Diedrichsen et al., 2020) between evoked regional patterns of different pairs of actions, separately for five single-finger presses and five chords, resulting in a total of 2 × 10 dissimilarities. To obtain a measure of overall encoding, we averaged these 20 dissimilarities within each cortical surface searchlight area (Oosterhof et al., 2011).

Model-based representational fMRI analysis

While the searchlight analysis tells us from which brain areas we can decode response identity, it does not reveal which specific aspects of the action are represented. We therefore tessellated the area with significantly positive dissimilarities using a discrete set of surface patches and then estimated the contribution of two different representational models within each patch using the pattern component modeling (PCM; Diedrichsen et al., 2011). The two representational components corresponded to a cue and motor representation, respectively. Because we do not a priori know what similarities to predict for the two components, we used the data from a group of subjects with the opposite cue-to-action assignment. We predicted that if a region represented the visual cue, the similarity between patterns should be the same as observed in the other group for the same cues (but different actions). Conversely, if a region represented the action, the pattern similarity should be the same as observed in the other group for the same action (but a different cue). Thus, we specified the two model components for Group 1 from the average data from Group 2 in the same area (and vice versa).

Based on these two components, we then formulated a model family containing all possible combinations of the two representational components (Yokoi and Diedrichsen, 2019). This resulted in four combinations, also containing the “null” model that predicated no differences among any of the activity patterns. We evaluated all four models using a cross-validated leave-one-subject-out scheme because different combination models had different numbers of free parameters. The component weights were fitted to maximize the likelihood of the data of N − 1 subjects. We then evaluated the likelihood of the observed activity patterns for each cortical patch [in volume space, see above, Regions of interest (ROIs)] of the left-out subject N under that model. The resultant cross-validated likelihoods were used as an estimate of model evidence for each of the four models (Diedrichsen et al., 2018). The log Bayes factor BF_m, the difference between the cross-validated log-likelihood of each model and the null model, characterizes the relative evidence for that model.

The log Bayes factor for each model component was calculated as the log of the ratio between averaged likelihood for the models that contained the component (c = 1) versus the averaged likelihood for the models that did not (c = 0 Shen and Ma, 2019):logBFc=log(1Nm:c=1∑m:c=1exp(logBFm)1Nm:c=0∑m:c=0exp(logBFm)), where N_m:c = 1 (N_m:c = 0) denotes the number of models (not) containing the component. Thus, a positive log Bayes factor indicated that there was evidence for the presence of the component. The analysis was performed separately for the two groups of subjects and for single and chord data. As chord and single datasets were independent, we finally summed the log Bayes factors for the corresponding components within each individual. Lastly, we projected the log Bayes factors of each cortical patch onto the vertices associated with that patch to create a cue and an action encoding map.

Statistical analysis of PCM

Within each cortical patch, the final log Bayes factors for cue and action components for each participant were then submitted to a Bayesian group analysis, which estimates the probability that the component is present in a given subject (Stephan et al., 2009; Rosa et al., 2010; spm_BMS function implemented in the SPM 12). The significance was assessed using the protected exceedance probability (PXP)—the posterior probability that a component is present in more than half of the participants. We deemed a model contribution significant when PXP is larger than 0.75.