Abstract
Because the sophistication of tool use is vastly enhanced in humans compared with other species, a rich understanding of its neural substrates requires neuroscientific experiments in humans. Although functional magnetic resonance imaging (fMRI) has enabled many studies of tool-related neural processing, surprisingly few studies have examined real tool use. Rather, because of the many constraints of fMRI, past research has typically used proxies such as pantomiming despite neuropsychological dissociations between pantomimed and real tool use. We compared univariate activation levels, multivariate activation patterns, and functional connectivity when participants used real tools (a plastic knife or fork) to act on a target object (scoring or poking a piece of putty) or pantomimed the same actions with similar movements and timing. During the Execute phase, we found higher activation for real versus pantomimed tool use in sensorimotor regions and the anterior supramarginal gyrus, and higher activation for pantomimed than real tool use in classic tool-selective areas. Although no regions showed significant differences in activation magnitude during the Plan phase, activation patterns differed between real versus pantomimed tool use and motor cortex showed differential functional connectivity. These results reflect important differences between real tool use, a closed-loop process constrained by real consequences, and pantomimed tool use, a symbolic gesture that requires conceptual knowledge of tools but with limited consequences. These results highlight the feasibility and added value of employing natural tool use tasks in functional imaging, inform neuropsychological dissociations, and advance our theoretical understanding of the neural substrates of natural tool use.
SIGNIFICANCE STATEMENT The study of tool use offers unique insights into how the human brain synthesizes perceptual, cognitive, and sensorimotor functions to accomplish a goal. We suggest that the reliance on proxies, such as pantomiming, for real tool use has (1) overestimated the contribution of cognitive networks, because of the indirect, symbolic nature of pantomiming; and (2) underestimated the contribution of sensorimotor networks necessary for predicting and monitoring the consequences of real interactions between hand, tool, and the target object. These results enhance our theoretical understanding of the full range of human tool functions and inform our understanding of neuropsychological dissociations between real and pantomimed tool use.
Introduction
Advances in human culture and technology have been tightly linked to the ability to use complex tools. A rich understanding of how the brain supports complex tool use has come from neuropsychological studies of patients with brain lesions who show impaired tool use and tool-related functions (Goldenberg and Hagmann, 1998; Buxbaum, 2001; Goldenberg et al., 2004; Mahon et al., 2007). A left-lateralized network of cortical areas has been implicated across diverse tool-related tasks studied with neuroimaging (Johnson-Frey, 2004; Lewis, 2006).
Although neuropsychological and neuroimaging studies have provided insight into tool processing, the study of the actual use of tools is limited. For technical reasons, most functional magnetic resonance imaging (fMRI) studies on “tool use” have not actually used real tools nor real actions, with very few exceptions (Gallivan et al., 2013; Brandi et al., 2014). Instead of real tool use, fMRI investigations have employed tool recognition tasks, naming tasks, or action tasks that simulate tool use (Lewis, 2006; Ishibashi et al., 2016; Lesourd et al., 2021). Growing evidence suggests that behavior and brain responses differ between the real world and common proxies (Króliczak et al., 2007; Freud et al., 2018; Holler et al., 2020; Snow and Culham, 2021). Moreover, proxies for real tool use lack aspects of real tool use that could affect brain responses (Table 1). One common proxy for real tool use in neuroimaging is pantomimed tool use, in which the participant demonstrates tool use without a tool in hand (Moll et al., 2000; Choi et al., 2001; Q. Chen et al., 2017). However, neuropsychological studies of patients have reported dissociations between pantomimed and real tool use. Specifically, patients who exhibit deficits in pantomiming tool use [ideomotor apraxia (IMA)] usually perform better during real than pantomimed tool use (Goldenberg and Hagmann, 1998; Hermsdörfer et al., 2012), although one patient showed the converse effect (Motomura and Yamadori, 1994).
Properties of different tool use tasks that differ in various properties
The few neuroimaging studies that have employed real or quasi-real tool use have partially corroborated the neuropsychological evidence for a difference from pantomimed tool use (Moll et al., 2000; Hermsdörfer et al., 2007; Higuchi et al., 2007; Imazu et al., 2007; Lausberg et al., 2015). Specifically, numerous brain areas showed higher activation for real than pantomimed tool use though few areas have shown higher activation for pantomimed than real tool use. However, given that even putatively “real” tool use in these studies involved artificialities, these studies may have underestimated the activation differences between pantomimed versus real tool use. There are numerous reasons to expect genuinely real tool use to differ from proxies such as pantomiming. Real tool use is a “closed-loop” process (Wolpert and Flanagan, 2001) involving direct physical interaction between the hand, tool, and target, with sensory feedback. In contrast, pantomimed tool use is a symbolic process that often serves a communicative function and relies heavily on cognitive and semantic knowledge. As Goldenberg et al. (2007) stated pointedly, “the pantomime of tool brushing does not clean your teeth.”
Here, we used advanced neuroimaging methods to revisit whether and how real and pantomimed tool use differ in their neural substrates. First, we developed a paradigm to make real tool use as realistic as possible under the constraints of fMRI (Table 1; Fig. 1). Second, whereas past studies only compared the activation for real versus pantomimed tool use during action execution, we also investigated the planning period before each action. Third, we applied advanced methods, including cortex-based alignment to enable clearer delineation of brain regions implicated in tool use, multivoxel pattern analysis (MVPA; Norman et al., 2006) to reveal differences in activation patterns, and psychophysiological interactions (PPIs; Friston et al., 1997; O'Reilly et al., 2012), to reveal differences in functional connectivity.
Experimental setup. A, Participants performed two types of tasks, Real and Pantomime tool use, using two different tools, a fork or a knife. At the beginning of each trial (leftmost panel of each film strip), the participant began with the hand resting at the home position while viewing the tool and a slab of red putty on a white plate. In the Real condition, the participant grasped the fork or knife and made a poking action or a slicing action on the putty, respectively. In the Pantomime condition, the participant pretended to grasp the tool above it and then pretended to perform the action above the putty. B, Participants lay with the head and coils inclined to allow for direct viewing (without mirrors) toward a fixation LED (yellow star) above the workspace of the hand. C, An event-related average time course shows the phases of each trial and the activation from a sample area. Each trial began with a View phase (6 s) where the participant saw the scene before the Task instruction (“Real” or “Pantomime”) was provided auditorily and participants could begin anticipating the action during the Plan phase (12 s). Once a beep was heard, participants performed the instructed action in the Execute phase (4 s) with full visual feedback before the light was extinguished for an intertrial interval (ITI; 12 s). Many areas such as this showed a weak response to the visual preview of the object, followed by increased activation during plan and a robust response following action execution (with the expected hemodynamic lag of 4–6 s).
Schematic diagram of multivariate pattern analysis (MVPA). For each patch of interest (POI), the β values were calculated for each vertex on the cortex-aligned surface. The β values in each condition and each run made up a multivariate pattern that was used as features for the pattern classification analysis. For Action-type decoding, data from knife and fork use were collapsed. A linear support vector machine (SVM) classifier was trained using the pattern from N−1 runs and was then used to predict the Action type of the condition in the remaining run. A leave-one-run-out cross-validation procedure was employed. For the Cross-tool Action type decoding, the training was performed with one kind of tool (knife or fork), but the test was performed on the other tool (fork or knife). Similar procedures were performed for Tool type decoding.
We predicted that the use of a more genuine “real tool use” condition, along with the enhanced sensitivity of advanced contemporary methods, would reveal clearer and stronger differences between pantomimed and real conditions (especially for the pantomimed > real contrast) during action planning as well as execution.
Materials and Methods
Participants
Thirteen right-handed participants (seven males) with an age range of 21–30 years took part in this experiment. Because setup time was considerable, we tested fewer participants than typical but acquired substantially more data from each than is typical (80+ min of functional data). Moreover, cortex-based alignment considerably improves statistical power by improving the overlap in activation foci across individuals by up to ∼100% (Frost and Goebel, 2012). All participants were right-handed according to the Edinburgh Handedness Inventory (Oldfield, 1971) and had vision that was normal or corrected-to-normal (with contact lenses). The procedures for this study were reviewed and approved by Western University's Health Sciences Research Ethics Board. Participants were compensated for their time.
Tasks
Blood oxygenation level-dependent (BOLD) brain activity was measured using functional magnetic resonance imaging at 3 Tesla during real and pantomimed tool use with two different tools. As shown in Figure 1A,B, participants directly viewed one of two Tool types, a plastic knife or fork, beside a target object, a flattened piece of putty on a plate, placed atop a black wooden platform over the hips. Two Action types were performed with the right hand: (1) during real tool use participants grasped a tool, performed an action on a target object, dropped the tool to discard it, and returned the hand to the starting position; and (2) during pantomimed tool use participants pretended to use the tool by making the same motions as with the tool in hand. During pantomimed tool use, both the tool and the target object remained in view but the participant did not interact with them. This ensured that any differences between conditions cannot result from confounds because of the presence/absence of the tool or target. When using the fork, the participants gently poked the putty repeatedly for 4 s. When using the knife, participants scored the putty for 4 s. The poking and scoring motions were chosen to restrict the amplitude of arm movements, particularly of the proximal musculature (shoulder and upper arm), which is more likely to yield fMRI artifacts than movements of the distal musculature (wrist and fingers; Barry et al., 2010). In total, there were four movement types, resulting in four conditions: Real Fork, Pantomime Fork, Real Knife, and Pantomime Knife actions. Thus, the design for this study was a factorial 2 × 2 with Action type (Real vs Pantomime) and Tool type (Fork vs Knife) as factors. Importantly, the four types of actions were matched as closely as possible for duration (4 s) and kinematics. Participants were asked to practice each of the actions before the scan to ensure the actions were properly executed. In addition to this, experimenters monitored each movement throughout the experiment.
Experimental setup
Participants lay supine with the head tilted ∼20° from horizontal such that the workspace was directly visible without mirrors (Fig. 1B). The head rested on the bottom half (six channels) of a 12-channel head coil, which, like the head, was tilted ∼20°. A four-channel flex coil was suspended above the head to increase signal-noise ratio over frontal regions without occluding the participant's view. A strap was placed across the participant's chest and upper arms to minimize shoulder motion that might translate into head motion. Arm movements were performed along an arc using the elbow as a pivot point such that the lower arm and wrist moved with minimal motion of the upper arm and shoulder. The participant wore headphones to hear auditory cues.
Procedure and design
Individual trials lasted 34 s each and consisted of three phases followed by an intertrial interval (ITI): (1) a View phase, (2) a Plan phase, and (3) an Execute phase (Fig. 1C). Participants were instructed to fixate on a central light for the duration of experimental runs. The beginning of the trial was signaled by the illumination of the workspace, which revealed the target object and the tool. This View phase lasted for 6 s, during which the participant saw the tool but was unaware of the upcoming task. After this phase, an auditory cue signaled the trial type with a voice stating “real” for the actual use condition or “fake” for the pantomimed use condition. The word “fake,” rather than “pantomime,” was selected as “fake” and “real” as they have an equal number of syllables. Once the Action type was cued, there was a 12-s. Plan phase in which participants prepared the upcoming movement. This long Plan phase enabled clear distinctions between the Plan and Execute phases despite the sluggishness of the hemodynamic response. A beep signaled the beginning of the Execute phase, in which participants either performed a real or pantomimed action over 4 s. Within this period, the tool was grasped, moved toward the target object, and the action executed repeatedly until the illumination light shut off. Then the participants discarded the tool gently by dropping it to their side and returned their hand to the start position. A 12-s ITI followed the Execute phase. During this time, the participant remained in darkness and maintained fixation while a new tool was placed on the platform by the experimenter. Each of the four trial types occurred four times per experimental run, and each run contained 16 trials. A 30-s resting baseline in darkness occurred at the beginning and end of each run. In total, each run lasted just over 10 min (604 s = 16 trials × 34 s/trial + 30-s initial baseline + 30-s final baseline). Each subject participated in at least 8 runs (range 8–10 runs), yielding at least 32 trials per condition.
To monitor performance and exclude error trials, actions were recorded using an MR-compatible infrared-sensitive camera (MRC Systems GmbH) that was positioned to record the participants' movements during functional runs.
MRI data acquisition
Functional and anatomic data were acquired on a 3-Tesla Siemens MAGNETOM TIM Trio MRI scanner. Anatomical images were collected using a T1-weighted ADNI MPRAGE sequence (time to repetition, TR = 2300 ms; time to echo, TE = 30 ms; flip angle, FA = 9°, field of view, FOV =192 × 240 × 256 mm; matrix size = 192 × 240 × 256; 1-mm isotropic voxels). Functional volumes were collected using a T2*-weighted single-shot gradient-echo echoplanar imaging (EPI) sequence (TR = 2000 ms; TE = 30 ms; FA = 90°; FOV = 340 × 240 × 240 mm; matrix size = 32 × 80 × 80; 3-mm isotropic voxels; acceleration factor [integrated parallel acquisition technologies, iPAT = 2 with generalized auto-calibrating partially parallel acquisitions (GRAPPA) reconstruction]. Each volume comprised 34 contiguous oblique slices (with no gap) acquired at a ∼30° caudal tilt with respect to the anterior-to-posterior commissure (AC-PC) plane, providing near-whole brain coverage.
MRI data preprocessing
Preprocessing and analyses were performed using BrainVoyager QX (Brain Innovation, version 2.4.2). Preprocessing of the functional volumes included: slice scan-time correction, 3D motion correction (such that each volume was aligned to the volume of the functional scan closest in time to the anatomic scan), high-pass temporal filtering (two cycles per run). Runs that had abrupt movements (>1 mm) or translation (>1° rotation) were removed from analysis (one run was removed for one participant, two runs were removed for a second participant). Functional and anatomic scans were coregistered and transformed into AC-PC space and then Talairach space (Talairach and Tournoux, 1988). In addition, a segmentation of the gray matter-white matter boundary was done to generate a cortical surface for each individual subject. We then used cortex-based alignment (CBA; Fischl et al., 1999; Goebel et al., 2006) to transform each subject's cortical surface into a dynamically aligned average surface based on cortical curvature (i.e., sulci and gyri). Because CBA aligns sulci and gyri rather than arbitrary landmarks (as in Talairach averaging), it substantially improves anatomic overlap between subjects and therefore the statistical power of the analysis (Fischl et al., 1999; Frost and Goebel, 2012). Functional data were resampled to create mesh time courses, surface maps, and patches of interest (POIs).
General linear model analysis
Data were analyzed using a group-level random-effects (RFX) general linear model (GLM). Predictors for each combination of the four conditions (Real Fork, Real Knife, Pantomime Fork, Pantomime Knife) and three phases (Preview, Plan, Execute) were generated by convolving square-wave functions with BrainVoyager's default two-γ hemodynamic response function (HRF). If an error occurred during the Execute phase of the trial (such as fumbling or dropping of a tool), the trial was excluded from analysis and predictors of no interest were included for each of the three phases of that trial (∼3.75% of trials were excluded on average).
Because the same patterns of results were observed when fork or knife was used as a tool (Figs. 3 and 4), we combined the data for both tools. Areas implicated more strongly in real tool use during the execution phase were identified by the conjunction of (Real Execute > Pantomime Execute) AND (Real Execute > Real View); areas implicated more strongly in pantomimed tool use during the Execute phase were identified by a conjunction of (Pantomime Execute > Real Execute) AND (Pantomime Execute > Pantomime View). We also performed similar conjunctions for the Plan phase, i.e., (Real Plan > Pantomime Plan) AND (Real Plan > Real View) and (Pantomime Plan > Real Plan) AND (Pantomime Plan > Pantomime View) but no significant regions were revealed.
The statistical analyses were performed on the inflated surface of the cortex. Only activation foci that remained significant after an false discovery rate (FDR) correction (q < 0.05) were reported. For the activation in subcortical structures, to maintain consistency, we used the same corrected t value as that used in the surface of cortex.
Movement-related artifacts are common in fMRI studies of actions; moreover, because body movement can involve changes to the magnetic field even if the head remains perfectly still, motion correction algorithms are not a panacea (Culham, 2006; Barry et al., 2010; Singhal et al., 2013). These artifacts can sometimes be observed as spikes around the time of action, even in group data (Singhal et al., 2013) or as noisy time courses. To examine the quality of the fMRI signal, time courses were extracted from POIs at the group level for the contrast of Real versus Pantomimed tool-use execution. These time courses also allow a qualitative assessment of activation levels during Plan and Execute phases; though our conclusions are based on quantitative (i.e., statistical) testing. To extract time courses from regions that were isolated [e.g., dorsal premotor cortex (PMd)], all significant cortical vertices or subcortical voxels within the region were included. For regions that were part of a larger blob of activation [e.g., primary somatosensory cortex (S1), within the swathe of activation along the postcentral sulcus], POI selection was constrained by size and sulcal locations (e.g., S1 was selected along the posterior bank of the central sulcus hand knob). The raw time course from each patch was then converted to percent signal change using the period 3 s before the onset of each trial as baseline. Time courses were averaged across trials and participants (Figs. 3 and 4). Because of the POI selection criteria, the observation that the time courses differ during action execution is expected. Nevertheless, the time courses illustrate how activation unfolds through all phases of the trial and to demonstrate that data quality were high, without any evidence of motion-related artifacts (Figs. 3–5).
Multivoxel pattern analysis (MVPA)
In addition to the univariate analysis above, we also performed multivariate analysis [multivoxel pattern analysis (MVPA)] using classifiers to decode the Action type and Tool type at the Plan and Execute phases for each POI (Fig. 2).
The multivariate activation pattern of each POI consisted of the β values of all vertices in that POI for each condition in each run obtained in the above GLM analysis. Therefore, for each POI, one pattern was obtained for each condition in each run. The fitcsvm function in MATLAB (The MathWorks; https://ww2.mathworks.cn/) was used to train a linear support vector machine (SVM) model (Chang and Lin, 2011) using two-class (binary) classification. A leave-one-run-out cross-validation procedure was adopted. Specifically, to decode Action type, data from N−1 runs were used to train a classifier that could classify Real and Pantomimed actions, and the remained one run was used as a test dataset to predict whether the test data in that run were Real or Pantomimed action. This procedure was repeated N times (once per run) and the decoding accuracy was calculated as the percent of correct predictions for each participant. Similar procedures were performed for the decoding of Tool type.
To assess the generalization of decoding performance, we also performed cross-tool decoding for Action type (i.e., train a classifier to decode Real and Pantomimed Knife use, but test the classifier to classify Real and Pantomimed Fork use, or vice versa; the results were averaged) and cross-action decoding for Tool type (i.e., train classifier to decode Fork and Knife in Real use, but test the classifier to classify Fork and Knife in Pantomimed use, or vice versa); t tests were performed to examine whether the decoding accuracy was significantly different from the chance level (i.e., 50%) at group level. FDR correction was performed to correct for multiple comparisons.
Psychophysiological interaction analysis
Psychophysiological interaction (PPI) analysis (Friston et al., 1997; O'Reilly et al., 2012) was performed to examine whether functional connectivity between brain areas was modulated by the task (Pantomimed vs Real tool use) at each phase (Plan or Execute). PPI searches for significant task-specific changes in connectivity between a predefined seed area and the other areas in the whole brain based on correlations that are above and beyond task-related and seed-region based fluctuations. We selected left primary motor cortex (M1) as the seed region because of its role as the fundamental cortical region involved in motor output for both tasks. Specifically, M1 was strongly activated in both (Real Execute > Real View) and (Pantomime Execute > Pantomime View) contrasts (Fig. 3A–C). This allowed us to examine whether the regions and networks influencing motor output differed between real and pantomimed tool use.
The PPI analysis was performed following the standard procedures described in the SPM8.0 manual (http://www.fil.ion.ucl.ac.uk/spm/doc/spm8_manual.pdf). To perform PPI analysis, we re-preprocessed the images and performed GLM analyses on volumetric data using SPM8. The preprocessing included slice-time correction, motion correction, normalization into the standardized Montreal Neurologic Institute (MNI) space, and spatial smoothing (6 mm full-width half-maximum Gaussian filter). The same predictors [i.e., predictors for each combination of the four conditions (Real Fork, Real Knife, Pantomime Fork, Pantomime Knife) and three phases (View, Plan, Execute)] were included to perform the GLM analysis with SPM8. Note: at the View phase, the stimulus was identical for real and pantomimed trials and, therefore, were grouped as Fork View and Knife View conditions for analysis. As a result, there were 10 conditions which corresponds to 10 predictors. We checked the effect of each contrast to ensure that the statistical maps appeared qualitatively similar despite differences in the processing pipelines between SPM versus BrainVoyager (especially volume-based alignment and MNI coordinates in SPM vs cortex-based alignment and Talairach space in BrainVoyager). Indeed, the outputs were largely in agreement with only slight differences in statistical significance levels.
Left M1 was defined for each participant as a sphere centered on the most significantly activated voxel revealed by the (Real Execute > Pantomime Execute) contrast (group level, p < 0.001, uncorrected) that is near the characteristic “hand knob” on the precentral gyrus (Yousry et al., 1997). Because the position of the “hand knob” is slightly different across participants, the position of left M1 defined for each participant was also different. The radius of the sphere was 6 mm for left M1 to reduce the inclusion of voxels in primary somatosensory cortex (S1) and dorsal premotor cortex (PMd).
Here, we were interested in the contrast between Real and Pantomime tool use in the Plan phase and in the Execute phase. Therefore, PPI analysis was performed two times, one for plan and one for execute for each of the two task contrasts ([Real > Pantomime] or [Pantomime > Real]).
PPI analyses used a GLM with three regressors: (1) the physiological term = a seed-area time course (the first eigenvariate of the blood oxygenation level-dependent (BOLD) time series extracted for that seed region); (2) the psychological term = the paradigm time course weighting contrasting conditions (e.g., Real Plan > Pantomime Plan based on +1 for Real Plan, −1 for Pantomime plan and 0 for other time points); and (3) the psychophysiological interaction term created by the element-by-element-product of (1) and (2). Given that the physiological term was affected by the BOLD response, whereas the psychological term was given in real time, the physiological term and the dependent variable (i.e., the signal of each voxel in the brain) were hemodynamic response function (HRF) deconvolved before being put into the GLM analysis. The interaction term was the regressor of interest to address whether connectivity with the seed area was higher in one condition than another; however, the other two terms were included to ensure the interaction was not driven solely by simple interarea correlations or by simple task effects. The β weights for the PPI term were estimated for each voxel in the whole brain for each participant, and were then subjected to a second-level RFX analysis to evaluate group differences (one sample t test). Only the clusters that were significant after FDR (q < 0.05) correction were reported.
Note that PPI may be believed to be more difficult to use with event-related designs than block designs for two reasons (O'Reilly et al., 2012). First, the shape of the hemodynamic response function (HRF) is more important for event-related designs. Second, event-related designs tend to have smaller effect sizes than block designs because of the shorter stimulus presentation time or task response time. Our design, however, was more like a block design because of its long “event” time (a Plan phase for 12 s and an Execute phase of 4 s) and a long intertrial interval (12 s).
Results
General linear model results
During the Execute phase, contrasts between (Real Execute vs Real View; Fig. 3A) and (Pantomime Execute vs Pantomime View; Fig. 3B) showed a high degree of concordance (Fig. 3C). Areas of common activation on the lateral cortical surface included the central sulcus (including M1), postcentral sulcus [including S1 and the anterior intraparietal sulcus (aIPS)], precentral sulcus [including dorsal and ventral premotor cortex (PMd and PMv)], insula and parietal operculum, anterior middle prefrontal gyrus (aMFG), inferior parietal lobule (IPL, including anterior supramarginal gyrus, aSMG), and lateral occipitotemporal cortex (LOTC). Areas of common activation on the medial cortical surface include occipital cortex (visual areas), the superior parieto-occipital cortex (SPOC) and a large swath of secondary motor areas [including the presupplementary motor area (pre-SMA), supplementary motor area (SMA), and cingulate zones]. Although activation was typically bilateral (except M1), it was stronger in the left hemisphere for motor, premotor, and parietal cortex and stronger in the right hemisphere for medial regions.
Similarity and overlap of Real Use and Pantomime Use activation in the Execute phase. Activation is shown for lateral and medial inflated views of the two cerebral hemispheres analyzed with cortex-based alignment. A, Activation during the Execute > View phases for Real Use. B, Activation during the Execute > View phases for Pantomime Use. C, Overlap of Activation for Real Use (> Real view) and Pantomime Use (> Pantomime View). Overlap between Real and Pantomime Use appears in a spectrum between purple (relatively low significance) and lime green (relatively high significance). D, Blue-green areas show higher activation for Real Execute than Pantomime Execute (and also higher activation for Real Execute than Real View). Orange-yellow areas show higher activation for Pantomime Execute than Real Execute (and also higher activation for Pantomime Execute than Pantomime View).
More importantly, to directly contrast real and pantomimed tool use during the Execute phase (while restricting activation to regions activated by action execution), the following conjunctions were performed: [(Real Execute > Pantomime Execute) AND (Real Execute > Real View)] and [(Pantomime Execute > Real Execute) AND (Pantomime Execute > Pantomime View)]. Note that although we were primarily interested in differences between Real Execute and Pantomime Execute, the inclusion of a contrast against the View condition in a conjunction ensured that we were examining only differences in positive activation (by eliminating differences in areas that typically show less activation during tasks than baseline periods, namely areas in the default mode network).
As shown in Figure 3D (with the left hemisphere shown in higher resolution in Fig. 4), stronger activation for Real Execute (vs Pantomime Execute) was observed in a large cluster along the central and postcentral sulci bilaterally, bilateral PMd, left anterior superior parietal lobule (aSPL), and bilateral insula/Sylvian fissure as well as secondary motor areas of the medial frontal cortex (in the bilateral cingulate sulcus, right cingulate gyrus and right SMA). The coordinates of these brain regions in Talairach space are listed in Table 2. The coordinates of these areas in MNI space were determined by converting the coordinates in Talairach space with the MNI 2 Talairach Converter (https://bioimagesuiteweb.github.io/webapp/mni2tal.html). The time courses of activation are shown for areas in the left hemisphere (Fig. 4). Note that the regions were selected based on activation differences between real and pantomimed tool use; as such, these differences seen in the time courses are expected (as the analysis is nonindependent. Nevertheless, the time courses show that the slow event-related design yielded reliable signals, without evidence of motion artifacts at the time of action (Culham, 2006; Barry et al., 2010), in which signals for several areas (Left PMd, SPL, M1, and S1) ramped up through the Plan phase, followed by a large motor response. The time courses for the fork (dashed lines) and knife (solid lines) largely overlapped, indicating high consistency and justifying our choice to collapse analyses across the two tools.
Activation time courses for left-hemisphere areas that were more active for Real Execute than Pantomime Execute (and also more active for Real Execute than Real View). Although the activation differences between Real and Pantomime Execute are expected based on the contrast used to define the regions, the time courses are presented to illustrate the magnitude activation in the three phases, the ramping up of the activation during the Plan phase for some regions, the similarity in activation levels between fork and knife use, and the absence of movement of movement-related artifacts (Culham, 2006; Barry et al., 2010). Time courses for regions of the right hemisphere were similar.
Stereotaxic coordinates of brain areas showing activation differences for (Real Execute > Pantomime Execute) and (Real Execute > Real View)
Stronger activation for Pantomime Execute (vs Real Execute) was observed in “association cortex” of the frontal, parietal and temporal lobes (Fig. 3D, with the left hemisphere shown in higher resolution in Fig. 5). Bilateral activation (stronger in the left hemisphere) was observed in PMv, the intraparietal sulcus (IPS), and cuneus. Activation of only the left hemisphere was observed in the posterior middle temporal gyrus or superior temporal sulcus (pMTG/STS), supramarginal gyrus (SMG), the anterior middle frontal gyrus (aMFG), and the superior frontal gyrus (SFG). In most regions, time courses revealed weak signals during the Plan Phase, with larger signals related to the Execute phase. The cuneus showed relatively high activation throughout the View, Plan and Execute phases, consistent with the properties of visual cortex. Once again, time courses appeared free of motion artifacts and consistent between actions with the knife and fork. Talairach coordinates (as well as MNI coordinates) of these areas are listed in Table 3.
Activation time courses for left-hemisphere areas that were more active for Pantomime Execute than Real Execute (and also more active for Pantomime Execute than Pantomime View). Time courses for regions of the right hemisphere were similar.
Stereotaxic coordinates of brain areas showing activation differences for (Pantomime Execute > Real Execute) and (Pantomime Execute > Pantomime View)
Consistent with previous studies (Imamizu et al., 2003; Obayashi et al., 2003; Rumiati et al., 2004; Hermsdörfer et al., 2007; Higuchi et al., 2007), activation differences between real and pantomimed tool use (and vice versa) were also observed in subcortical structures (Fig. 6). A preference for Real Execute (vs Pantomime Execute) was observed in the bilateral thalamus as well as in the tegmentum of the midbrain (which based on stereotaxic coordinates may correspond to the left substantia nigra and right red nucleus). Although there is considerable uncertainty in localizing subcortical structures based on stereotaxic averaging, the event-related time courses all indicate large Execute responses and weak or no Plan responses, consistent with motor and/or somatosensory roles. Within the cerebellum, task preferences were also observed with bilateral foci in the anterior-medial parts showing a preference for Real Execute (vs Pantomime Execute) and bilateral foci in the posterior-lateral parts showing a preference for Pantomime Execute (vs Real Execute). All cerebellar foci showed activation during Execute but not Plan phases (Fig. 6, time courses). The Talairach (original) and MNI (converted) coordinates for subcortical activation are listed in Tables 2 and 3.
Subcortical and cerebellar activation. A, Activation in thalamus and tegmentum (based on stereotaxic coordinates). Blue/green represents greater activation for the execution of Real Use than both the execution of Pantomime and the View phase. Event-related average profiles present time courses for these areas. No areas showed greater activation for the execution of Pantomime than Real Use. B, Activation in cerebellum. Contrasts shown represent activation greater for the execute phase of either the Real Use condition (blue/green) or the Pantomime Use (orange/yellow) with respect to the other condition as well as activation related to the View phase. Event-related average time courses are shown for areas active at p < 0.015.
We also searched for differential activation between Real and Pantomime conditions in the Plan phase [(Real Plan > Pantomime Plan) AND (Real Plan > Real View)], as well as [(Pantomime Plan > Real Plan) AND (Pantomime Plan > Pantomime View)]. These contrasts did not reveal any foci that survived the false discovery rate (FDR) correction for multiple comparisons.
Overall, these results reflect important differences between real tool use and pantomimed tool use in the Execute phase. Real tool use evoked more activation of motor and somatosensory regions. In contrast, Pantomime tool use evoked activation in higher-order sensorimotor regions and association cortex, consistent the more abstract cognitive nature of the task.
MVPA decoding results
Although the univariate analysis did not reveal any differences between Real and Pantomimed tool use during the Plan phase, MVPA can provide a greater sensitivity, as observed in other cases where multivariate coding was found despite no differences in activation levels (Gallivan et al., 2011). Here, we further examined whether the Real and Pantomimed tool use differed at a multivariate pattern level at the Plan phase, as well as the Execute phase, using decoding and cross decoding approaches.
Multivoxel pattern analysis was performed for brain areas (POIs) in the left hemisphere that showed differential activation between Real and Pantomimed tool use at the Execute phase (Figs. 4 and 5). First, we found that consistent with the results of the univariate analysis, during the Execute phase, Action type could be decoded from the activation patterns of all POIs (all accuracy > 50%; FDR corrected, all q < 0.05; Fig. 7A, right) when the data from knife and fork were combined. These results can be anticipated given that the POIs were defined based on the contrast between Real and Pantomimed tool use at the Execute phase. Moreover, in all these POIs, the Action type decoding in one tool transferred to the other tool (Action type decoding across tools; FDR corrected for the number of tests, all q < 0.05; Fig. 7B, right). This suggests that the differences in neural processing between real and pantomimed actions are not tied to the specific kinematics of a tool; rather, they are related to more general factors of the two tasks.
MVPA results. A, Decoding accuracy of Action types (real vs pantomimed) at the Plan and Execute phases. Data from knife and fork tool use were combined. B, Decoding accuracy of Action types cross tools at the Plan and Execute phases. For cross-tool Action type decoding, the train data set was from one tool (knife or fork) and test data set was from the other (fork or knife), and the results were averaged. C, Decoding accuracy of Tool types (knife vs fork). Data from real and pantomimed tool use was combined. The decoding accuracies were compared with chance level (50%). FDR correction was performed to correct for multiple comparisons. * indicates significant after FDR correction (q < 0.05). Error bars indicate ±1 SE.
Interestingly, even during the Plan phase, Action type could also be decoded from the activation patterns of left precuneus, left PMd, and left S1. This was the case when the data from knife and fork were combined (Fig. 7A, left), and when training with one tool and testing with the other (i.e., cross-tool decoding; Fig. 7B, left). This is particularly notable considering that these areas did not show an overall activation difference between real and pantomimed tool use at the Plan phase, consistent with the enhanced sensitivity of multivariate analyses.
Tool type could not be decoded in any of the POIs. The absence of a tool effect is somewhat surprising given that two recent papers have reported multivariate pattern differences between tools (Q. Chen et al., 2016; Malfatti and Turella, 2021); however, those studies used combinations of tools that were much more different in their functionality, kinematics, and semantics than the knife and fork used here. Specifically, in our study, both the knife and fork were white and plastic. The target was exactly the same (i.e., a flattened piece of putty on a plate). When using the fork, the participants gently poked the putty repeatedly for 4 s. When using the knife, participants scored the putty for 4 s. Therefore, when contrasting knife and fork tool use, the difference was essentially between poking and scoring, which were similar in kinematics.
We also did representational similarity analysis (RSA) in which we calculated the correlation of the dissimilarity matrix of the neural activation model with the dissimilarity matrices of the Action type model and Tool type model. The results were similar with the MVPA decoding results.
Psychophysiological interaction (PPI) results
In addition to univariate and multivariate activation analyses, we also examined whether or not the connectivity between areas were different for real and pantomimed tool use at the Plan and Execute phases. The left primary motor cortex (M1) was selected as the seed region because of its role as the fundamental cortical region involved in motor output. M1 was strongly activated in both (Real Execute > Real View) and (Pantomime Execute > Pantomime View) contrasts (Fig. 3A–C), although the activation was stronger in the (Real Execute > Real View) contrast. We used PPI to investigate how connectivity between left M1, the major source of cortical motor output, changed based on the Action type during the Plan and Execute phases of the actions.
The contrast of (Real Plan > Pantomime Plan) revealed stronger connectivity between left M1 and bilateral S1, bilateral insula, and right anterior-medial cerebellum (Fig. 8A). In comparison, the contrast of (Real Execute > Pantomime Execute) revealed stronger connectivity between left M1 and left S1, left PMd and bilateral anterior supramarginal gyrus (aSMG) and right anterior-medial cerebellum (Fig. 8B). The coordinates of these areas are listed in Table 4.
Stereotaxic coordinates of brain areas showing connectivity differences with left M1 between real tool use and pantomimed tool use at the plan and execute phases
Results of PPI analyses with of left M1 as a seed area. A, Regions shown have significantly higher connectivity with left M1 during Real Plan than Pantomime Plan. B, Regions shown have significantly higher connectivity with left M1 during Real Execute than Pantomime Execute. C, Regions shown have significantly higher connectivity with left M1 during Pantomime Plan than Real Plan. D, Regions shown have significantly higher connectivity with left M1 during Pantomime Execute than Real Execute. Only regions that are significant after FDR correction (q < 0.05) are shown. The results were visualized with the BrainNet Viewer (Xia et al., 2013; http://www.nitrc.org/projects/bnv/).
The contrast of (Pantomime Plan > Real Plan) revealed stronger connectivity between left M1 and bilateral MTG, bilateral angular gyrus (AG), left SMG, and multiple frontal foci, including left premotor cortex and the aMFG and posterior-lateral cerebellum (Fig. 8C). In comparison, the contrast of (Pantomime Execute > Real Execute) revealed stronger connectivity between left M1 and bilateral superior temporal gyrus (STG), bilateral LOTC, bilateral caudal intraparietal sulcus (cIPS) and paracentral lobule, bilateral PMv, as well as left inferior frontal gyrus (IFG) and left medial fusiform gyrus (mFG) and posterior-lateral cerebellum (Fig. 8D). The coordinates of these areas are also listed in Table 4.
These results are consistent with the finding from activation contrasts: during the Execute phase, a lower-level sensorimotor network was more involved in real than the pantomimed tool use, whereas a higher-level cognitive network was more involved in the pantomimed than the real tool use. Although regions were selected based on the differences during Execution, PPI investigates differences in synchrony between regions after differences in activation have been factored out. Moreover, there were also notable differences between comparisons of activation and comparisons of functional connectivity. Interestingly, aSMG, which has been proposed as a major area implicated in human tool use (Orban and Caruana, 2014) showed higher activation for pantomimed than real tool use (in the left hemisphere) but stronger functional connectivity with M1 for real than pantomimed tool use (in both hemispheres), which suggest that aSMG plays an important role for both real and pantomimed actions though demonstrated as a different approach (activation or connectivity).
While no differences in activation levels for real and pantomimed tool use were found during the Plan phase, functional connectivity was indeed different for these two kinds of tool use during the Plan phase.
Although PPI analyses may be vulnerable to differences between conditions related to differences in head (or bodily) motion, these factors are unlikely to account for the differences found here. First, the time courses show no evidence of artifacts (Figs. 3 and 4). Second, the nature of the movements was closely matched between the two tasks. Third, functional connectivity differences were observed during action planning, before any movements occurred.
Discussion
We found clear differences in activation levels, activation patterns, and functional connectivity between genuinely real tool use versus pantomimed tool use. Specifically, execution of real tool use preferentially activated somatosensory cortex (S1), motor areas (M1, PMd, aSPL, and anterior-medial cerebellum) and a parietal hub for tool processing (aSMG), whereas execution of pantomimed tool use preferentially activated higher-order association areas implicated in conceptual and semantic aspects of tools (including pMTG/STS, AG, SMG, PMv, and the posterior-lateral cerebellum). MVPA classification showed that activation patterns were also affected by the Action type in these same regions during the Execution Phase. Moreover, MVPA decoding showed that Action type also affected activation patterns during the Plan phase in S1, PMd, and precuneus. Using PPI, M1 showed stronger connectivity with other sensorimotor areas during real tool use but stronger connectivity with the conceptual tool network during pantomimed tool use. Importantly, functional connectivity differences were also observed during the Plan phase despite the absence of differences in activation levels.
Our results not only corroborate past studies showing higher activation in somatosensory/motor cortex (Hermsdörfer et al., 2007; Imazu et al., 2007); they go beyond these past studies in several ways. First, by examining the planning phase as well as execution, we found that S1 activation patterns could differentiate between tasks before somatosensory feedback occurred, consistent with the anticipation of feedback, one key difference between real versus pantomimed actions. Second, we found that M1 connectivity differed between the two tasks, even during planning, suggesting that even for tasks with highly similar kinematics, motor output is guided by input from different networks. Third, whereas previous studies reported differences for Real > Pantomime tool use, using a more naturalistic setup for real tool use, we also find that the classic tool-selective network is more activated for Pantomime > Real tool use. These results suggest that using pantomimed tool use as a proxy for real tool use may overestimate the contribution of brain regions implicated in conceptual aspects of tool use but underestimate the contribution of brain regions implicated in motor control and closed-loop feedback. These results add to a growing body of literature showing that the realism of objects and actions can affect behavior and brain activation (Westwood et al., 2000; Króliczak et al., 2007; Randerath et al., 2011; Snow et al., 2011; Holmes et al., 2013; J. Chen et al., 2015a,b; Lausberg et al., 2015; Freud et al., 2018).
Real versus pantomimed tool use: sensorimotor processing
We find that regions associated with sensorimotor processing are more active in real (than pantomimed) tool use, even during planning, likely because it is more of a “closed-loop,” goal-directed process than pantomimed tool use. That is, real tool use has real consequences and provides real (visual and somatosensory) feedback that can be used to correct actions and update internal models to improve performance on subsequent trials (Wolpert and Flanagan, 2001).
Indeed, consistent with the closed-loop nature of real (vs pantomimed) tool use, we see greater activation and/or M1 connectivity in somatosensory regions, which process somatosensory consequences, the premotor cortex, which can modulate motor commands based on context, and the anterior-medial cerebellum, which is implicated in forward models of actions (Imamizu et al., 2000, 2003). Interestingly, only one of the critical brain areas within the classic tool network, aSMG, showed higher activation for real tool use than for pantomimed tool use. aSMG is activated across a broad range of tool-related tasks (Ishibashi et al., 2016) and has been suggested as a hub in the tool network that integrates sensory, semantic, and mechanical aspects of tool use (Orban and Caruana, 2014). Our results suggest that this region also has enhanced crosstalk with the motor network, including enhanced connectivity with M1, during real tool use.
While higher activation for real (vs pantomimed) action execution in S1 would be expected, the more interesting findings are the effects of Action type on multivariate representations and functional connectivity during the Plan phase. S1, PMd, and precuneus activation patterns were affected by Action type during the Plan phase, before actions began and induced somatosensory and visual feedback, suggesting these regions are responding to the anticipation of sensory consequences that occur for real actions. Moreover, during the Plan phase, M1 shows stronger connectivity with one key node in the tool network, aSMG, for real than pantomimed actions, perhaps indicating stronger predictions about the somatomotor consequences of the tool kinematics.
Surprisingly, although participants performed actions with the right hand and past studies on tool processing revealed left-lateralized activation (J. Chen et al., 2018), activation for real tool use was bilateral. The finding of bilateral motor activation for real tool use is corroborated by other studies that found similar effects (Hermsdörfer et al., 2007; Lausberg et al., 2015). This may be related to the fact that many tools are used with both hands (e.g., shovels) and even those where the tool is used with by the dominant hand, the nondominant hand stabilizes the target object (e.g., hammer and nail). As such, real tool use may require greater coordination between the hands and thus between the hemispheres; and therefore, the reliance of proxies in the study of tool use may have overestimated the degree of hemispheric lateralization.
Pantomimed versus real tool use: conceptual processing
We find that tool-selective regions implicated in conceptual processing are more activated in pantomimed (than real) tool use including pMTG/STS, SMG, IPS, PMv, and aMFG in the left hemisphere and the posterior-lateral cerebellum in the right hemisphere. Action type differences were observed not only in the univariate activation during the Execute phase but also in multivariate patterns and functional connectivity during both the Plan and Execute phases. Several of these regions are reliably found in studies of tool processing (see Johnson-Frey, 2004; Lewis, 2006; Ishibashi et al., 2016), including conceptual and semantic aspects (pMTG/STS) as well as sensorimotor aspects (parietal foci and PMv) of tool cognition. However, our finding that these areas are less activated by real use suggests their role in actual tool use may be somewhat overestimated by the reliance on proxies. In fact, we found no significant activation in pMTG/STS for Real Execute > Real View (Fig. 3A), suggesting that under closed-loop real use, semantic/functional knowledge about tools may be less relevant. During real tool use, the consequences of an action are determined in large part by real-world physics (e.g., once the hand swings the hammer, the trajectory of the functional end moves based on momentum, gravity, and torque until it contacts the nail; see also Reynaud et al., 2016). In contrast, during pantomimed tool use, these relations between hand, tool and target object must be anticipated (during Plan) and modeled (during Execute). Perhaps for these reasons, pantomimed tool use recruits regions involved in tool manipulation knowledge [IPL, specifically posterior SMG and AG (Buxbaum, 2001; Binkofski and Buxbaum, 2013)], memory (aMFG), and semantics (STG; Zhao et al., 2017).
Implications for ideomotor apraxia
Our results have implications for understanding ideomotor apraxia (IMA), particularly patients who show dissociations in the performance of pantomimed versus real tool use (Geschwind and Damasio, 1985; Goldenberg and Hagmann, 1998). Patients with pantomime deficits typically have left brain damage, most often in and around the inferior frontal gyrus (Goldenberg et al., 2007), near one of the sites (PMv) where we observed stronger activation for pantomimed than real tool use. Notably, however, the typical damage spares many of the regions, including those in the right hemisphere, where we observed stronger activation for real tool use, allowing the possibility that these areas contribute to the patients improved performance as realism is added.
Does realness matter?
Although these results are consistent with the growing evidence that the realness of stimuli and tasks affects neural processing, there may be merit to the continued use of proxies. For example, localization of semantic/cognitive tool-related areas may be more effective with pantomimed use. However, if the goal is to understand the true neural substrates of actual tool use, greater realism is desirable.
Having established clear differences between realistic versus pantomimed tool use, one key question for future research is to determine which factors affect activation and which may reasonably be sacrificed in the interests of experimental convenience (Table 1). Given the growing popularity of virtual and augmented reality environments, another key question is whether the simulated consequences and closed-loop feedback provided is sufficient to invoke the same systems as real tool use.
In conclusion, we find that sensorimotor networks support real tool use whereas higher-order association areas subserve pantomimed tool use. This distinction between real and pantomimed tool use appears as early as the Plan phase. We suggest that these differences arise because real actions rely on real-world physics and real-time feedback, whereas pantomimed actions rely on knowledge, semantics, and memory. The spatial precision of fMRI also suggests specific areas that may account for dissociations between pantomimed and real tool use in neuropsychological patients.
Footnotes
This work was supported by the Natural Sciences and Engineering Research Council of Canada Discovery Grant RGPIN-2016-04748 and the Canadian Institutes of Health Research Operating Grant MOP 130345 and a Canada Research Chair in Immersive Neuroscience (to J.C.C.) and by the National Science and Technology Innovation 2030 Major Program STI2030-Major Projects 2022ZD0204802 and National Natural Science Foundation of China Grants No. 31970981 and No. 31800908 (to J.C.). We thank D. Adam McLean, Derek Quinlan, and Haitao Yang for technical assistance.
The authors declare no competing financial interests.
- Correspondence should be addressed to Juan Chen at juanchen{at}m.scnu.edu.cn
