Disintegration of Multisensory Signals from the Real Hand Reduces Default Limb Self-Attribution: An fMRI Study

Integration of visual, tactile, and proprioceptive signals from the hand

In this experiment, we tested the hypothesis that the neural and perceptual integration of visual and tactile signals from the hand requires a concurrent match between the seen and felt positions of the hand. Moreover, we sought to examine the link between the BOLD signatures of multisensory integration and the default perception that the real hand in view is one's own. The overall procedure involved the preparation of tailor-made visual stimuli of each participant's own body and hand; the experimental setup and the number and timing of the stimuli were identical to that described in Experiment 1. To avoid extending the total duration of the scanning sessions, we included VT stimuli on only one of the two parts of the hand that was stimulated in Experiment 1, namely the right index finger. Furthermore, we selected only one of the congruence manipulations, specifically the temporal aspect of the VT stimuli. This decision was further warranted by the detailed analysis of the data from Experiment 1, which revealed no significant differences between the two target zones on the hand in terms of sensitivity to spatiotemporal multisensory congruence (Fig. 3). We implemented a 2 × 2 factorial design in which we independently manipulated the congruence of the VT stimuli (congruent vs temporally incongruent) and the posture of the hand (hand on the table: visuoproprioceptive [VP] match; hand retracted: VP mismatch). This arrangement resulted in a total of four experimental conditions of interest (Fig. 1D). Two of these conditions were identical to those containing congruent or temporally incongruent VT stimuli on the right index finger used in Experiment 1, which allowed internal validation of the findings from the first experiment. The two remaining conditions differed only in terms of the posture of the participant's own right hand in the scanner. Although the visual and tactile stimuli remained identical, in two the four acquisition sessions, the right hand was retracted from the table, placed on the participant's chest, and rotated by ∼90 degrees within the right hemispace in proximity to the body's midline. This postural manipulation introduced a clear mismatch between the visual and proprioceptive hand signals. The order of the acquisition sessions for the two positions of the hand was counterbalanced across the participants. Each condition was repeated a total of eight times in each session. The participants were instructed to perform the same tasks assigned in Experiment 1 comprising continuous fixation throughout the acquisition sessions and the detection of two catch trials per session (see Experiment 1 above for details).

Quantifying self-perception of the hand

Our experimental setup allowed us to relate BOLD responses reflecting different levels of congruence between visual, tactile, and proprioceptive signals directly with changes in the multisensory perception of the hand in view as one's own. The degree of self-attribution of the seen hand was quantified by objective (BOLD and skin conductance responses during fMRI) and subjective (postscan questionnaire) measures, as described in detail below.

Recording of skin conductance responses during fMRI

Previous studies of the feeling of body ownership that have used perceptual illusions have described how the skin conductance responses (SCRs) evoked by physical threats applied to artificial limbs or mannequins can be used as objective physiological evidence of self-attribution (Armel and Ramachandran, 2003; Ehrsson, 2007; Petkova and Ehrsson, 2008; Guterstam et al., 2011; Newport and Gilpin, 2011; Guterstam and Ehrsson, 2012). Similarly, a previous fMRI study described threat-evoked BOLD responses in key areas related to pain anticipation and anxiety when a sharp object approached the participants' real hand or an artificial hand during the rubber hand illusion (Ehrsson et al., 2007). Here, instead of using perceptual illusions to transfer the feeling of ownership over an artificial limb, we experimentally altered the congruence of multiple sensory signals to probe the maintenance of self-attribution of one's real hand in the context of incongruent multisensory stimulation. We concurrently recorded BOLD and SCR responses to threats on the participant's own right hand after periods of exposure to multisensory stimuli under different levels of congruence between visual, tactile, and proprioceptive signals. To accomplish this goal, a randomly selected 50% of all of the trials for each of the 4 main experimental conditions (see above) were followed by a 2 s threat stimulus. The latter featured a kitchen knife appearing in the field of view of the head-mounted displays and sliding swiftly just above the participant's own right hand (Fig. 8A; adopted from Guterstam et al., 2011). To record the physiological skin conductance response to the threat stimuli, we used an MR-compatible SCR-recording module (Brain Products). Before the onset of the acquisition sessions, two electrodes were attached to the index and middle fingers of the participant's left hand using electrode gel. The electrodes were connected to an MR-compatible amplifier (BrainAmp ExG MR; Brain Products), and continuous recordings for each session were collected using a computer running the BrainVision Recorder (acquisition sampling rate 5000 Hz). All recordings were stored and imported into MATLAB (MathWorks) for further offline analysis. For each threat event, we identified the maximal and minimal values of the SCR within a 5 s temporal window that was aligned to the event onset. For all trials, the event-specific SCR amplitude was then calculated as the difference between the maximal and minimal values and an average value was computed for each participant and condition (Dawson et al., 1990; Ehrsson, 2007; Petkova and Ehrsson, 2008; Guterstam and Ehrsson, 2012). For technical reasons, SCR data could not be recorded for one participant. Therefore, all further analyses featuring SCR data included 14 participants. The data passed the Kolmogorov–Smirnoff test for normality and were analyzed using a 2 × 2 repeated-measures ANOVA in SPSS version 20 software.

FMRI data acquisition

FMRI acquisition was performed using a Siemens TIM Trio 3T scanner equipped with a 12-channel head coil. Gradient echo T2*-weighted EPIs with BOLD contrast were used as an index of brain activity (Logothetis et al., 2001). A functional image volume was composed of 40 continuous near-axial slices of 3 mm thickness (with a 0.1 mm interslice gap), which ensured that the whole brain was within the FOV (58 × 76 matrix, 3.0 mm × 3.0 mm in-plane resolution, TE = 40 ms). One complete volume was collected every 2.54 s (TR = 2540 ms). A total of 900 functional volumes were collected for each participant for Experiment 1, which was equally divided into three sessions. A total of 660 volumes were acquired for Experiment 2, which comprised four sessions of equal length. Finally, 165 volumes were collected in Experiment 3, which comprised a single session. An initial baseline of 15 s and a final baseline of 15 s were included in each session for all experiments. The first three volumes of each session were discarded to account for non-steady-state magnetization. To facilitate the anatomical localization of statistically significant activations, a high-resolution structural image was acquired for each participant at the end of the experiment (3D MPRAGE sequence, voxel size = 1 mm × 1 mm × 1 mm, FOV = 250 mm × 250 mm, 176 slices, TR = 1900 ms, TE = 2.27 ms, flip angle = 9°).

Data preprocessing, modeling, and statistical inference

All fMRI data were screened for potential artifacts using ART (Massachusetts Institute of Technology). No noteworthy artifacts were detected in any of the acquired datasets. The functional imaging data from all three experiments underwent the same series of preprocessing steps using SPM8 (Wellcome Trust Center for Neuroimaging) before all successive analyses. The functional volumes were motion corrected with respect to the first volume of each series, corrected for slice-timing errors, and coregistered to the high-resolution structural image. The latter was segmented into gray matter, white matter, and CSF partitions and was normalized to the MNI standard space. The same transformation was then applied to all functional images, which were resliced to a resolution of 2 mm × 2 mm × 2 mm and spatially smoothed with an 8 mm FWHM Gaussian kernel.

For each experiment, we fitted a general linear model (GLM) to the data for each individual participant. We defined boxcar regressors for the conditions of interest (see below for experiment-specific details) and convolved them with the standard hemodynamic response function modeled in SPM8. Linear contrasts of interest were defined for each participant as appropriate combinations of the model parameters and exported to a second-level random-effects analysis. Given our strong a priori hypotheses on the anatomical localization of the multisensory brain regions, we applied a correction for multiple comparisons in all statistical tests within regions of interest defined around peaks from a previous study (Gentile et al., 2011). Therefore, unless otherwise specified, all reported peaks of activations are statistically significant at a threshold of p < 0.05 after correction for multiple comparisons using familywise error corrections within the corresponding volumes of interest. For visualization purposes only, all activation maps are displayed at a threshold of p < 0.001 (uncorrected) and are overlaid onto a representative inflated cortical surface using Freesurfer (MGH) as well as onto axial, sagittal, or coronal sections from the average anatomical image for all participants in the study. All anatomical localizations of the significant peaks of activation refer to this average anatomical image, with the nomenclature from the human brain atlas of Duvernoy and Parratte (Duvernoy and Parratte, 1999). Contrast estimates for all significant peaks of activation were extracted using MATLAB (Mathworks) and displayed as bar charts together with the corresponding SEs.

FMRI regional analyses

In Experiment 1, we intended to identify brain regions that display BOLD responses to VT stimuli that are enhanced by the temporal and spatial congruence of the unisensory components. We defined two linear contrasts, VT Congruent vs VT Time Incongruent and VT Congruent vs VT Space Incongruent, by pooling data from the trials that involved both the index finger and the back of the hand. These two contrasts are fully matched in terms of all visual and tactile inputs across the two locations on the hand. Therefore, significant voxels obtained from the above contrasts are taken to contain neuronal populations that specifically integrate visual and tactile stimuli from the hand that are congruent in time and colocalized on the same part of the hand. Significant peaks of activations, corrected for multiple comparisons, were obtained from both contrasts. To combine the activation maps from the two contrasts and display them as a single map, we used an inclusive masking procedure. One of the two contrasts, VT Congruent vs VT Space Incongruent, served as an inclusive mask for the other contrast, VT Congruent vs VT Time Incongruent. Please note that using one of the contrasts as an inclusive mask for the other contrast has no effect on the validity of the statistical methods applied and is not in any way circular. Swapping the role of the two contrasts, for example, using VT Congruent vs VT Time Incongruent as an inclusive mask for VT Congruent vs VT Space Incongruent, yielded the same significant peaks of activation, as expected.

Next, we moved on to test the hypothesis that the spatiotemporal multisensory congruence effect obtained from the previous analysis was present for both stimulated parts of the hand. We repeated the procedure described above with linear contrasts separately defined for the two parts of the hand. In this case, spatial incongruence was first defined as identical visual input in the presence of incongruent tactile input. The same results were obtained by labeling as spatially incongruent all trials with identical tactile input but incongruent visual input for each part of the hand.

In Experiment 2, we first tested the hypothesis that the integration of visual and tactile signals from the hand is modulated by proprioceptive signals from the upper limb to construct a unitary percept of one's hand in space. To this end, we defined the interaction contrast from the 2 × 2 factorial design (see above) as (VT Congruent_{VP Match} vs VT Time Incongruent_{VP Match}) vs (VT Congruent_{VP Mismatch} vs VT Time Incongruent_{VP Mismatch}), where the labels VP Match and VP Mismatch indicate the two different postures of the right hand, respectively. This contrast yielded voxels that exhibited BOLD responses that cannot be accounted for by an additive effect of the (temporal) congruence between the visual and tactile stimuli and the congruence between the seen and felt postures of one's own right hand, as determined by the match between vision and proprioception. Instead, this contrast revealed voxels containing neuronal populations that compute the integration of the visual and tactile signals only in the context of a postural match between the seen and felt location of the hand. This response pattern would be expected from brain areas that perform multisensory integration in hand-centered coordinates.

Next, we tested the hypothesis that the disintegration of the congruence between vision, touch, and proprioception would impair the perceptual mechanisms that underlie the “default” self-attribution of the visible real hand. We defined a 2 × 2 factorial interaction contrast that was identical to the contrast described above but that modeled the two-second threat events after the periods of exposure to multisensory stimuli under different levels of congruence. This contrast revealed voxels with BOLD responses that reflected the modulation of the cortical-threat-evoked response in areas related to pain anticipation and anxiety (Lloyd et al., 2006; Ehrsson et al., 2007).

In Experiment 3, our aim was to reproduce the sensitivity to multisensory congruence that we observed in the first two experiments during a situation in which we explicitly controlled for endogenous visuospatial attention. Behavioral data were collected in terms of the response times and accuracies for each trial. The fMRI data were analyzed by defining the linear contrast VT Congruent vs VT Time Incongruent in the presence of an explicit task that shifted the focus of the participants' endogenous visuospatial attention away from the stimuli on the hand. Statistical inference was conducted by testing the above contrast on all the significant peaks of activation identified by the main analysis in Experiment 2 (see above). Please note that because the visuospatial attention task was introduced in a separate acquisition session at the end of the main experiment, this statistical analysis is not in any way circular.

FMRI effective connectivity analyses

We used psychophysiological interaction (PPI) analyses to test the hypothesis that the integration of spatially and temporally congruent signals in the fronto-parieto-cerebellar regions would be associated with increases in the effective connectivity between different nodes of the involved networks. The PPI reflects context-induced changes in the strength of the connectivity between two brain regions, a seed and a target, as measured by a change in the magnitude of the linear regression slope that relates their underlying activities. A significant PPI indicates that the contribution of one area to the activity of another area changes significantly with the experimental or psychological context (Friston et al., 1997). Given that such interactions are assumed to take place at the neuronal level, it is recommended to estimate the underlying neuronal signal from the measured BOLD response. This process is built on the deconvolution of the hemodynamic response function from the measured BOLD time series and was described in detail in a seminal paper by Gitelman et al. (2003). In both Experiments 1 and 2, connectivity changes between a seed region in the left intraparietal sulcus (IPS) and the other areas of the brain were assessed for specific manipulations of the congruence of visual, tactile and proprioceptive signals from the hand. We chose to place the seed in the left IPS on the basis of both neurophysiological evidence and previous brain imaging studies. Given its ideal anatomical location for receiving converging inputs from sensory areas that process visual, tactile, and proprioceptive signals and its well known anatomical connectivity with other multisensory areas in the frontal and inferior parietal lobes (Grefkes and Fink, 2005; Culham et al., 2006; Avillac et al., 2007), the IPS is ideally suited to act as a central node for the processing of multisensory bodily signals (Makin et al., 2008; Ehrsson, 2012). Indeed, the left IPS has been implicated in a number of studies that have investigated the processing of multisensory signals from the contralateral right hand (Lloyd et al., 2003; Ehrsson et al., 2004; Makin et al., 2007; Brozzoli et al., 2011, 2012; Gentile et al., 2011). The seed region was defined for each participant and was centered on the peak voxel found within a 10 mm radius sphere centered on the group peak for the contrast of interest. The seed region's time series was computed as the first eigenvariate of all voxels within a 4 mm radius sphere centered on the participant-specific peak voxel. At the individual level, three regressors were created that represented the time series of the seed region (the physiological factor), the experimental manipulation of interest (the psychological factor), and their product (the PPI). A GLM containing these three regressors was estimated for each participant and contrast estimates for the PPI regressor were analyzed at the group level using one-sample t tests in a random-effects model. Statistical inference was applied in a manner that was identical to the approach used for all of the fMRI regional analyses (see above).

In Experiment 1, we tested the hypothesis that both the temporal congruence and the spatial congruence of visual and tactile stimuli on one's own seen real hand modulate the effective connectivity between key nodes of multisensory circuits that integrate hand signals (Makin et al., 2008; Gentile et al., 2011; Guterstam et al., 2013). The seed region in the left IPS was centered on the group peak from the main analysis of Experiment 1 (MNI coordinates: x = −36, y = −44, z = 58; Figs. 2B, 4A–C). The average distance of the participant-specific seed region from the group peak was 6.52 ± 2.40 mm. The PPI analyses followed the same logic as the corresponding regional analyses. We conducted two separate analyses using VT Congruent vs VT Time Incongruent and VT Congruent vs VT Space Incongruent as experimental factors and extracted significant voxels for both contrasts by correcting for multiple comparisons on the basis of our a priori hypotheses (see above). In the first of these two analyses, the psychological regressor in the PPI model corresponded to VT Congruent vs VT Time Incongruent, where the first condition was coded with a weight of +1 and the second with a weight of −1, respectively. All other conditions in the original GLM were coded with a weight of 0. In the second PPI analysis for Experiment 1, the psychological regressor corresponding to VT Congruent vs VT Space Incongruent was generated by coding the first condition with a weight of +1 and the second with a weight of −1, respectively. Again, the remaining conditions were coded with 0.

In Experiment 2, our aim was to demonstrate that the effective neural connectivity between the multisensory regions described above changes as a function of the proprioceptive inputs from the upper limb. We defined the comparison VT Congruent _{VP Match} vs VT Congruent _{VP Mismatch} as the experimental factor for a PPI analysis with the seed in the left IPS (peak voxel from the factorial interaction analysis: x = −36, y = −44, z = 58; Figs. 5B, 6A–C). The psychological regressor was therefore generated by coding the condition VT Congruent _{VP Match} with +1 and the condition VT Congruent _{VP Mismatch} with −1, respectively. All remaining conditions in the original GLM were coded with a weight of 0. The average distance of the participant-specific seed region from the group peak was 6.06 ± 2.24 mm. It is noteworthy that, because the PPI-GLM contains a regressor that explicitly models the experimental manipulation of interest, the effective connectivity results cannot be explained by differential levels of brain activation in response to the different experimental conditions. Instead, these findings provide independent evidence in favor of the hypothesis that the identified brain regions work in concert to build a representation of the hand that draws from multiple sensory modalities and strictly depends on their relative congruence.

Correlations between subjective and objective measures of self-attribution of the hand and multisensory brain activity

To further corroborate the finding that different levels of congruence among visual, tactile, and proprioceptive signals impact the way participants maintained a stable perception of their own hand as part of their body, we performed multiple regression analyses to relate observed changes in the BOLD response to subjective and objective indices of hand self-perception for Experiment 2. First, we defined a “self-attribution index” by computing the interaction score for subjective ratings to statement S1 for each participant according to the formula: (VT Congruent _{VP Match} vs VT Time Incongruent _{VP Match}) vs (VT Congruent _{VP Mismatch} vs VT Time Incongruent _{VP Mismatch}). The indices for all participants were then entered as a covariate in a multiple regression model alongside the BOLD interaction effect. The model was estimated for the whole brain and yielded voxels that displayed a significant positive correlation between the subjective ratings of hand self-perception and the corresponding brain responses. Second, we computed the interaction effect size for the skin conductance responses to the four experimental conditions in Experiment 2 according to the same formula: (VT Congruent _{VP Match} vs VT Time Incongruent _{VP Match}) vs (VT Congruent _{VP Mismatch} vs VT Time Incongruent _{VP Mismatch}). The SCR indices for each participant were entered as a covariate in a new multiple regression model alongside the BOLD interaction effect, which was related to the intervals of multisensory stimulations preceding the threat events. This analysis revealed voxels with a BOLD effect size (which is related to the magnitude of the response to multisensory inputs) that significantly predicted the physiological response to a threat event that followed in time. Both multiple regression analyses were anatomically unbiased (i.e., they were whole-brain analyses) and independent from the main regional analyses (see above) and therefore were not circular.