Article Figures & Data
Figures
- Figure 1.
Experimental setup and design. We developed a novel setup to manipulate the congruence of visual, tactile, and proprioceptive signals directly on the participants' hands without the need for perceptual illusions involving artificial limbs. A, The participants lay supine on the bed of the MR scanner with the right hand placed on a tilted support and looked into a pair of head-mounted displays. B, Subjects were presented with a high-quality, stereoscopic view of their own right hand. C, Experimental design for Experiment 1. VT stimuli were delivered on two locations on the hand, the right index finger and the back of the hand. VT stimuli could be congruent in time and space, temporally incongruent (the onsets of the tactile and visual components were delayed by 1.25 s, resulting in an asynchronous mode of stimulation with a 250 ms gap between the visual and tactile stimuli), or spatially incongruent (the seen stroke was delivered to the index finger, whereas the felt stroke was delivered to the back of the hand, or vice versa; the strokes also differed in relative direction by 90 degrees). D, Experimental design for Experiment 2. A 2 × 2 factorial block design was implemented that manipulated the temporal congruence of the VT stimuli and the match between proprioceptive and visual information concerning hand position.
- Figure 2.
VT integration of hand signals depends on spatial and temporal congruence. A, Overview of the brain regions that are significantly modulated by both the temporal and spatial congruence of VT stimuli on one's own real hand (Experiment 1). The activation map was obtained by merging the two contrasts VT Congruent vs VT Time Incongruent and VT Congruent vs VT Space Incongruent with an inclusive masking procedure (see Materials and Methods section for details). For display purposes only, the activation map was displayed at a threshold of p < 0.001 (uncorrected for multiple comparisons) and overlaid onto a representative inflated cortical surface. The same procedure was used to display all of the activation maps in the successive figures. RH/LH, right/left hemisphere; CS, central sulcus; PoCS, postcentral sulcus; PrCR, precentral sulcus; IFS, inferior frontal sulcus; LOS, lateral occipital sulcus. B, Bar charts displaying the parameter estimates and SEs for all significant (p < 0.05 corrected for multiple comparisons; Table 1) peaks of activation. The coordinates are given in MNI space. For display purposes only, the anatomical location of the peak is indicated by a red circle on an activation map (p < 0.001 uncorrected for multiple comparisons) displayed on a coronal, sagittal, or axial section from the average structural image. L/R, left/right; PMv/d, ventral/dorsal premotor cortex; SMG, supramarginal gyrus.
- Figure 3.
Spatiotemporal congruence effects generalize across the two stimulation sites. A separate analysis confirmed statistically significant VT integration effects for both locations on the hand (index finger and dorsum). This result demonstrates that the spatiotemporal congruence effects generalize across the two stimulation sites, justifying the pooling of the data across the two conditions presented in the main analysis of Experiment 1 (Fig. 2). The coordinates are given in MNI space. All of the peaks display significant (p < 0.05 corrected for multiple comparisons) differences obtained from the contrasts VT Congruent vs VT Time Incongruent and VT Congruent vs VT Space Incongruent for the two locations on the hand.
- Figure 4.
Spatial and temporal congruence between visual and tactile signals from the hand enhances the effective connectivity of the multisensory regions. Two separate whole-brain PPI analyses revealed enhanced connectivity between a region in the left IPS and key multisensory areas in other parts of the brain. A, Group results: an overview of all brain regions displaying enhancements of effective connectivity with the left IPS (indicated by a white star) in conjunction with the temporal and spatial congruence of visual and tactile stimuli. B, PPI plots for one representative participant. The plots show significantly steeper regression slopes that relate activity levels in the seed region with response magnitudes in the left ventral premotor cortex under conditions of both temporal (left plot; red line) and spatial (right plot; red line) congruence, as opposed to the corresponding incongruence manipulations (black lines). C, Group results: significant (p < 0.05 corrected for multiple comparisons) regions that display enhanced connectivity with the seed region (blue square) under temporal and spatial congruence of VT stimuli.
- Figure 5.
The integration of congruent VT signals depends on the match between the seen and felt positions of the hand. A, Overview of the brain regions that display significantly greater integration effects for temporally congruent as opposed to incongruent VT signals only in the context of matching visual and proprioceptive signals about hand position (as obtained from the interaction contrast in Experiment 2 defined as (VT CongruentVP Match vs VT Time IncongruentVP Match) vs (VT CongruentVP Mismatch vs VT Time IncongruentVP Mismatch). B, Bar charts displaying the parameter estimates and SEs for all significant (p < 0.05 corrected for multiple comparisons; Table 2) peaks of activation. The labels table and retracted indicate the experimental conditions that involved VP match and mismatch, respectively (Fig. 1D). PcG, postcentral gyrus.
- Figure 6.
Matching visual and proprioceptive signals about hand position enhance the effective connectivity of the brain regions that integrate congruent VT signals. A whole-brain PPI analysis revealed changes in the effective connectivity between a region in the left IPS and other multisensory areas that depend on the congruence of visual and proprioceptive signals concerning hand position (Experiment 2). A, Group results: overview of all brain regions displaying enhanced effective connectivity with the left IPS (indicated by a white star). B, PPI plot for one representative participant. The plot displays a significantly steeper regression slope (red line) relating activity levels in the seed region with the response magnitude in the left ventral premotor cortex under conditions of visual, tactile, and proprioceptive match, as opposed to the incongruent condition (black line). C, Group results: significant (p < 0.05 corrected for multiple comparisons) regions that display enhanced connectivity with the seed region (blue square).
- Figure 7.
The maintenance of default limb self-attribution requires congruent multisensory stimulation: subjective reports and correlations with BOLD responses. A, Questionnaire data indicating how the subjective experience of the seen real hand as part of the own body changes as a function of the different experimental conditions. S1 assessed the multisensory perceptual binding of visual, tactile, and proprioceptive signals into a single owned hand, a percept that depended on the three-way congruence of visual, tactile, and proprioceptive information (2 × 2 repeated-measures ANOVA, F(1,14), 19.286, p, 0.001). S2 refers to the perceptual fusion of the visual and tactile signals from the hand into a unique coherent percept (interaction between VT congruence and VP match, F(1,14), 4.808, p, 0.046). Subjective ratings to S3 revealed an increasing loss of perceptual self-identification with one's own real hand, with increasing incongruence between the sensory signals from the hand. Namely, the participants had an increasingly vivid impression that they were looking at someone else's hand rather than at their own hand, with a loss of congruence between vision, touch, and proprioception (main effect of condition, F(1,14), 182.21, p < 0.001; pairwise t tests, all p < 0.014; VT CongruentVP Match < VT Time IncongruentVP Match < VT CongruentVP Mismatch < VT Time IncongruentVP Mismatch, two-tailed paired samples t tests. Bonferroni-corrected for multiple comparisons). B, We generated a subjective perceptual self-attribution index by computing the interaction effect size for the ratings to S1. We conducted an independent multiple regression analysis and searched for voxels whose BOLD interaction effect size for VT proprioceptive congruence significantly predicted the subjective self-attribution index. We identified such relations in the left premotor cortex (left PMv; x = −48, y = −4, z = 28, MNI coordinates), left supramarginal gyrus (left SMG; x = −62, y = −38, z = 28), right IPS (x = 36, y = −46, z = 64), and right LOC (x = 56, y = −54, z = −6).
- Figure 8.
Threat-evoked SCRs and BOLD responses correlate with neural signatures of multisensory integration. A, After periods of multisensory stimulation under the different levels of congruence between vision, touch, and proprioception, participants were presented with threat events consisting of a knife appearing and swiftly sliding over the hand without touching it. B, The blue curve displays a representative SCR for a single threat event with the onset indicated by the red arrow. C, Significantly greater SCRs after congruent multisensory stimulation compared with the three conditions with VT and/or VP incongruence (a repeated-measures ANOVA revealed a significant interaction between VT congruence and VP match). D, Significantly greater BOLD responses in the anterior insular and cingulate cortices—areas that are related to pain anticipation and anxiety evoked by physical threats (Ehrsson et al., 2007)—are triggered by the presentation of the knife sliding over the hand after congruent multisensory stimulation compared with the three incongruent conditions (significant interaction in the 2 × 2 factorial design; p < 0.05 corrected for multiple comparisons). A significant interaction was also found in the right premotor cortex and in the right cerebellum. E, The BOLD interaction effect size that reflects multisensory integration significantly predicted the effect size related to the threat-evoked SCRs. A whole-brain multiple regression analysis revealed such relations in the right ventral and dorsal premotor cortices (R PMv, x = 48, y = 8, z = 16; R PMd, x = 46, y = 8, z = 50) and in the right LOC (R LOC, x = 56, y = −60, z = 4).
- Figure 9.
VT congruence effects could be dissociated from an explicit manipulation of endogenous visuospatial attention. In a separate control experiment, we reproduced the finding of VT congruence effects in the premotor, posterior parietal, lateral occipital, and cerebellar cortices in the context of an explicit visuospatial attention task (Zimmer and Macaluso, 2007). A, A circle containing black and white lines changed orientation randomly every 2 s. The participants were instructed to press a button with their left hand as soon as they detected that the lines were in a vertical orientation (red square). B, Neither the reaction time nor the performance accuracy differed across epochs of VT Congruent and VT Time Incongruent stimuli on the hand, which indicated that the participants engaged their visuospatial attention equally across the experimental conditions. C, Significant differences between the BOLD responses to VT Congruent and VT Time Incongruent conditions were observed in all areas identified in the main analysis for Experiment 2 (*p < 0.05, **p < 0.01 corrected for multiple comparisons based on peaks from the main analysis of Experiment 2; Table 2).
Tables
- Table 1.
Integration of visual and tactile signals from the hand under conditions of full temporal and spatial congruence as shown by Experiment 1: VT Congruent vs VT Time Incongruent and VT Congruent vs VT Space Incongruent
Anatomical region MNI x, y, z (mm) Peak t p-value corrected L IPS −36, −44, 58 6.58 0.001 L IPS −46, −34, 40 6.07 0.004 R IPS 42, −34, 58 5.62 0.008 R IPS 38, −52, 58 4.33 0.019 L precentral gyrus (PMv) −54, −2, 28 4.02 0.026 L precentral gyrus (PMd) −52, 2, 40 4.02 0.026 R precentral sulcus (PMv/d) 42, 2, 42 3.43 0.043 L supramarginal gyrus −56, −38, 28 4.01 0.026 R supramarginal gyrus 50, −38, 34 4.81 0.014 L LOC −48, −66, 6 4.03 0.026 R LOC 56, −62, −4 3.72 0.039 R cerebellum (lobule VIIa) 36, −68, −46 3.75 0.039 L parietal operculum −40, −30, 22 3.27 <0.001* L anterior insula −32, 18, 18 3.44 <0.001* L putamen −28, 6, 4 3.01 <0.001* R postcentral gyrus 54, −20, 28 3.40 <0.001* PMv/d, ventral/dorsal premotor cortex.
↵*Uncorrected for multiple comparisons based on a priori hypotheses.
- Table 2.
Visuotactile integration depends on congruent visuoproprioceptive signals from the upper limb as shown by Experiment 2: (VT CongruentVP Match vs VT Time IncongruentVP Match) vs (VT CongruentVP Mismatch vs VT Time IncongruentVP Mismatch)
Anatomical region MNI x, y, z (mm) Peak t p-value corrected L IPS −36, −44, 58 3.62 0.021 L IPS −42, −40, 50 3.46 0.030 L superior parietal gyrus −30, −50, 64 3.60 0.021 R IPS 38, −42, 56 5.86 <0.001 R IPS 36, −56, 48 3.24 0.049 L precentral sulcus (PMv) −48, 4, 26 4.05 0.014 R precentral gyrus (PMv) 48, 6, 36 4.94 0.001 R precentral gyrus (PMd) 38, −4, 58 4.32 0.010 L supramarginal gyrus −56, −32, 36 4.71 0.003 R supramarginal gyrus 44, −38, 36 5.86 <0.001 R LOC 56, −58, −4 4.33 0.007 R postcentral gyrus 60, −22, 32 3.78 0.019 R cerebellum (lobule VIIa/b) 24, −66, −46 3.24 0.049 L precentral gyrus (PMd) −30, −4, 46 3.22 <0.001* R parietal operculum 48, −28, 22 3.05 <0.001* PMv/d, ventral/dorsal premotor cortex.
↵*Uncorrected for multiple comparisons based on a priori hypotheses.
