Article Figures & Data
Figures
- Figure 1.
Experimental stimuli: two sets (1 per actor) of dynamic and time-synchronized audiovisual stimuli were presented within a continuous carryover design (Aguirre, 2007) in interleaved type 1–index 1 sequences over two experimental sessions (sequential presentation indicated by dotted lines in the left panels). Every face morph (extending from 90% angry to 90% happy) was paired with every voice morph (extending from 90% angry to 90% happy) within actor, so to create two sets of 25 face–voice stimuli, parametrically varying in congruence (examples in colored rectangles). In a carryover design, every stimulus precedes and follows every other stimulus, such that each stimulus serves as an adaptor for the following stimulus. The right panels show parts of an example type 1–index 1 sequence. In each experimental run, sequences were blocked by actor. Participants performed a two-alternative forced-choice task (angry or happy) on emotion. The right panels indicate examples of within-block sequences of stimuli. ISI, Interstimulus interval.
- Figure 2.
Behavioral results: direct effects of face and voice emotion morph. a, Categorization results. Categorization (percentage angry responses) as a function of face morph (middle), voice morph (right), and both (left). b, Reaction time results. Reaction time (milliseconds) as a function of face morph (middle), voice morph (right), and both (left). Both face and voice morph were morphed between 10% happy and 90% happy in 20% steps. Both categorization and reaction time results are averaged across actor. Note the greater influence of facial versus vocal emotional cues on behavioral responses. A, Angry; H, Happy.
- Figure 3.
Imaging results: multimodal localizer. Left, A cluster in the right STG/STS responding more to audiovisual, compared with either visual or auditory, information alone localized using a conjunction analysis (AV > A ∩ AV > V; conjunction null hypothesis; Nichols et al., 2005). Results were thresholded at p < 0.05 (cluster corrected). Right, Condition effects at the peak voxel of the cluster.
- Figure 4.
Imaging results. a, Unimodal face adaptation. Activation in left putamen and right FG in response to varying percentage difference in face morph between consecutive stimuli. Left and right, Parameter estimate at the peak activated voxel of left putamen and right FG, respectively, as a result of varying percentage difference in face morph. Results were thresholded at p < 0.05 (FWE voxel corrected) and a minimum cluster size of more than five contiguous voxels. b, Unimodal voice adaptation. Activation in bilateral STG/STS in response to varying percentage difference in voice morph between consecutive stimuli. Left and right, Parameter estimate at the peak activated voxel in left and right STG/STS, respectively, as a result of varying percentage difference in voice morph. Results were thresholded at p < 0.05 (FWE voxel corrected) and a minimum cluster size of more than five contiguous voxels. c, Crossmodal adaptation. Red, Results from the independent functional multimodal localizer. An ROI analysis showed that voice-to-face emotion morph difference evoked a significant response in this region (p < 0.025, t = 2.12). Green, Activation in right pSTS as a result of varying percentage difference between voice and the following face morph of consecutive stimuli. Results were thresholded at p < 0.001 (voxel uncorrected) with a minimum cluster size of more than five contiguous voxels. Yellow, Overlap between the localizer and voice-to-face morph difference activation. Left and right, Parameter estimate at the peak activated voxel of right pSTS as a result of varying percentage difference in face-to-voice morph, and voice-to-face morph, respectively. It should be noted that face-to-voice morph difference did not evoke a significant response in this region.
Tables
Brain regions Coordinates (mm) k t statistic x y z a, Multimodal localizer STG/STS 48 −40 13 153 5.23 b, Unimodal adaptation i, Adaptation to face emotion Putamen −21 8 10 14 7.46 FG 30 −52 −23 8 6.40 ii, Adaptation to voice emotion STG/STS 54 −22 10 51 7.98 STG/STS −60 −37 7 24 7.37 Inferior frontal gyrus 48 23 22 11 6.32 c, Crossmodal adaptation i, Adaptation to voice-to-face emotion STS 66 −46 4 9 4.20 ii, Adaptation to face-to-voice emotion No significant voxels a, Results from multisensory functional localizer experiment. Contrasts were cluster thresholded at p < 0.05 (FWE corrected). MNI coordinates and t scores are from the peak voxel of a cluster. b, Unimodal adaptation results. bi, Adaptation to face emotion. bii, Adaptation to voice emotion. Contrasts were thresholded to display voxels reaching a significance level of p < 0.05 (FWE corrected) and an additional minimum cluster size of more than five contiguous voxels. Contrasts were also masked by an AV versus baseline contrast thresholded at p < 0.001 (voxel uncorrected). MNI coordinates and t scores are from the peak voxel of a cluster. c, Crossmodal adaptation results. ci, Voice-to-face adaptation. cii, Face-to-voice adaptation. Contrasts were thresholded to display voxels reaching a significance level of p < 0.001 (uncorrected) and an additional minimum cluster size of more than five contiguous voxels. Contrasts were masked by an AV versus baseline contrast thresholded at p < 0.001 (voxel uncorrected). MNI coordinates and t scores are from the voxel of a cluster.
