Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices

Modulation of BOLD responses attributable to audiovisual correspondence

For fMRI analyses, the random-effect SPM analysis confirmed that unisensory visual streams activated sensory-specific occipital visual cortex, as expected, whereas auditory streams activated auditory core, belt, and parabelt regions in temporal cortex, as expected (Table 1). Candidate multisensory regions, activated by both the unisensory visual and unisensory auditory streams, included bilateral posterior STS and posterior parietal and dorsolateral prefrontal areas. However, within these candidate multisensory regions, only STS showed the critical effects of audiovisual temporal correspondence (Table 2A, Fig. 2A). Within the functionally defined multisensory regions, AVC minus NC specifically activated (at p < 0.001) the contralateral mSTS (i.e., right mSTS for LVF group, peak at 60, −48, 12; left mSTS for RVF group, peak at −54, −52, 8) (Fig. 2A). Additional tests on individually defined maxima within mSTS confirmed that, contralateral to the visual stream, responses to AVC were significantly elevated not only relative to the NC condition but also relative to either unisensory stream alone (p < 0.03). Noncoincidence led instead to a reliably decreased response relative to either unisensory baseline (p < 0.01) (see bar graph for mSTS in Fig. 2A). All individual subjects showed this pattern (for illustrative single subject, see Fig. 3A).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

fMRI results: BOLD signal differences for corresponding minus noncorresponding audiovisual stimulation. Group effects in the following: A, contralateral multisensory superior temporal sulcus; B, contralateral early visual cortex; C, bilateral auditory cortex, with contralateral peak. Shown for the RVF and LVF groups (columns 1, 2 and 3, 4, respectively). The intersubject mean parameter estimates (SPM betas, proportional to percentage signal change) are plotted for contralateral mSTS, primary visual cortex, and primary auditory cortex (each plot in corresponding rows to the brain activations shown) from the subject-specific maxima used in the individual analyses, averaged across LVF and RVF groups, with mean Montreal Neurological Institute coordinates below each bar graph. Brackets linking pairs of bars in these graphs all indicate significant differences across those conditions (p < 0.05 or better).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

fMRI results: BOLD signal differences for corresponding minus noncorresponding audiovisual stimulation in an illustrative single subject. mSTS, visual cortex, and auditory cortex activations are shown, with the STS, the calcarine fissure, and Heschl's gyrus highlighted in blue on that individual's anatomical scan. Localization of the effects with respect to these anatomical landmarks was implemented in every individual.

Importantly, an analogous pattern was found within sensory-specific cortices. For visual cortex, we found increased BOLD responses for the AVC > NC comparison near the contralateral calcarine fissure (peaks at −12, −76, 0 and 12, −82, 12 for RVF and LVF groups, respectively; both p < 0.001) (Fig. 2B, Table 2B). Again, this effect was found for each individual subject, in the anterior lower lip of their calcarine fissure (for illustrative single subject, see Fig. 3B) representing the contralateral peripheral upper visual quadrant, in which the visual stimuli appeared.

Finally, enhanced BOLD response for AVC > NC stimulation was found also within sensory-specific auditory cortex, in the vicinity of Heschl's gyrus, also peaking contralateral to the coincident visual hemifield (peaks at −48, −20, 10 for RVF and 50, −16, 8 for LVF group; both p values < 0.001) (Fig. 2C, Table 2C), albeit with some bilateral activations also found yet a systematically contralateral peak. We found this pattern in 23 of the 24 individual subjects, within the medial part of anteriormost Heschl's gyrus (typically considered as primary auditory cortex), often extending into posterior insula and planum temporale (for illustrative single subject, see Fig. 3C).

Mean parameter estimates (SPM betas, proportional to percentage signal change) from individual peaks in contralateral calcarine sulcus and contralateral Heschl's gyrus are plotted in Figure 2, B and C (bar graphs), respectively. In addition to the clear AVC > NC effect, AVC also elicited a higher BOLD signal than the relevant unisensory baseline (i.e., vision for calcarine sulcus, auditory for Heschl's gyrus; each at p < 0.008 or better), whereas the NC condition was significantly lower than those unisensory baselines (p < 0.007 or better).

Comparison of our two multisensory conditions with the unisensory baselines

Although our main focus was on comparing audiovisual correspondence minus noncorrespondence (AVC > NC), the plots in Figure 2 show that AVC also elicited higher activity than either unisensory baseline in mSTS, whereas NC was lower than both these baselines there. This might reflect corresponding auditory and visual events becoming “allies” in neural representation (attributable to their correspondence), whereas noncorresponding instead become “competitors,” leading to the apparent suppression observed for them in mSTS. This might also hold for the A1 and V1 results, in which the most relevant unisensory baseline (audition or vision, respectively) was again significantly below the AVC condition yet significantly above the NC. Alternatively, one might argue that the level of activity for NC in A1 or V1 may correspond to the combined mean of the separate auditory and visual baselines for that particular area (NC did not differ from that mean for V1 and A1, although it did for STS). However, this would still imply that combining noncorresponding sounds and lights can reduce activity in primary sensory cortices relative to the preferred modality alone, although temporally corresponding audiovisual stimulation boosts activity in both V1 and A1.

Our finding of enhanced BOLD signal in the AVC condition, but reduced in the NC, relative to unisensory baselines is reminiscent in some respects of an interesting recent audiotactile (rather than audiovisual) study by Lakatos et al. (2007). Unlike the present whole-brain human fMRI method, they measured lamina-specific multiunit activity (MUA) invasively in macaque primary auditory cortex and calculated current-source density distributions (CSDs). Responses to combined audiotactile stimuli differed from summed unisensory tactile and auditory responses, indicating a modulatory influence of tactile stimuli on auditory. The stimulus onset asynchronies producing either response enhancement or suppression for multisensory stimulation hinted at a phase-resetting mechanism affecting neural oscillations. In particular, because of rapid somatosensory input into supragranular layers of A1 (at ∼8 ms), corresponding auditory signals may arrive (∼9 ms) at an optimal excitable phase (hence producing an enhanced response, potentially analogous to our result for the AVC condition) but may arrive during an opposing non-excitable phase when somatosensory inputs do not correspond (hence producing a depressed response, potentially analogous to our result for the NC condition, although note that enhanced audiotactile CSDs have been observed at various SOAs because of the oscillatory nature of the underlying mechanism). Although some analogies can be drawn to our fMRI results, the sluggish temporal resolution of fMRI (compared with MUA and CSD) precludes any links between our study and that of Lakatos et al. (2007) from being pushed too far. Electroencephalography (EEG) or magnetoencephalography (MEG) might be more suitable for studying the timing and oscillatory nature of the present effects, or our present paradigm could be applied to monkeys during invasive recordings (because the only task required is fixation monitoring). The architecture and timing of possible visual inputs into A1 might differ from those for somatosensory (as studied by Lakatos et al., 2007) and are likely to be slower (because of retinal transduction time), probably too slow to act exactly like the somatosensory inputs in the study by Lakatos et al. Brosch et al. (2005) reported modulated MUAs in A1 starting only at ∼60–100 ms after presentation of a visual stimulus. Ghazanfar et al. (2005) reported their first audiovisual interactions in A1 at ∼80 ms after stimulus (see also below). Finally, here we found fMRI effects not only for A1 but also for V1 and STS. As will be seen below, analyses of effectivity suggested some possible feedback influences from STS on A1 [as had been suggested by Ghazanfar et al. (2005), their pp 5004, 5011] and on V1, for the present fMRI data in our paradigm.

Several different contrasts and analysis approaches have been introduced in previous multisensory research when comparing multisensory conditions with unisensory baselines. Although the present study focuses on fMRI measures, Stein and colleagues conducted many influential single-cell studies on the superior colliculus and other structures (Stein, 1978; Stein and Meredith, 1990; Stein et al., 1993; Wallace et al., 1993, 1996; Wallace and Stein, 1994, 1996, 1997). They suggested that, depending on the relative timing and/or location of multisensory inputs, neural responses can sometimes exceed (or fall below) the sum of the responses for each unisensory input (Lakatos et al., 2007). Nonlinear analysis criteria have also been applied to EEG data in some multisensory studies that typically manipulated presence/absence of costimulation in a second modality (Giard and Peronnet, 1999; Foxe et al., 2000; Fort et al., 2002; Molholm et al., 2002, 2004; Murray et al., 2005) rather than a detailed relationship in temporal patterning as here. Similar nonadditive criteria have even been applied to fMRI data (Calvert et al., 2001). Conversely, such criteria have been criticized for some situations (for ERP contrasts, see Teder-Sälejärvi et al., 2002). Moreover, Stein and colleagues subsequently reported that some of the cellular phenomena that originally inspired such criteria may in fact more often reflect linear rather than nonlinear phenomena (Stein et al., 2004) when considered at the population level. Such considerations have led to proposals of revised criteria for fMRI studies of multisensory integration, including suggestions that a neural response significantly different from the maximal unisensory response may be taken to signify a multisensory effect (Beauchamp, 2005b). Most importantly, we note that the critical fMRI results reported here cannot merely reflect summing (or averaging) of two entirely separate BOLD responses to incoming auditory and visual events, because otherwise the outcome should have been comparable for corresponding and noncorresponding conditions. Recall that the auditory and visual stimuli themselves were equivalent and fully counterbalanced across our AVC and NC conditions; only their temporal relationship varied. Hence, our critical effects must reflect multisensory effects that depend on the temporal correspondence of incoming auditory and visual temporal patterns.

Analysis of functional changes in effective connectivity between brain areas

Given the activation results, we seeded our PPI connectivity analysis at mSTS (Fig. 4D, blue region) in a spherical region (4 mm diameter) surrounding the maximum found in the main analyses for each individual (for coordinates of the group average, see Fig. 4A). This PPI analysis revealed that functional coupling of seeded mSTS, contralateral to the crossmodal coincidence, was specifically enhanced (showed stronger covariation) with early visual cortex (mean peak coordinates, ±4, −82, 6; p < 0.008) (Fig. 4E) and auditory cortex (±44, 22, 6; p < 0.02) (Fig. 4F) ipsilaterally to the mSTS seed in the context of audiovisual coincidence (vs noncoincidence). This modulation is not redundant with the overall BOLD activations reported above, because it reflects condition-dependent covariation between brain regions, after mean activations by condition for each region have been discounted (see Materials and Methods) (Friston et al., 1997). Nevertheless, these connectivity results closely resembled the activation pattern in terms of the brain regions implicated (Fig. 4, compare A–C with D–F; see also Fig. 2), providing additional evidence to highlight an mSTS–A1–V1 interconnected network for the present effects of audiovisual temporal correspondence in our paradigm. The highly specific pattern of condition-dependent functional coupling with mSTS was found in visual cortex for all 24 individual subjects and in auditory cortex for 23 of 24 subjects (for a representative subject, see Fig. 5) (for every single individual, see supplemental data, available at www.jneurosci.org as supplemental material).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

A–C, Combined results of corresponding minus noncorresponding audiovisual stimulation for LVF and RVF groups, with hemisphere flipping to pool results contralateral to the audiovisual coincidence, which thereby appear in the apparently left hemisphere here (see Materials and Methods). Overall activations for AVC > NC in the following: A, contralateral mSTS; B, contralateral early visual cortex; C, contralateral auditory cortex. D–F, Enhanced functional coupling of seeded mSTS (this seeded region shown in D as filled blue circle) with visual (E) and auditory (F) areas in the context of audiovisual temporal coincidence versus noncoincidence. Voxels showing significantly greater functional coupling with the STS seed for that context are highlighted in red.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Overlap of observed activation, for audiovisual correspondence minus noncorrespondence with functional coupling results from one illustrative participant, with STS, calcarine fissure, and Heschl's gyrus draw in blue onto the individual's structural scan. STS activation was used as the seed for the PPI analysis, whereas regions in Heschl's gyrus and calcarine fissure show both increased activation for audiovisual correspondence minus noncorrespondence and also enhanced coupling with ipsilateral STS in the context of audiovisual temporal correspondence [this overlap was formally tested for each individual by a conjunction of PPI results and experimental AVC minus NC within areas that showed a sensory-specific effect (visual > auditory and auditory > visual, respectively)].

Although several studies also implicate a role for multisensory thalamic nuclei in cortical response profiles (Baier et al., 2006, Lakatos et al., 2007) and the thalamus is increasingly regarded as a potentially major player in multisensory phenomena (Jones, 1998; Schroeder et al., 2003; Fu et al., 2004; Ghazanfar and Schroeder, 2006), we did not observe any BOLD effects in the thalamus with the human fMRI method used here, only cortically. Although fMRI may not be an ideal method for detecting any effects in the thalamus (particularly if subtle or layer specific), this does not undermine our positive cortical findings. Moreover, new fMRI methods are now being developed to enhance sensitivity to subcortical thalamic structures (Schneider et al., 2004) and, as noted previously, our new paradigm may also be suitable for invasive animal work in the future because the only task required is fixation monitoring.

Because PPI analyses are nondirectional in nature (see Materials and Methods), we further assessed possible influences between mSTS, V1, and A1 for the present fMRI data using a DIT measure (Hinrichs et al., 2006). Data from A1 and V1 were derived from the subject-specific maxima for overlap between the basic activation analysis and the PPI analysis (mean coordinates: V1, ±8.9, −78.4, 6.9; A1, ±47.6, 19.7, 7.1). Inferred information flow from mSTS toward V1 and toward A1 was significantly higher than the opposite direction during audiovisual temporal coincidence (p < 0.05 in both cases) (Fig. 6) relative to temporal noncoincidence. No reliable condition-specific differences were found for any direct A1–V1 influences.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Results of directed information transfer analysis for temporally corresponding minus noncorresponding audiovisual conditions (i.e., difference in the inferred directional information transfer, attributable to condition) between STS (indicated schematically with purple circle), calcarine fissure (V1, indicated schematically with red circle), and Heschl's gyrus (A1, schematic blue circle), with direction of information transfer indicated via colored arrows. Numbers by each arrow indicate the measured change in influence (larger = stronger for temporally corresponding than noncorresponding condition) in the direction of each colored arrow. Colored brackets link pairs of numbers showing significant differences between the impact of condition, indicating that one direction of influence changed more than the converse direction attributable to temporal correspondence (p < 0.05 or better). White brackets indicate no such significant differences [nonsignificant (n.s.)]. The absolute values for DIT measures matter less than the reliability of any differences because absolute values can depend on imaging parameters (Hinrichs et al., 2006).

Thus, visual and auditory cortices not only showed activation by audiovisual temporal correspondence in the present fMRI data; over and above this, they also showed some functional coupling with mSTS, as confirmed when seeding the PPI analysis there revealed condition-specific effective connectivity with A1 and V1. Moreover, DIT analysis suggested a significantly increased influence from mSTS on A1 and V1 specifically during audiovisual temporal correspondence, rather than direct A1–V1 influences, for these fMRI data.

As noted previously, possible thalamic influences on multisensory effects (Lakatos et al., 2007) may also need to be considered and may not be readily detected with human fMRI, although this does not undermine the positive evidence we did find for feedback influences from mSTS on A1 [as also hypothesized by Ghazanfar et al. (2005), their pp 5004, 5011]. More generally, the relative balance between bottom-up multisensory influences (e.g., via the thalamus, or cortical–cortical as between A1 and V1) and top-down feedback influences (as suggested here, by the DIT analysis, for STS influences on A1 and V1 in the AVC condition) may depend on the paradigm used. It is possible that, in our paradigm, temporal correspondence between auditory and visual streams tended to attract some attention to those streams, which might favor feedback influences such as the DIT effect we observed from STS on A1 and V1. Conversely, the AVC condition did not activate those brain structures (including parietal cortex and more anterior regions) that are classically associated with attention capture (Corbetta and Shulman, 2002; Watkins et al., 2007). Moreover, performance of the central task (which was off ceiling at ∼83% correct) did not suffer in the AVC condition, whereas an attention-capture account might have predicted a decrement for that. Nevertheless, we return to consider the possible attention issue below.