Cross-modal processing of auditory–visual stimuli in a no-task paradigm: A topographic event-related potential study

https://doi.org/10.1016/j.clinph.2007.11.178Get rights and content

Abstract

Objective

The aim of this study was to investigate in healthy adults the electrophysiological correlates of auditory–visual interactions involved in perception of bimodal events in a no-task paradigm.

Methods

Event-related potentials were recorded in response to unimodal auditory (A), unimodal visual (V) and bimodal (AV) stimuli. Cross-modal interactions were estimated using the additive [AV  (A + V)] model.

Results

The spatio-temporal analysis of ERPs and scalp current densities revealed several interaction patterns occurring at both early and late stages of sensory cortical processing: (1) amplitude decrease of the unimodal auditory N1 wave as early as 55 ms, (2) amplitude increase of the unimodal auditory P2 wave from 150 to 195 ms concomitant with new neural activity over the right fronto-temporal region, and (3) amplitude increase of the late unimodal visual response within 245–350 ms post-stimulus.

Conclusions

Our study provides evidence that several patterns of cross-modal interactions can be generated even if no task is required from subjects.

Significance

The paradigm used here can thus be utilized for studying the maturation of the cross-modal processes in young children and in children with pathological development.

Introduction

One main ability of the brain is to combine information from anatomically different sensory pathways to form unified percepts of the external world. There has recently been increased interest in studying the neurophysiological correlates of multisensory integration using various methodological approaches in normal adults. Yet, it may be fundamental to study these phenomenons in young children for whom interaction of vision and audition may play an important role in language acquisition for example, and in children with developmental communication disorders such as autism. Among the different methodological approaches that have been used in multisensory research, event-related potentials (ERPs) have a double advantage: first, they have an excellent time resolution, thus allowing to characterize cross-modal interactions in the early steps of sensory analysis; second, they are non-invasive and tolerant to body/head movements, and thus seem to be a most adapted technique to study the maturation of cross-modal processing in children and young patients.

To date, all the electrophysiological studies on auditory–visual integration in humans have used paradigms in which the subjects were required to perform active tasks such as discrimination, simple detection or localization of multisensory events (Fort et al., 2002a, Fort et al., 2002b, Giard and Peronnet, 1999, Molholm et al., 2002, Molholm et al., 2004, Schroger and Widmann, 1998, Teder-Salejarvi et al., 2002). Yet, demanding an active task may not be possible for particular populations such as young children or severely impaired patients. Here we propose a paradigm using various and attractive stimuli to focus children’s attention even if they have no specific task to perform.

Such a paradigm may also correspond to everyday life experience in which we implicitly process events stimulating several of our senses simultaneously. Previous ERP studies on multisensory integration have shown that the mechanisms of integration are multiple and that different interaction effects can be observed according to the nature of the task at hand (Fort and Giard, 2004). It is therefore likely that in passive conditions of perception, that is, when no specific task is required on the stimuli, interactions between visual and auditory processing, if they exist, should differ from those previously described with active tasks. Although several functional neuroimaging studies have shown cross-modal effects in various brain structures (superior colliculus, superior temporal and intraparietal sulci, insula and/or frontal lobe) when subjects were passively exposed to bimodal events (Calvert et al., 2001, Foxe et al., 2002), the variability of the stimuli used together with the lack of temporal information on the effects observed preclude direct comparison of these findings with ERP data. On the other hand, using ERPs Foxe et al. (2000) have reported early cross-modal interactions during passive perception of auditory–somatosensory events. These studies have thus shown that multisensory integration may occur even when subjects do not have to respond actively to bimodal targets.

The present ERP study will analyze the patterns of auditory–visual interactions generated during a no-task paradigm in adults with two main objectives: (i) enrich our knowledge on the multiple forms of cross-modal processing by characterizing possibly other interaction effects than those previously described with active tasks; and (ii) validate a paradigm and provide normative data in adults before using the same protocol to explore cross-modal processing in normal and pathological brain maturation.

Section snippets

Subjects

Eighteen healthy young adults (ten females) took part in the experiment (mean age: 25.2 ± 0.6). All were right-handed as assessed by the Edinburgh handedness inventory (Oldfield, 1971) and had normal hearing and normal or corrected-to-normal vision.

All subjects gave informed written consent for their participation in the protocol which had previously been approved by the local Research Ethics Committee.

The results from one subject (male) were excluded because there were too many movement

ERP waveforms

Fig. 1 presents group-averaged ERPs to unimodal (auditory and visual) and bimodal stimuli over 500 ms at a subset of selected electrode sites where the potentials were recorded at the largest amplitude.

Auditory ERPs were characterized by three successive prominent waves. The negative N1 wave (audN1) peaked around 100 ms post-stimulus at fronto-central sites (109 ± 3 ms; −6.1 ± 0.6 μV at Cz), with polarity reversal at mastoid electrodes. The audN1 wave was followed by the positive P2 wave (audP2)

Discussion

The present results provided electrophysiological evidence of auditory–visual interactions during automatic processing of bimodal events: i.e. reduced auditory N1 and enhanced auditory P2 responses, increased amplitude of the late visual response and new activity in right fronto-temporal regions for bimodal compared to unimodal responses. This indicates that even when no task was required from the subject, cross-modal interactions occurred in multiple cortical areas at several stages of sensory

Acknowledgements

This research was supported by grants from INSERM, the Region Centre, the Bretonneau Hospital of Tours and the Foundation France Telecom.

We thank Doreen Raine for helpful comments on the English.

References (48)

  • R.C. Oldfield

    The assessment and analysis of handedness: the edinburgh inventory

    Neuropsychologia

    (1971)
  • N. Perrault et al.

    Event-related potentials recorded from the scalp and nasopharynx. I. N1 and p2

    Electroencephalogr Clin Neurophysiol

    (1984)
  • F. Perrin et al.

    Spherical splines for scalp potential and current density mapping

    Electroencephalogr Clin Neurophysiol

    (1989)
  • K.S. Rockland et al.

    Multisensory convergence in calcarine visual areas in macaque monkey

    Int J Psychophysiol

    (2003)
  • K.N. Spreckelmeyer et al.

    Combined perception of emotion in pictures and musical sounds

    Brain Res

    (2006)
  • W.A. Teder-Salejarvi et al.

    An analysis of audio–visual crossmodal integration by means of event-related potential (erp) recordings

    Brain Res Cogn Brain Res

    (2002)
  • J. Besle et al.

    Bimodal speech: early suppressive visual effects in human auditory cortex

    Eur J Neurosci

    (2004)
  • J. Besle et al.

    Interest and validity of the additive model in electrophysiological studies of multisensory interactions

    Cogn Process

    (2004)
  • K.O. Bushara et al.

    Neural correlates of auditory–visual stimulus onset asynchrony detection

    J Neurosci

    (2001)
  • K.O. Bushara et al.

    Neural correlates of cross-modal binding

    Nat Neurosci

    (2003)
  • G.A. Calvert

    Crossmodal processing in the human brain: insights from functional neuroimaging studies

    Cereb Cortex

    (2001)
  • G.A. Calvert et al.

    Response amplification in sensory-specific cortices during crossmodal binding

    Neuroreport

    (1999)
  • J. Downar et al.

    A multimodal cortical network for the detection of changes in the sensory environment

    Nat Neurosci

    (2000)
  • A. Falchier et al.

    Anatomical evidence of multimodal integration in primate striate cortex

    J Neurosci

    (2002)
  • Cited by (48)

    • The spatial reliability of task-irrelevant sounds modulates bimodal audiovisual integration: An event-related potential study

      2016, Neuroscience Letters
      Citation Excerpt :

      We provide a discussion comparing the integration processing audiovisual stimuli in the two experiments and discuss the effect of spatial reliability of sound on bimodal audiovisual integration. Sixteen university students (10 male students and 6 female students; mean age 22.6 [21–26] years) participated in the present study. All of the participants had normal or corrected-to-normal vision and hearing capabilities.

    • The multisensory function of the human primary visual cortex

      2016, Neuropsychologia
      Citation Excerpt :

      The application of distributed source estimations (and statistical analyses thereof) further allowed Cappe et al. (2010) to localise early nonlinear effects to primary visual cortices, primary auditory cortices, as well as the posterior superior temporal sulcus (see also Raij et al. (2010) for MEG findings using a distributed inverse solution). These results resolve an ongoing debate regarding whether or not the early effects emanate from nominally visual (Fort et al., 2002a; Molholm et al., 2002) or nominally auditory cortices (Vidal et al., 2008) or both (Raij et al., 2010; Senkowski et al., 2007; Teder-Sälejärvi et al., 2002). Specifically, the findings of Cappe et al. and Raij et al. show there to be a widely distributed network of functionally coupled brain regions operating in concert during the time period of these early multisensory interactions.

    • The temporal reliability of sound modulates visual detection: An event-related potential study

      2015, Neuroscience Letters
      Citation Excerpt :

      A major effect was found at the latency of 180–240 ms over the central area that extended to the peripheral electrodes (Fig. 3a). A similar ERP component was observed around 200 ms at a central area in some previous studies on AV integration [7,15–17]. Talsma and Woldoff reported that the ERP component was only elicited when AV stimuli were attended.

    • Audiovisual congruency and incongruency effects on auditory intensity discrimination

      2015, Neuroscience Letters
      Citation Excerpt :

      On the other hand, when auditory and visual stimuli are incongruent, perception in one modality could be biased by the other, e.g., the McGurk effect [5–10]. Beside behavioral evidences, the underlying neural substrates for audiovisual interaction phenomena have also been extensively studied by neuroimaging [11–15] or electrophysiological [16] techniques. However, previous audiovisual studies mainly focused on cross-modal interactions within single synchronously presented audiovisual stimuli, regardless of the influence from the preceding stimulations.

    View all citing articles on Scopus
    View full text