Cross-modal processing of auditory–visual stimuli in a no-task paradigm: A topographic event-related potential study
Introduction
One main ability of the brain is to combine information from anatomically different sensory pathways to form unified percepts of the external world. There has recently been increased interest in studying the neurophysiological correlates of multisensory integration using various methodological approaches in normal adults. Yet, it may be fundamental to study these phenomenons in young children for whom interaction of vision and audition may play an important role in language acquisition for example, and in children with developmental communication disorders such as autism. Among the different methodological approaches that have been used in multisensory research, event-related potentials (ERPs) have a double advantage: first, they have an excellent time resolution, thus allowing to characterize cross-modal interactions in the early steps of sensory analysis; second, they are non-invasive and tolerant to body/head movements, and thus seem to be a most adapted technique to study the maturation of cross-modal processing in children and young patients.
To date, all the electrophysiological studies on auditory–visual integration in humans have used paradigms in which the subjects were required to perform active tasks such as discrimination, simple detection or localization of multisensory events (Fort et al., 2002a, Fort et al., 2002b, Giard and Peronnet, 1999, Molholm et al., 2002, Molholm et al., 2004, Schroger and Widmann, 1998, Teder-Salejarvi et al., 2002). Yet, demanding an active task may not be possible for particular populations such as young children or severely impaired patients. Here we propose a paradigm using various and attractive stimuli to focus children’s attention even if they have no specific task to perform.
Such a paradigm may also correspond to everyday life experience in which we implicitly process events stimulating several of our senses simultaneously. Previous ERP studies on multisensory integration have shown that the mechanisms of integration are multiple and that different interaction effects can be observed according to the nature of the task at hand (Fort and Giard, 2004). It is therefore likely that in passive conditions of perception, that is, when no specific task is required on the stimuli, interactions between visual and auditory processing, if they exist, should differ from those previously described with active tasks. Although several functional neuroimaging studies have shown cross-modal effects in various brain structures (superior colliculus, superior temporal and intraparietal sulci, insula and/or frontal lobe) when subjects were passively exposed to bimodal events (Calvert et al., 2001, Foxe et al., 2002), the variability of the stimuli used together with the lack of temporal information on the effects observed preclude direct comparison of these findings with ERP data. On the other hand, using ERPs Foxe et al. (2000) have reported early cross-modal interactions during passive perception of auditory–somatosensory events. These studies have thus shown that multisensory integration may occur even when subjects do not have to respond actively to bimodal targets.
The present ERP study will analyze the patterns of auditory–visual interactions generated during a no-task paradigm in adults with two main objectives: (i) enrich our knowledge on the multiple forms of cross-modal processing by characterizing possibly other interaction effects than those previously described with active tasks; and (ii) validate a paradigm and provide normative data in adults before using the same protocol to explore cross-modal processing in normal and pathological brain maturation.
Section snippets
Subjects
Eighteen healthy young adults (ten females) took part in the experiment (mean age: 25.2 ± 0.6). All were right-handed as assessed by the Edinburgh handedness inventory (Oldfield, 1971) and had normal hearing and normal or corrected-to-normal vision.
All subjects gave informed written consent for their participation in the protocol which had previously been approved by the local Research Ethics Committee.
The results from one subject (male) were excluded because there were too many movement
ERP waveforms
Fig. 1 presents group-averaged ERPs to unimodal (auditory and visual) and bimodal stimuli over 500 ms at a subset of selected electrode sites where the potentials were recorded at the largest amplitude.
Auditory ERPs were characterized by three successive prominent waves. The negative N1 wave (audN1) peaked around 100 ms post-stimulus at fronto-central sites (109 ± 3 ms; −6.1 ± 0.6 μV at Cz), with polarity reversal at mastoid electrodes. The audN1 wave was followed by the positive P2 wave (audP2)
Discussion
The present results provided electrophysiological evidence of auditory–visual interactions during automatic processing of bimodal events: i.e. reduced auditory N1 and enhanced auditory P2 responses, increased amplitude of the late visual response and new activity in right fronto-temporal regions for bimodal compared to unimodal responses. This indicates that even when no task was required from the subject, cross-modal interactions occurred in multiple cortical areas at several stages of sensory
Acknowledgements
This research was supported by grants from INSERM, the Region Centre, the Bretonneau Hospital of Tours and the Foundation France Telecom.
We thank Doreen Raine for helpful comments on the English.
References (48)
- et al.
The spatiotemporal organization of auditory, visual, and auditory–visual evoked potentials in rat cortex
Brain Res
(1995) - et al.
Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey
Exp Neurol
(1977) - et al.
Detection of audio–visual integration sites in humans by application of electrophysiological criteria to the bold effect
Neuroimage
(2001) - et al.
Effects of flash mode and intensity on p2 component latency and amplitude
Int J Psychophysiol
(2005) - et al.
A review of the evidence for p2 being an independent component process: age, sleep and modality
Clin Neurophysiol
(2004) - et al.
Early auditory–visual interactions in human cortex during nonredundant target identification
Brain Res Cogn Brain Res
(2002) - et al.
Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping
Brain Res Cogn Brain Res
(2000) - et al.
Neuromagnetic source localization of auditory evoked fields and intracerebral evoked potentials: a comparison of data in the same patients
Clin Neurophysiol
(2001) - et al.
Inversion and contrast polarity reversal affect both encoding and recognition processes of unfamiliar faces: a repetition study using erps
Neuroimage
(2002) - et al.
Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study
Brain Res Cogn Brain Res
(2002)
The assessment and analysis of handedness: the edinburgh inventory
Neuropsychologia
Event-related potentials recorded from the scalp and nasopharynx. I. N1 and p2
Electroencephalogr Clin Neurophysiol
Spherical splines for scalp potential and current density mapping
Electroencephalogr Clin Neurophysiol
Multisensory convergence in calcarine visual areas in macaque monkey
Int J Psychophysiol
Combined perception of emotion in pictures and musical sounds
Brain Res
An analysis of audio–visual crossmodal integration by means of event-related potential (erp) recordings
Brain Res Cogn Brain Res
Bimodal speech: early suppressive visual effects in human auditory cortex
Eur J Neurosci
Interest and validity of the additive model in electrophysiological studies of multisensory interactions
Cogn Process
Neural correlates of auditory–visual stimulus onset asynchrony detection
J Neurosci
Neural correlates of cross-modal binding
Nat Neurosci
Crossmodal processing in the human brain: insights from functional neuroimaging studies
Cereb Cortex
Response amplification in sensory-specific cortices during crossmodal binding
Neuroreport
A multimodal cortical network for the detection of changes in the sensory environment
Nat Neurosci
Anatomical evidence of multimodal integration in primate striate cortex
J Neurosci
Cited by (48)
Audiovisual integration in children with cochlear implants revealed through EEG and fNIRS
2023, Brain Research BulletinMultiple phases of cross-sensory interactions associated with the audiovisual bounce-inducing effect
2020, Biological PsychologyThe spatial reliability of task-irrelevant sounds modulates bimodal audiovisual integration: An event-related potential study
2016, Neuroscience LettersCitation Excerpt :We provide a discussion comparing the integration processing audiovisual stimuli in the two experiments and discuss the effect of spatial reliability of sound on bimodal audiovisual integration. Sixteen university students (10 male students and 6 female students; mean age 22.6 [21–26] years) participated in the present study. All of the participants had normal or corrected-to-normal vision and hearing capabilities.
The multisensory function of the human primary visual cortex
2016, NeuropsychologiaCitation Excerpt :The application of distributed source estimations (and statistical analyses thereof) further allowed Cappe et al. (2010) to localise early nonlinear effects to primary visual cortices, primary auditory cortices, as well as the posterior superior temporal sulcus (see also Raij et al. (2010) for MEG findings using a distributed inverse solution). These results resolve an ongoing debate regarding whether or not the early effects emanate from nominally visual (Fort et al., 2002a; Molholm et al., 2002) or nominally auditory cortices (Vidal et al., 2008) or both (Raij et al., 2010; Senkowski et al., 2007; Teder-Sälejärvi et al., 2002). Specifically, the findings of Cappe et al. and Raij et al. show there to be a widely distributed network of functionally coupled brain regions operating in concert during the time period of these early multisensory interactions.
The temporal reliability of sound modulates visual detection: An event-related potential study
2015, Neuroscience LettersCitation Excerpt :A major effect was found at the latency of 180–240 ms over the central area that extended to the peripheral electrodes (Fig. 3a). A similar ERP component was observed around 200 ms at a central area in some previous studies on AV integration [7,15–17]. Talsma and Woldoff reported that the ERP component was only elicited when AV stimuli were attended.
Audiovisual congruency and incongruency effects on auditory intensity discrimination
2015, Neuroscience LettersCitation Excerpt :On the other hand, when auditory and visual stimuli are incongruent, perception in one modality could be biased by the other, e.g., the McGurk effect [5–10]. Beside behavioral evidences, the underlying neural substrates for audiovisual interaction phenomena have also been extensively studied by neuroimaging [11–15] or electrophysiological [16] techniques. However, previous audiovisual studies mainly focused on cross-modal interactions within single synchronously presented audiovisual stimuli, regardless of the influence from the preceding stimulations.