Research reportAn analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings
Introduction
The ability to integrate stimuli in different modalities to form unified percepts is a fundamental component of sensory guided behavior and cognition. Psychophysical studies have shown that behavioral responses to multimodal inputs presented in close spatial and temporal proximity are typically faster and more accurate than those made to the unimodal stimuli alone [19], [15]. The neuronal bases for this cross-modal enhancement of perception and behavior is being actively investigated in both animals and humans (reviewed in Refs. [17], [3]), but our understanding of the underlying mechanisms is still at a preliminary level. A common paradigm for demonstrating cross-modal facilitation has been to compare neural responses to each of two unimodal stimuli presented alone with the response to the bimodal combination of the two [17]. If the bimodal response is greater than the sum of the individual unimodal responses, it may be concluded that some form of facilitatory intermodal integration is taking place. Using this type of design, evidence for cross-modal enhancement has been demonstrated in neurophysiological recordings from multisensory neurons in a variety of brain structures including the superior colliculus, posterior parietal cortex, superior temporal cortex, and neurostriatum [5] (reviewed in Ref. [17]).
In humans, cross-modal enhancement of neural activity has been demonstrated using both electrophysiological recordings of event-related brain potentials [6], [5] and neuroimaging [3] measures. One of the advantages of ERP recordings is that the precise time course of cross-modal integration can be studied in different brain areas. However, a potential problem may arise when subjects are required to attend to the stimuli in the bimodal–unimodal comparison paradigm. For example, in a study of auditory (A)/visual (V) integration, evidence for cross-modal interaction would be obtained when the amplitude of the bimodal (AV) ERP differs from the sum of the individual unimodal ERPs (A+V). Such an interaction would be evident as voltage deflections in the AV−(A+V) difference ERP. The problem comes in if anticipatory slow potentials are elicited in association with each of the three types of stimulus events (A, V, and AV). Such potentials would be added only once but subtracted twice in the AV−(A+V) difference wave, creating a deflection that might be mistaken for a true auditory–visual interaction. For example, an anticipatory ramp-like negativity (such as the CNV [18]) that arose before each stimulus and continued for a time after stimulus onset might appear as a ramp-like positivity in the AV−(A+V) difference wave, due to its having been subtracted twice. Depending on how the difference wave was baselined, such an anticipatory slow wave could appear as a significant voltage deflection at a very short latency after stimulus onset. The principal aim of the present study was to separate the possible influences of anticipatory slow waves from true cross-modal interactions in processing as reflected in the AV−(A+V) difference wave. This separation was achieved by showing, first, that slow wave deflections in the difference wave became significant at shorter latencies when earlier baseline periods were chosen for the measurements and, second, that dipole modeling can distinguish the sources of these early slow-wave effects from later, genuine cross-modal interactions. In addition, phasic cross-modal interactions were separated from anticipatory slow potentials by means of high-pass filtering of the AV−(A+V) difference waves. A secondary aim was to estimate the locations of the brain areas in which cross-modal interactions take place using dipole modeling.
Section snippets
Participants
Fifteen healthy adults (eight female; ages 19–29 years, mean age 21.1 years) participated in this study after giving written informed consent. Each participant had normal or corrected-to-normal vision and was tested in the laboratory to confirm normal hearing.
Stimuli and apparatus
The experiment was conducted in a sound-attenuated chamber with a background sound level of 32 dB (A) and a background luminance of 2 cd/m2. Participants sat in the chamber and faced a loudspeaker with a red light-emitting diode (LED)
Accuracy
Overall, only about 60% of the auditory targets were correctly responded to during the experimental runs, a significant difference compared to about 85% correctly identified visual targets (t[14]=6.17, P<0.0001). The bimodal targets were correctly identified at a significantly higher rate (92%) than either the auditory targets (t[14]=6.25, P<0.0001), or the visual targets alone (t[14]=3.48, P<0.004). False alarm rates were very low, 2.52% for auditory, 2.89% for visual, and 6.56% for
Discussion
The present results provide evidence that slow potentials beginning prior to stimulus onset can contribute to the bimodal minus unimodal difference waveform (AV−(A+V)) that is often taken as an index of cross-modal interactions in neural processing. In this study, the slow anticipatory potentials were found to be positive over the anterior scalp and negative over the posterior scalp and probably belong to the CNV family of ERPs that precede perceptual decisions and discriminative responses [8].
Acknowledgements
This study was supported by a grant from the National Institute for Mental Health (MH 25594). The authors thank Daniel R. Heraldez and Matthew M. Marlow for technical assistance.
References (20)
- et al.
Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex
Curr. Biol.
(2000) - et al.
Multisensory auditory–somatosensory interactions in early cortical processing revealed by high-density electrical mapping
Cogn. Brain Res.
(2000) - et al.
On the processing of spatial frequencies as revealed by evoked-potential source modeling
Clin. Neurophysiol.
(2000) - et al.
ERP effects of spatial attention and display search with unilateral and bilateral stimulus displays
Biol. Psychol.
(1999) - et al.
Integrating electrophysiology and neuroimaging of spatial selective attention to simple isolated visual stimuli
Vis. Res.
(2001) Guidelines for standard electrode position nomenclature
J. Clin. Neurophysiol
(1994)- et al.
Spatio-temporal dynamics of attention to color: evidence from human electroencephalography
Hum. Brain Mapping
(1998) - et al.
Spatial selective attention affects early extrastriate but nor striate components of the visual evoked potential
J. Cogn. Neurosci.
(1996) - et al.
Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study
J. Cogn. Neurosci.
(1999) - et al.
Combined spatial and temporal imaging of brain activity during visual selective attention in humans
Nature
(1994)