Trends in Neurosciences
Volume 31, Issue 8, August 2008, Pages 401-409
Journal home page for Trends in Neurosciences

Review
Crossmodal binding through neural coherence: implications for multisensory processing

https://doi.org/10.1016/j.tins.2008.05.002Get rights and content

Picture yourself on a crowded sideway with people milling about. The acoustic and visual signals generated by the crowd provide you with complementary information about their locations and motion which needs to be integrated. It is not well understood how such inputs from different sensory channels are combined into unified perceptual states. Coherence of oscillatory neural signals might be an essential mechanism supporting multisensory perception. Evidence is now emerging which indicates that coupled oscillatory activity might serve to link neural signals across uni- and multisensory regions and to express the degree of crossmodal matching of stimulus-related information. These results argue for a new view on multisensory processing which considers the dynamic interplay of neural populations as a key to crossmodal integration.

Introduction

The inputs delivered by different sensory organs provide us with both complementary and redundant information about the environment. Constantly, multisensory interactions occur in the brain to evaluate whether there is a matching of the information arriving through different channels, or whether the signals give rise to conflict and need to be processed separately. The outcome of these interactions is of critical importance for perception, cognitive processing and the control of action 1, 2, 3, 4. Recent studies have revealed that a vast number of cortical operations, including those carried out by primary regions, are shaped by inputs from multiple sensory modalities 4, 5, 6.

In recent years, an increasing number of studies have aimed at characterizing multisensory cortical regions, revealing multisensory processing in the superior temporal sulcus, the intraparietal sulcus and frontal regions, as well as insula and claustrum 4, 6, 7. Interestingly, there is increasing evidence that neurons in areas formerly considered as unisensory like, such as auditory belt areas 3, 4, 6, 8, 9, can also exhibit multisensory characteristics. Furthermore, numerous subcortical structures are involved in multisensory processing. In addition to the superior colliculus [1], these include the striatum [10], the cerebellum [11] and the amygdala [12], as well as nuclei of the thalamus [13].

Whereas the ubiquity and fundamental relevance of multisensory processing are becoming increasingly clear, the neural mechanisms underlying crossmodal interactions are much less well understood. In this article, we review recent studies that cast new light on this issue. Whereas classical studies have postulated a feedforward convergence of unimodal signals as the primary mechanism for multisensory integration 1, 2, there is now evidence that both feedback and lateral interactions are also relevant 4, 6, 14, 15. Beyond this changing view on the anatomical substrate of multisensory interactions, there is increasing awareness that complex dynamic interactions of cell populations, leading to coherent oscillatory firing patterns, might play a key role in mediating cross-systems integration in the brain 16, 17, 18, 19, 20, 21. Here we consider the hypothesis that synchronized oscillations (see Glossary) might provide a potential mechanism for crossmodal integration and for the selection of information that matches across different sensory channels.

Section snippets

Views on crossmodal integration

The classical view posits that multisensory integration occurs in a hierarchical manner by progressive convergence of pathways. In this view, sensory signals are merged only in higher association areas and specialized subcortical regions 1, 2. A core assumption of this approach is that perceptual information is primarily encoded in the firing rate of the cells involved. Multisensory integration, accordingly, is expressed by firing rate changes in neural populations receiving convergent inputs

Oscillatory activity in crossmodal processing

A variety of different paradigms have been used to study the role of oscillatory responses and neural coherence during multisensory processing (Table 1). Most studies have been performed in humans using EEG or MEG (see Glossary), with relatively few animal studies currently available. The approaches used address different aspects of multisensory processing, including (i) bottom-up processing of multisensory stimuli, (ii) crossmodally induced perceptual changes, (iii) modulation by top-down

Functional role of neural synchrony for crossmodal interactions

The data available so far support the hypothesis that coherence of oscillatory responses might play a crucial role in multisensory processing. They show that multisensory interactions can be accompanied by condition-specific changes in oscillatory responses which often, albeit not always, occur in the gamma band 28, 30, 32, 33, 34, 35, 37, 40, 41, 45, 46, 47, 48. The effects observed in EEG or MEG signals are likely to result not only from changes in oscillatory power but also in the phase

Outlook

Clearly, the hypothesis that neural synchrony might play a role in multisensory processing requires further experimental testing (Box 2). Thus far, only a relatively small number of multisensory studies have used coherence measures to explicitly address interactions across different neural systems. Substantial progress will require studies in humans with approaches suitable to capture dynamic cross-systems interactions in source space 53, 54. Furthermore, to characterize the role of correlated

Acknowledgements

We thank Markus Siegel, Peter König, Jörg Hipp and Gernot Supp for helpful comments on the manuscript. D.S. received support from a NARSAD 2006 young investigator award and the German Research Foundation (SE 1859/1–1). J.J.F. received support from a U.S. National Institute of Mental Health grant (NIMH–RO1 MH65350). A.K.E. acknowledges support by the European Union (IST-027268, NEST-043457), the German Research Foundation (GRK 1247/1), the German Federal Ministry of Education and Research

Glossary

Electroencephalography (EEG)
The standard noninvasive method to measure electrical potential changes arising from the brain. The EEG signal is recorded on the scalp surface and reflects the summed postsynaptic activity in the underlying cortical regions. A key advantage is the high temporal resolution of the method.
Evoked oscillatory activity
Oscillatory neuronal activity that is phase locked to the onset of eliciting events (e.g. sensory inputs). Evoked oscillatory activity can be quantified

References (78)

  • O.W. Sakowitz

    Bisensory stimulation increases gamma-responses over multiple cortical regions

    Brain Res. Cogn. Brain Res.

    (2001)
  • O.W. Sakowitz

    Spatio-temporal frequency characteristics of intersensory components in audiovisual evoked potentials

    Brain Res. Cogn. Brain Res.

    (2005)
  • P. Lakatos

    Neuronal oscillations and multisensory interaction in primary auditory cortex

    Neuron

    (2007)
  • J. Kaiser

    Gamma-band activity over early sensory areas predicts detection of changes in audiovisual speech stimuli

    Neuroimage

    (2006)
  • D. Senkowski

    Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations

    Neuropsychologia

    (2007)
  • K.M. Fu

    Attention-dependent suppression of distracter visual input can be cross-modally cued as indexed by anticipatory parieto-occipital alpha-band oscillations

    Brain Res. Cogn. Brain Res.

    (2001)
  • P.J. Uhlhaas et al.

    Neural synchrony in brain disorders: relevance for cognitive dysfunctions and pathophysiology

    Neuron

    (2006)
  • L.A. Ross

    Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions

    Schizophr. Res.

    (2007)
  • E. Basar

    Brain oscillations in perception and memory

    Int. J. Psychophysiol.

    (2000)
  • W. Klimesch

    Upper alpha ERD and absolute power: their meaning for memory performance

    Prog. Brain Res.

    (2006)
  • S. Palva et al.

    New vistas for α-frequency band oscillations

    Trends Neurosci.

    (2007)
  • G.G. Knyazev

    Motivation, emotion, and their inhibitory control mirrored in brain oscillations

    Neurosci. Biobehav. Rev.

    (2007)
  • O. Jensen et al.

    Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer

    Trends Neurosci.

    (2005)
  • M.M. Müller

    Modulation of induced gamma band activity in the human EEG by attention and visual information processing

    Int. J. Psychophysiol.

    (2000)
  • C.E. Schroeder

    Neuronal oscillations and visual amplification of speech

    Trends Cogn. Sci.

    (2008)
  • B.E. Stein et al.

    The Merging of the Senses

    (1993)
  • C. Kayser et al.

    Do early sensory cortices integrate crossmodal information?

    Brain Struct. Funct.

    (2007)
  • A. Amedi

    Functional imaging of human crossmodal identification and object recognition

    Exp. Brain Res.

    (2005)
  • G.A. Calvert

    Crossmodal processing in the human brain: insights from functional neuroimaging studies

    Cereb. Cortex

    (2001)
  • J.J. Foxe

    Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study

    J. Neurophysiol.

    (2002)
  • A. Nagy

    Multisensory integration in the basal ganglia

    Eur. J. Neurosci.

    (2006)
  • O. Baumann et al.

    Neural correlates of coherent audiovisual motion perception

    Cereb. Cortex

    (2007)
  • H. Nishijo

    Topographic distribution of modality-specific amygdalar neurons in alert monkey

    J. Neurosci.

    (1988)
  • Y. Komura

    Auditory thalamus integrates visual inputs into behavioral gains

    Nat. Neurosci.

    (2005)
  • J.J. Foxe et al.

    The case for feedforward multisensory convergence during early cortical processing

    Neuroreport

    (2005)
  • W. Singer et al.

    Visual feature integration and the temporal correlation hypothesis

    Annu. Rev. Neurosci.

    (1995)
  • A.K. Engel

    Dynamic predictions: oscillations and synchrony in top-down processing

    Nat. Rev. Neurosci.

    (2001)
  • E. Salinas et al.

    Correlated neuronal activity and the flow of neural information

    Nat. Rev. Neurosci.

    (2001)
  • T. Womelsdorf

    Modulation of neuronal interactions through neuronal synchronization

    Science

    (2007)
  • Cited by (301)

    View all citing articles on Scopus
    View full text