Elsevier

Hearing Research

Volume 258, Issues 1–2, December 2009, Pages 80-88
Hearing Research

Research paper
Multisensory interactions in primate auditory cortex: fMRI and electrophysiology

https://doi.org/10.1016/j.heares.2009.02.011Get rights and content

Abstract

Recent studies suggest that multisensory integration does not only occur in higher association cortices but also at early stages of auditory processing, possibly in primary or secondary auditory cortex. Support for such early multisensory influences comes from functional magnetic resonance imaging experiments in humans and monkeys. However we argue that the current understanding of neurovascular coupling and of the neuronal basis underlying the imaging signal does not permit the direct extrapolation from imaging data to properties of neurons in the same region. While imaging can guide subsequent electrophysiological studies, only these can determine whether and how neurons in auditory cortices combine information from multiple modalities. Indeed, electrophysiological studies only partly confirm the findings from imaging studies. While recordings of field potentials reveal strong influences of visual or somatosensory stimulation on synaptic activity even in primary auditory cortex, single unit studies find only a small minority of neurons as being influenced by non-acoustic stimuli. We propose the analysis of the information coding properties of individual neurons as one way to quantitatively determine whether the representation of our acoustic environment in (primary) auditory cortex indeed benefits from multisensory input.

Introduction

Our ability to recognize sounds or to understand speech profits considerably from the information provided by the other sensory systems. The best known example for such multisensory benefits of hearing is the cocktail party: with loud music playing and people cheering and chatting we can much better understand somebody speaking when we watch the movements of his lips at the same time. In such situations, the visual information can boost hearing capabilities by an equivalent of about 10–20 dB sound intensity (Ross et al., 2007, Sumby and Polack, 1954), although the exact gain depends on the signal to noise ratio and the kind of target signal itself. However, multisensory input not only improves our understanding of speech, and recent work employing a wide range of behavioral paradigms clearly shows that multisensory stimuli are often processed faster, are easier to recognize and are remembered better than unisensory stimuli (Driver and Spence, 1998, Hershenson, 1962, Lehmann and Murray, 2005, McDonald et al., 2000, Seitz et al., 2006, Vroomen and de Gelder, 2000). As a consequence, multisensory stimulation not only eases our perception during every day interactions, but can also be exploited in learning and rehabilitation programs seeking to improve impoverished sensory capabilities (Oakland et al., 1998, Seitz et al., 2006).

This importance of multisensory input for our ability to hear begs the questions of where and how the brain combines the acoustic and non-acoustic information. Earlier studies had found little evidence for multisensory interactions at early stages of processing and promoted a hierarchical view, suggesting that sensory information converges only in higher association areas such as the superior temporal and intra-parietal sulci and regions in the frontal lobe (Benevento et al., 1977, Bruce et al., 1981, Felleman and Van Essen, 1991, Hikosaka et al., 1988, Hyvarinen and Shelepin, 1979, Jones and Powell, 1970). More recent work, in contrast, emphasizes the importance of lower-level regions and suggests that multisensory interactions already occur at the first processing stages in auditory cortex (Foxe and Schroeder, 2005, Ghazanfar and Schroeder, 2006, Schroeder et al., 2004). In the following we will review the evidence for multisensory influences in primate (human and monkey) auditory cortex, and place particular emphasis on the complementary nature of the imaging and electrophysiological approaches typically employed to localize and characterize multisensory interactions.

While the majority of studies reporting multisensory influences are based on functional magnetic resonance imaging (fMRI) experiments, the neuronal basis underlying this signal is still controversial. As we discuss, the uncertain nature of the fMRI-BOLD signal makes it extremely difficult to extrapolate from functional imaging data to neuronal activity. Hence, functional imaging neither unequivocally resolves whether neurons in a particular region of the brain have access to information from several modalities, nor characterizes potential multisensory interactions at the neuronal level. We argue that electrophysiological studies, which provide direct and spatio-temporally refined measurements of neuronal activity, are required to provide sensitive tests of multisensory interactions in a given brain region.

Section snippets

Functional imaging of multisensory influences in auditory cortex

To address ‘where’ along the auditory processing pathways influences from other sensory modalities first arise, one needs to localize those regions where neuronal activity to acoustic stimulation can be modulated by stimulation of another modality. A simple and direct way of doing so is to compare neuronal responses to the presentation of an acoustic stimulus to the response when the same stimulus is paired with, for instance, a visual stimulus. A seemingly efficient, non-invasive way of doing

Predicting neuronal activity from functional imaging data

To interpret the results of functional imaging in the context of neuronal activity we need to better understand the neuronal substrate that couples the hemodynamic responses to neuronal activity – hence the neural correlate of the imaging signal. The fMRI-BOLD signal reflects cerebral blood flow (CBF) and tissue oxygenation, both of which change in proportion to the local energy demands in or near an image voxel. Following current understanding, this energy demand originates mostly due to peri

Multisensory influences at the level of neuronal activity

Several electrophysiological studies have demonstrated multisensory influences in auditory regions in a number of species, a range of paradigms, and most importantly, to different degrees in the different electrophysiological signals. Recordings of local field potentials or current source densities in auditory regions revealed a widespread influence of visual or somatosensory stimuli on acoustic responses (Ghazanfar et al., 2008, Ghazanfar et al., 2005, Lakatos et al., 2007, Schroeder and Foxe,

Conclusions

The notion that auditory fields in or near primary auditory cortex receive inputs from other modalities and possibly integrate this with the acoustic information has become increasingly popular over the last years (Ghazanfar and Schroeder, 2006). A good deal of this evidence comes from functional imaging experiments. However, as we argue here, this technique provides good means to localize regions of interest, but does not warrant direct conclusions about the properties of individual or

Acknowledgements

This work was supported by the Max Planck Society and the Alexander von Humboldt Foundation.

References (102)

  • J. Goense et al.

    Neurophysiology of the BOLD fMRI Signal in Awake Monkeys

    Curr. Biol.

    (2008)
  • J. Hyvarinen et al.

    Distribution of visual and somatic functions in the parietal associative area 7 of the monkey

    Brain Res.

    (1979)
  • C. Kayser et al.

    Integration of touch and sound in auditory cortex

    Neuron

    (2005)
  • C. Kayser et al.

    Spike-phase coding boosts and stabilizes the information carried by spatial and temporal spike patterns

    Neuron

    (2009)
  • P. Lakatos et al.

    Neuronal oscillations and multisensory interaction in primary auditory cortex

    Neuron

    (2007)
  • C. Lehmann et al.

    Differential patterns of multisensory interactions in core and belt areas of human auditory cortex

    Neuroimage

    (2006)
  • S. Lehmann et al.

    The role of multisensory memories in unisensory object discrimination

    Brain Res. Cogn. Brain Res.

    (2005)
  • B. Lutkenhoner et al.

    Magnetoencephalographic correlates of audiotactile interaction

    Neuroimage

    (2002)
  • M.M. Merzenich et al.

    Representation of the cochlear partition of the superior temporal plane of the macaque monkey

    Brain Res.

    (1973)
  • I. Nelken et al.

    Information theory in auditory research

    Hear. Res.

    (2007)
  • Y. Nir et al.

    Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations

    Curr. Biol.

    (2007)
  • P. Poirazi et al.

    Pyramidal neuron as two-layer neural network

    Neuron

    (2003)
  • J.P. Rauschecker

    Cortical processing of complex sounds

    Curr. Opin. Neurobiol.

    (1998)
  • F. Rivier et al.

    Cytochrome oxidase, acetylcholinesterase, and NADPH-diaphorase staining in human supratemporal and insular cortex: evidence for multiple auditory areas

    Neuroimage

    (1997)
  • C.E. Schroeder et al.

    The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex

    Brain Res. Cogn. Brain Res.

    (2002)
  • C.E. Schroeder et al.

    Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing

    Int. J. Psychophysiol.

    (2003)
  • M. Schurmann et al.

    Touch activates human auditory cortex

    Neuroimage

    (2006)
  • A.R. Seitz et al.

    Sound facilitates visual learning

    Curr. Biol.

    (2006)
  • N. van Atteveldt et al.

    Integration of letters and speech sounds in the human brain

    Neuron

    (2004)
  • A. Vatakis et al.

    Audiovisual synchrony perception for music, speech, and object actions

    Brain Res.

    (2006)
  • J. Warnking et al.

    FMRI retinotopic mapping – step by step

    Neuroimage

    (2002)
  • A. Belitski et al.

    Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information

    J. Neurosci.

    (2008)
  • L.E. Bernstein et al.

    Visual speech perception without primary auditory cortex activation

    Neuroreport

    (2002)
  • Bizley, J.K., Nodal, F.R., Bajo, V.M., Nelken, I., King, A.J. 2006. Physiological and anatomical evidence for...
  • A. Borst et al.

    Information theory and neural coding

    Nat. Neurosci.

    (1999)
  • C. Bruce et al.

    Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque

    J. Neurophysiol.

    (1981)
  • G.A. Calvert et al.

    Reading speech from still and moving faces: the neural substrates of visible speech

    J. Cogn. Neurosci.

    (2003)
  • G.A. Calvert et al.

    Response amplification in sensory-specific cortices during crossmodal binding

    Neuroreport

    (1999)
  • G.A. Calvert et al.

    Activation of auditory cortex during silent lipreading

    Science

    (1997)
  • C. Cappe et al.

    Neuronal Responses to Visual Stimuli in Auditory Cortical Areas of Monkeys Performing an Audio–visual Detection Task

    (2007)
  • O. Chiry et al.

    Patterns of calcium-binding proteins support parallel and hierarchical organization of human auditory areas

    Eur. J. Neurosci.

    (2003)
  • S. Clarke et al.

    Compartments within human primary auditory cortex: evidence from cytochrome oxidase and acetylcholinesterase staining

    Eur. J. Neurosci.

    (1998)
  • F. Crivello et al.

    Comparison of spatial normalization procedures and their impact on functional maps

    Hum. Brain Mapp.

    (2002)
  • S.A. Engel et al.

    fMRI of human visual cortex

    Nature

    (1994)
  • D.J. Felleman et al.

    Distributed hierarchical processing in the primate cerebral cortex

    Cereb. Cortex

    (1991)
  • A. Fergus et al.

    GABAergic regulation of cerebral microvascular tone in the rat

    J. Cereb. Blood Flow Metab.

    (1997)
  • J.J. Foxe et al.

    The case for feedforward multisensory convergence during early cortical processing

    Neuroreport

    (2005)
  • J.J. Foxe et al.

    Auditory–somatosensory multisensory processing in auditory association cortex: an fMRI study

    J. Neurophysiol.

    (2002)
  • B.C. Fullerton et al.

    Architectonic analysis of the auditory-related areas of the superior temporal region in human brain

    J. Comp. Neurol.

    (2007)
  • A.A. Ghazanfar et al.

    Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys

    J. Neurosci.

    (2008)
  • Cited by (81)

    • Multisensory integration of anticipated cardiac signals with visual targets affects their detection among multiple visual stimuli

      2022, NeuroImage
      Citation Excerpt :

      Interestingly, the association between behavioral improvement and the sub-additive ERP modulation has been well established in the field of auditory-visual integration. For example, studies using animal models have repeatedly reported sub-additive neural response interactions that enhanced sensory processing (Angelaki et al., 2009; Bizley et al., 2007; Kayser et al., 2009). Likewise, many human studies have shown that the behavioral benefits of multisensory stimuli are related to sub-additive ERP responses (Mercier et al., 2013; Stekelenburg and Vroomen, 2012; Van der Burg et al., 2011).

    • 2.32 - Primary Auditory Cortex and the Thalamo-Cortico-Thalamic Circuitry I. Anatomy

      2020, The Senses: A Comprehensive Reference: Volume 1-7, Second Edition
    View all citing articles on Scopus
    1

    Present address: The Institute of Neuroscience, University of Newcastle, Newcastle upon Tyne, NE2 4HH, UK.

    View full text