Elsevier

Hearing Research

Volume 258, Issues 1–2, December 2009, Pages 72-79
Hearing Research

Research papers
Neuronal mechanisms, response dynamics and perceptual functions of multisensory interactions in auditory cortex

https://doi.org/10.1016/j.heares.2009.06.018Get rights and content

Abstract

Most auditory events in nature are accompanied by non-auditory signals, such as a view of the speaker’s face during face-to-face communication or the vibration of a string during a musical performance. While it is known that accompanying visual and somatosensory signals can benefit auditory perception, often by making the sound seem louder, the specific neural bases for sensory amplification are still debated. In this review, we want to deal with what we regard as confusion on two topics that are crucial to our understanding of multisensory integration mechanisms in auditory cortex: (1) Anatomical Underpinnings (e.g., what circuits underlie multisensory convergence), and (2) Temporal Dynamics (e.g., what time windows of integration are physiologically feasible). The combined evidence on multisensory structure and function in auditory cortex advances the emerging view of the relationship between perception and low level multisensory integration. In fact, it seems that the question is no longer whether low level, putatively unisensory cortex is accessible to multisensory influences, but how.

Section snippets

Introduction: no place (for unisensory auditory neurons) to hide

While peripheral transduction in the cochlea appears specifically designed to capture the physical characteristics of sound, the auditory cortex is not purely a sound processor. The advent of new techniques, coupled with new conceptual models, has made it clear that even at the level of A1, auditory processing is influenced by multimodal stimuli Following on initial indications from experiments in humans (Sams et al., 1991, Calvert et al., 1997, Giard and Peronnet, 1999) and a parallel line of

Anatomical convergence of visual and somatosensory inputs in auditory cortex

Anatomical evidence forms a framework for thinking about potential non-auditory influences on auditory cortex, though these data cannot tell us whether and to what extent these input routes function in multisensory integration. As was already apparent in our initial review of this topic (Schroeder et al., 2003), there is strong evidence for potential cortical and subcortical sources of non-auditory input to auditory cortex and more recent investigations continue to support and elaborate on this

Response timing of non-auditory signals in auditory cortex

Analysis of the response timing to stimulus onset in auditory cortex can be used to establish a model of auditory and non-auditory input timing into auditory cortex. In this section, we first consider the important experimental variables affecting response latency. Because differences across experimental paradigms and analytical methods can greatly influence reported latency values, we have summarized response timing data from our group, which employed the same paradigm (and largely concur with

Multisensory mechanisms in auditory cortex and relationships to perception and behavior

Low level multisensory integration in auditory cortex has the potential to greatly benefit perception and behavior because input from multiple modalities can enhance the representation of sound that the rest of the auditory cortices (and higher-order regions) receives. Audiovisual integration in low level auditory areas has been linked to specific perceptual benefits such as improved speech understanding (Grant et al., 1998, Sekiyama et al., 2003, van Wassenhove et al., 2005, Schroeder et al.,

Conclusions and future directions

The overwhelming evidence of multisensory integration at even the lowest levels of the auditory system suggests that multisensory interaction may be the rule rather than the exception for brain processing in humans and other primates. This view presents a challenging paradox for the classic hierarchical view that increasingly complex information is incorporated into the neural code in a series of subsequent downstream areas. On the one hand, incorporation of multisensory information at low

Acknowledgements

We would like to thank our colleagues Tammy McGinnis, M. Noelle O’Connel and Aimee Mills for their assistance. We also thank Peter Lakatos, Arnaud Falchier and Troy A. Hackett for their helpful comments and suggestions. This work was supported by the National Institutes of Health DC 09918 and MH 61989.

References (93)

  • K.S. Rockland et al.

    Multisensory convergence in calcarine visual areas in macaque monkey

    Int. J. Psychophysiol.

    (2003)
  • K.S. Rockland et al.

    Laminar origins and terminations of cortical connections of the occipital lobe in the rhesus monkey

    Brain Res.

    (1979)
  • L.A. Ross et al.

    Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions

    Schizophr. Res.

    (2007)
  • M. Sams et al.

    Seeing speech: visual information from lip movements modifies activity in the human auditory cortex

    Neurosci. Lett.

    (1991)
  • C.E. Schroeder et al.

    Multisensory contributions to low-level, ‘unisensory’ processing

    Curr. Opin. Neurobiol.

    (2005)
  • C.E. Schroeder et al.

    The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex

    Brain Res. Cogn. Brain Res.

    (2002)
  • C.E. Schroeder et al.

    Neuronal oscillations and visual amplification of speech

    Trends Cogn. Sci.

    (2008)
  • C.E. Schroeder et al.

    Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing

    Int. J. Psychophysiol.

    (2003)
  • K. Sekiyama et al.

    Auditory–visual speech perception examined by fMRI and PET

    Neurosci. Res.

    (2003)
  • M. Steinschneider et al.

    Cellular generators of the cortical auditory evoked potential initial component

    Electroencephalogr. Clin. Neurophysiol.

    (1992)
  • B.W. Vines et al.

    Cross-modal interactions in the perception of musical performance

    Cognition

    (2006)
  • P. Zurita et al.

    Changes of single unit activity in the cat’s auditory thalamus and cortex associated to different anesthetic conditions

    Neurosci. Res.

    (1994)
  • M. Brosch et al.

    Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys

    J. Neurosci.

    (2005)
  • G.A. Calvert et al.

    Activation of auditory cortex during silent lipreading

    Science

    (1997)
  • C. Cappe et al.

    Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey

    Eur. J. Neurosci.

    (2005)
  • Cappe, C., Morel, A., Barone, P., Rouiller, E.M., 2009. The thalamocortical projection systems in primate: an...
  • G.G. Celesia

    Auditory evoked responses. Intracranial and extracranial average evoked responses

    Arch. Neurol.

    (1968)
  • C. Chandrasekaran et al.

    The natural statistics of audiovisual speech

    PLoS Comput. Biol.

    (2009)
  • C.M. Chen et al.

    Functional anatomy and interaction of fast and slow visual pathways in macaque monkeys

    Cereb. Cortex

    (2007)
  • P.W. Davies et al.

    Single unit activity in the auditory cortex of the cat

    Bull. Johns Hopkins Hosp.

    (1956)
  • L.A. de la Mothe et al.

    Cortical connections of the auditory cortex in marmoset monkeys: core and medial belt regions

    J. Comp. Neurol.

    (2006)
  • L.A. de la Mothe et al.

    Thalamic connections of the auditory cortex in marmoset monkeys: core and medial belt regions

    J. Comp. Neurol.

    (2006)
  • T. Elbert et al.

    Increased cortical representation of the fingers of the left hand in string players

    Science

    (1995)
  • A. Falchier et al.

    Anatomical evidence of multimodal integration in primate striate cortex

    J. Neurosci.

    (2002)
  • J.J. Foxe et al.

    The case for feedforward multisensory convergence during early cortical processing

    Neuroreport

    (2005)
  • K.M. Fu et al.

    Auditory cortical neurons respond to somatosensory stimulation

    J. Neurosci.

    (2003)
  • B.H. Gaese et al.

    Anesthesia changes frequency tuning of neurons in the rat primary auditory cortex

    J. Neurophysiol.

    (2001)
  • M.H. Giard et al.

    Auditory–visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study

    J. Cogn. Neurosci.

    (1999)
  • C.M. Gomez Gonzalez et al.

    Sources of attention-sensitive visual event-related potentials

    Brain Topogr.

    (1994)
  • K.W. Grant et al.

    Auditory–visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory–visual integration

    J. Acoust. Soc. Am.

    (1998)
  • T.A. Hackett et al.

    Multisensory convergence in auditory cortex. II. Thalamocortical connections of the caudal superior temporal plane

    J. Comp. Neurol.

    (2007)
  • T.A. Hackett et al.

    Sources of somatosensory input to the caudal belt areas of auditory cortex

    Perception

    (2007)
  • T.A. Hackett et al.

    Thalamocortical connections of the parabelt auditory cortex in macaque monkeys

    J. Comp. Neurol.

    (1998)
  • W.D. Hairston et al.

    Auditory enhancement of visual temporal order judgment

    Neuroreport

    (2006)
  • D.A. Hodges et al.

    Aspects of multisensory perception: the integration of visual and auditory information in musical experiences

    Ann. NY Acad. Sci.

    (2005)
  • E.G. Jones

    A new view of specific and nonspecific thalamocortical connections

    Adv. Neurol.

    (1998)
  • Cited by (83)

    • Cross-sensory inhibition or unisensory facilitation: A potential neural architecture of modality switch effects

      2020, Journal of Mathematical Psychology
      Citation Excerpt :

      Moreover, previous studies demonstrated that these synapses can be trained by experience, to implement the prior probability of the co-occurrence of audio–visual stimuli in close temporal and spatial proximity (Ursino et al., 2017). Various data in the literature confirm the existence of cross-modal links between the primary visual and auditory regions ((Alais et al., 2010; Driver & Noesselt, 2008; Foxe & Schroeder, 2005; Ghazanfar & Schroeder, 2006; Musacchia & Schroeder, 2009; Recanzone, 2009; Shams & Kim, 2010; Stein & Stanford, 2008; Ursino et al., 2014) for reviews). Finally, we wish to remark that these cross-modal synapses in the model are not strong enough to evoke a phantom response in the other area in the case of unisensory inputs, i.e., only a single area is active in the unisensory condition.

    View all citing articles on Scopus
    View full text