Is neocortex essentially multisensory?

https://doi.org/10.1016/j.tics.2006.04.008Get rights and content

Although sensory perception and neurobiology are traditionally investigated one modality at a time, real world behaviour and perception are driven by the integration of information from multiple sensory sources. Mounting evidence suggests that the neural underpinnings of multisensory integration extend into early sensory processing. This article examines the notion that neocortical operations are essentially multisensory. We first review what is known about multisensory processing in higher-order association cortices and then discuss recent anatomical and physiological findings in presumptive unimodal sensory areas. The pervasiveness of multisensory influences on all levels of cortical processing compels us to reconsider thinking about neural processing in unisensory terms. Indeed, the multisensory nature of most, possibly all, of the neocortex forces us to abandon the notion that the senses ever operate independently during real-world cognition.

Introduction

Early investigations of the cerebral cortex and its role in perception recognized that the convergence and integration of information from different sensory modalities is an essential component of cognition [1]. In traditional models of the sensory brain, however, multisensory integration is deferred until after extensive ‘unisensory’ processing has occurred [2]. This view arose from early neuroanatomical studies in cats and monkeys, which suggested few if any interconnections between somatosensory, auditory and visual cortices [3], and experimental lesions of discrete regions, which appeared to produce unimodal behavioural deficits [4].

Rapidly accumulating evidence on multisensory processing in primates and other mammals directly challenges this classical view. Strikingly, several anatomical studies suggest that the substrate for multisensory integration is already present at the primary cortical level 5, 6, 7. In fact, the work published to date may reveal only the ‘tip of the iceberg’, as ongoing studies continue to reveal extensive interactions among low-level sensory areas and between those areas and association cortex. Here, we review multisensory processes involving vision, audition and somatosensation across the neocortical mantle. We advance the perspective that multisensory influences are integral to primary as well as higher-order cortical operations. Although chemosensory processes are not considered here, it is noteworthy that pervasive multisensory interactions are also characteristic of these sensory systems [8].

Recasting the role of lower- and higher-order cortical processes in the light of multisensory interactions will compel us to reconsider the wisdom of reducing sensory perception and cognition into unimodal components. That is, the neurobiological data reviewed here suggest that focusing solely on unisensory processes will continue to provide us only with an impoverished view of both brain and behavior. The recent recognition of this fact by many scientists has led to different research strategies that are providing unique and fruitful perspectives in our understanding of perception and cognition [9].

Section snippets

Multisensory regions of higher-order association cortex

Several cortical areas were considered multisensory by virtue of their connections with multiple unimodal areas [10], their single neuron responses to multisensory input 11, 12, and the behavioural deficits of patients with lesions in these areas [13]. Thus, included among the classical multisensory cortical areas of primates were the superior temporal sulcus (STS), the intraparietal (IP) complex, and the frontal cortex (Figure 1a). In this section, we will review the structural and functional

Visual and somatosensory processing in auditory cortex

Functional imaging [38] and later, event related potential 39, 40, 41, 42, 43 studies, raised the possibility of audio-tactile and audio-visual interactions in human auditory cortex. Localization of these effects in the superior temporal plane was independently supported by techniques that have better anatomical resolution, including magnetoencephalography 44, 45 and fMRI 38, 46, 47.

In parallel to the human studies, intracranial recordings have directly confirmed multisensory convergence in

Higher-order versus lower-order cortical areas

The findings reviewed here demonstrate that both higher-order association areas and presumptive unisensory areas of the cerebral cortex are in fact multisensory in nature (Figure 1b). The research defining the specific role each type of cortical area plays in real-world sensory/cognitive processing suggests the emergence of two broad themes. First, the multisensory processes in association cortex reinforce the widely held view that a primary role of these regions is to compute a veridical

Conclusions

The integration of information from different sensory systems is a fundamental characteristic of perception and cognition – qualitatively different kinds of information from the various sense organs are put together in the brain to produce a unified, coherent representation of the outside world. Traditionally, it has been assumed that the integration of such disparate information at the cortical level was the task of specialized, higher-order association areas of the neocortex. In stark

Acknowledgements

We thank Joost Maier for his helpful comments on an earlier draft of this manuscript. C.E.S. acknowledges support from the National Institute of Mental Health (MH61989 and TW05674).

References (80)

  • B. Lutkenhoner

    Magnetoencephalographic correlates of audiotactile interaction

    Neuroimage

    (2002)
  • N. van Atteveldt

    Integration of letters and speech sounds in the human brain

    Neuron

    (2004)
  • C. Kayser

    Integration of touch and sound in auditory cortex

    Neuron

    (2005)
  • U. Werner-Reiss

    Eye position affects activity in primary auditory cortex of primates

    Curr. Biol.

    (2003)
  • L.E. Bernstein

    Auditory speech detection in noise enhanced by lipreading

    Speech Commun.

    (2004)
  • J-L. Schwartz

    Seeing to hear better: evidence for early audio-visual interactions in speech identification

    Cognition

    (2004)
  • K. Sathian et al.

    Feeling with the mind's eye: contribution of visual cortex to tactile perception

    Behav. Brain Res.

    (2002)
  • T.W. James

    Haptic study of three-dimensional objects activates extrastriate visual areas

    Neuropsychologia

    (2002)
  • Y.D. Zhou et al.

    Somatosensory cell responses to an auditory cue in a haptic memory task

    Behav. Brain Res.

    (2004)
  • M.A. Frens et al.

    Visual–auditory interactions modulate saccade-related activity in monkey superior colliculus

    Brain Res. Bull.

    (1998)
  • E.D. Adrian

    The Sherrington Lectures: I. Sensory Integration

    (1949)
  • D.J. Felleman et al.

    Distributed hierarchical processing in the primate cerebral cortex

    Cereb. Cortex

    (1991)
  • L.C. Massopust

    Auditory frequency discrimination in cortically ablated monkeys

    J. Aud. Res.

    (1965)
  • C. Cappe et al.

    Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey

    Eur. J. Neurosci.

    (2005)
  • A. Falchier

    Anatomical evidence of multimodal integration in primate striate cortex

    J. Neurosci.

    (2002)
  • D.B. Katz

    Dynamic and multimodal responses of gustatory cortical neurons in awake rats

    J. Neurosci.

    (2001)
  • G.A. Calvert

    The Handbook of Multisensory Processes

    (2004)
  • E.G. Jones

    An anatomical study of converging sensory pathways within the cerebral cortex of the monkey

    Brain

    (1970)
  • C. Bruce

    Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque

    J. Neurophysiol.

    (1981)
  • K. Hikosaka

    The polysensory region in the anterior bank of the caudal superior temporal sulcus of the macaque monkey

    Biomed. Res. (Tokyo)

    (1993)
  • Teuber, H.L. (1966) Alterations of perception after brain injury. In Brain and Conscious Experience. (Eccles, J.C.,...
  • N.E. Barraclough

    Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions

    J. Cogn. Neurosci.

    (2005)
  • R.A. Andersen

    Multimodal representation of space in the posterior parietal cortex and its use in planning movements

    Annu. Rev. Neurosci.

    (1997)
  • P. Mazzoni

    Spatially-tuned auditory responses in area LIP of macaques performing delayed memory saccades to acoustic targets

    J. Neurophysiol.

    (1996)
  • J.F. Linden

    Responses to auditory stimuli in macaque lateral intraparietal area II. behavioral modulation

    J. Neurophysiol.

    (1999)
  • G.W. Gifford et al.

    Effect of a central fixation light on auditory spatial responses in area LIP

    J. Neurophysiol.

    (2004)
  • M. Avillac

    Reference frames for representing visual and tactile locations in parietal cortex

    Nat. Neurosci.

    (2005)
  • F. Bremmer

    Visual–vestibular interactive responses in the macaque ventral intraparietal area (VIP)

    Eur. J. Neurosci.

    (2002)
  • J-R. Duhamel

    Ventral intraparietal area of the macaque: congruent visual and somatic response properties

    J. Neurophysiol.

    (1998)
  • A. Schlack

    Multisensory space representations in the macaque ventral intraparietal area

    J. Neurosci.

    (2005)
  • Cited by (1086)

    • Concepts require flexible grounding

      2023, Brain and Language
    View all citing articles on Scopus
    View full text