Skip to main content
Log in

On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies

Experimental Brain Research Aims and scope Submit manuscript

Abstract

A growing number of brain imaging studies are being undertaken in order to better understand the contributions of multisensory processes to human behavior and perception. Many of these studies are designed on the basis of the physiological findings from single neurons in animal models, which have shown that multisensory neurons have the capacity for integrating their different sensory inputs and give rise to a product that differs significantly from either of the unisensory responses. At certain points these multisensory interactions can be superadditive, resulting in a neural response that exceeds the sum of the unisensory responses. Because of the difficulties inherent in interpreting the results of imaging large neuronal populations, superadditivity has been put forth as a stringent criterion for identifying potential sites of multisensory integration. In the present manuscript we discuss issues related to using the superadditive model in human brain imaging studies, focusing on population responses to multisensory stimuli and the relationship between single neuron measures and functional brain imaging measures. We suggest that the results of brain imaging studies be interpreted with caution in regards to multisensory integration. Future directions for imaging multisensory integration are discussed in light of the ideas presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Banati RB, Goerres GW, Tjoa C, Aggleton JP, Grasby P (2000) The functional anatomy of visual-tactile integration in man: a study using positron emission tomography. Neuropsychologia 38:115–124

    Article  CAS  PubMed  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823

    Article  CAS  PubMed  Google Scholar 

  • Benedek G, Fischer-Szatmari L, Kovacs G, Perenyi J, Katoh YY (1996) Visual, somatosensory and auditory modality properties along the feline suprageniculate-anterior ectosylvian sulcus/insular pathway. Prog Brain Res 112:325–334

    CAS  PubMed  Google Scholar 

  • Binder MD (2002) Integration of synaptic and intrinsic dendritic currents in cat spinal motoneurons. Brain Res Rev 40:1–8

    Article  PubMed  Google Scholar 

  • Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300–304

    CAS  PubMed  Google Scholar 

  • Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213–2218

    Article  PubMed  Google Scholar 

  • Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16:805–816

    Article  PubMed  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  CAS  PubMed  Google Scholar 

  • Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997) Activation of auditory cortex during silent lipreading. Science 276:593–596

    Article  CAS  PubMed  Google Scholar 

  • Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619–2623

    CAS  PubMed  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  CAS  PubMed  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438

    Article  CAS  PubMed  Google Scholar 

  • Dieterich M, Bense S, Stephan T, Yousry TA, Brandt T (2003) fMRI signal increases and decreases in cortical areas during small-field optokinetic stimulation and central fixation. Exp Brain Res 148:117–127

    Article  PubMed  Google Scholar 

  • Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77–83

    Google Scholar 

  • Gail A, Brinksmeyer HJ, Eckhorn R (2004) Perception-related modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry. Cereb Cortex 14:300–313

    Article  PubMed  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    Article  CAS  PubMed  Google Scholar 

  • Gobbele R, Schurmann M, Forss N, Juottonen K, Buchner H, Hari R (2003) Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction. Neuroimage 20:503–511

    Article  PubMed  Google Scholar 

  • Goldman-Rakic P (1995) Architecture of the prefrontal cortex and the central executive. In: Grafman J, Holyoak K, Boller F (eds) Structure and functions of the human prefrontal cortex. The New York Academy of Sciences, NY USA, pp 71–83

    Google Scholar 

  • Gonzalez-Burgos G, Krimer LS, Urban NN, Barrionuevo G, Lewis DA (2004) Synaptic efficacy during repetitive activation of excitatory inputs in primate dorsolateral prefrontal cortex. Cereb Cortex 14:530–542

    Article  PubMed  Google Scholar 

  • Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18:1072–1084

    CAS  PubMed  Google Scholar 

  • Hayes EA, Tiippana K, Nicol TG, Sams M, Kraus N (2003) Integration of heard and seen speech: a factor in learning disabilities in children. Neurosci Lett 351:46–50

    Article  CAS  PubMed  Google Scholar 

  • Heeger DJ, Huk AC, Geisler WS, Albrecht DG (2000) Spikes versus BOLD: what does neuroimaging tell us about neuronal activity?. Nat Neurosci 3:631–633

    Article  CAS  PubMed  Google Scholar 

  • Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522

    Google Scholar 

  • King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500

    Article  CAS  PubMed  Google Scholar 

  • Klucharev V, Mottonen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65–75

    Article  Google Scholar 

  • Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE (2002a) Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci 14:420–429

    Article  PubMed  Google Scholar 

  • Laurienti PJ, Field AS, Burdette JH, Maldjian JA, Yen YF, Moody DM (2002b) Dietary caffeine consumption modulates fMRI measures. Neuroimage 17:751–757

    Article  PubMed  Google Scholar 

  • Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223

    Article  PubMed  Google Scholar 

  • Lauritzen M, Gold L (2003) Brain function and neurophysiological correlates of signals used in functional neuroimaging. J Neurosci 23:3972–3980

    CAS  PubMed  Google Scholar 

  • Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cereb Cortex 10:873–888

    Article  CAS  PubMed  Google Scholar 

  • Logothetis NK (2003a) The underpinnings of the BOLD functional magnetic resonance imaging signal. J Neurosci 23:3963–3971

    CAS  PubMed  Google Scholar 

  • Logothetis NK (2003b) MR imaging in the non-human primate: studies of function and of dynamic connectivity. Curr Opin Neurobiol 13:630–642

    Article  CAS  PubMed  Google Scholar 

  • Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412:150–157

    Article  CAS  PubMed  Google Scholar 

  • Macaluso E, Frith C, Driver J (2000a) Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. J Neurophysiol 83:3062–3075

    Google Scholar 

  • Macaluso E, Frith CD, Driver J (2000b) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208

    Article  CAS  PubMed  Google Scholar 

  • Macaluso E, Frith CD, Driver J (2001) Multimodal mechanisms of attention related to rates of spatial shifting in vision and touch. Exp Brain Res 137:445–454

    Google Scholar 

  • Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21:725–732

    Article  CAS  PubMed  Google Scholar 

  • Margulis M, Tang C-M (1998) Temporal integration can readily switch between sublinear and supralinear summation. J Neurophysiol 79:2809–2813

    Google Scholar 

  • Mathiesen C, Caesar K, Akgoren N, Lauritzen M (1998) Modification of activity-dependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex. J Physiol 512(Pt 2):555–566

    Article  CAS  PubMed  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

    Google Scholar 

  • Meredith MA, Stein BE (1986a) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662

    Google Scholar 

  • Meredith MA, Stein BE (1986b) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354

    Google Scholar 

  • Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857

    Google Scholar 

  • Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I Temporal factors. J Neurosci 7:3215–3229

    Google Scholar 

  • Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88:181–186

    Google Scholar 

  • Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465

    Article  PubMed  Google Scholar 

  • Nozawa G, Stanford TR, Vaughan JW, Quessy S, Kadunce D, Stein BE (1997) A factorial approach to modeling multisensory integration in the superior colliculus. Soc Neurosci Abstr 23:451

    Google Scholar 

  • Ogawa S, Lee TM, Kay AR, Tank DW (1990) Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci USA 87:9868–9872

    CAS  PubMed  Google Scholar 

  • Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90:4022–4026

    Google Scholar 

  • Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586

    Article  PubMed  Google Scholar 

  • Populin LC, Yin TC (2002) Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22:2826–2834

    CAS  PubMed  Google Scholar 

  • Quessy S, Sweatt A, Stein BE, Stanford TR (2000) The influence of stimulus intensity and timing on the responses of multisensory neurons in the superior colliculus: comparison to a model’s prediction. Soc Neurosci Abstr 26:1221

    Google Scholar 

  • Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625

    Google Scholar 

  • Rees G, Friston K, Koch C (2000) A direct quantitative relationship between the functional properties of human and macaque V5. Nat Neurosci 3:716–723

    Article  CAS  PubMed  Google Scholar 

  • Schurmann M, Kolev V, Menzel K, Yordanova J (2002) Spatial coincidence modulates interaction between visual and somatosensory evoked potentials. Neuroreport 13:779–783

    Article  PubMed  Google Scholar 

  • Selemon LD, Rajkowska G, Goldman-Rakic PS (1998) Elevated neuronal density in prefrontal area 46 in brains from schizophrenic patients: application of a three-dimensional, stereologic counting method. J Comp Neurol 392:402–412

    Article  CAS  PubMed  Google Scholar 

  • Shams L, Kamitani Y, Thompson S, Shimojo S (2001) Sound alters visual evoked potentials in humans. Neuroreport 12:3849–3852

    Article  CAS  PubMed  Google Scholar 

  • Soltysik DA, Peck KK, White KD, Crosson B, Briggs RW (2004) Comparison of hemodynamic response nonlinearity across primary cortical areas. Neuroimage 22:1117–1127

    Article  PubMed  Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA

    Google Scholar 

  • Stein BE, Jiang W, Stanford TR (2004a) Multisensory integration in single neurons in midbrain and cortex. In: Calvert G, Spence C, Stein BE (eds) A handbook of multisensory processes. MIT Press, Cambridge, MA, pp 243–264

    Google Scholar 

  • Stein BE, Stanford TR, Wallace MT, Vaughan JW, Jiang W (2004b) Cross-modal spatial interactions in subcortical and cortical circuits. In: Driver J (ed) Crossmodal and crossmodal attention. Oxford University Press, Oxford, pp 25–50

    Google Scholar 

  • Wallace MT, Stein BE (1996) Sensory organization of the superior colliculus in cat and monkey. Prog Brain Res 112:301–311

    CAS  PubMed  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488

    Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol 69:1797–1809

    Google Scholar 

  • Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266

    Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010

    Google Scholar 

  • Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci USA 101:2167–2172

    Article  CAS  PubMed  Google Scholar 

  • Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 13:1034–1043

    Article  PubMed  Google Scholar 

  • de Zubicaray GI, McMahon KL, Eastburn MM, Wilson SJ (2002) Orthographic/phonological facilitation of naming responses in the picture-word task: an event-related fMRI study using overt vocal responding. NeuroImage 16:1084–1093

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Supported in part by Grants NS 42568 (PJL), MH 63861 (MTW), and NS 36916 (BES).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Paul J. Laurienti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Laurienti, P.J., Perrault, T.J., Stanford, T.R. et al. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166, 289–297 (2005). https://doi.org/10.1007/s00221-005-2370-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-005-2370-2

Keywords

Navigation