Abstract
A growing number of brain imaging studies are being undertaken in order to better understand the contributions of multisensory processes to human behavior and perception. Many of these studies are designed on the basis of the physiological findings from single neurons in animal models, which have shown that multisensory neurons have the capacity for integrating their different sensory inputs and give rise to a product that differs significantly from either of the unisensory responses. At certain points these multisensory interactions can be superadditive, resulting in a neural response that exceeds the sum of the unisensory responses. Because of the difficulties inherent in interpreting the results of imaging large neuronal populations, superadditivity has been put forth as a stringent criterion for identifying potential sites of multisensory integration. In the present manuscript we discuss issues related to using the superadditive model in human brain imaging studies, focusing on population responses to multisensory stimuli and the relationship between single neuron measures and functional brain imaging measures. We suggest that the results of brain imaging studies be interpreted with caution in regards to multisensory integration. Future directions for imaging multisensory integration are discussed in light of the ideas presented.
Similar content being viewed by others
References
Banati RB, Goerres GW, Tjoa C, Aggleton JP, Grasby P (2000) The functional anatomy of visual-tactile integration in man: a study using positron emission tomography. Neuropsychologia 38:115–124
Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823
Benedek G, Fischer-Szatmari L, Kovacs G, Perenyi J, Katoh YY (1996) Visual, somatosensory and auditory modality properties along the feline suprageniculate-anterior ectosylvian sulcus/insular pathway. Prog Brain Res 112:325–334
Binder MD (2002) Integration of synaptic and intrinsic dendritic currents in cat spinal motoneurons. Brain Res Rev 40:1–8
Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300–304
Callan DE, Jones JA, Munhall K, Callan AM, Kroos C, Vatikiotis-Bateson E (2003) Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport 14:2213–2218
Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16:805–816
Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123
Calvert GA, Bullmore ET, Brammer MJ, Campbell R, Williams SC, McGuire PK, Woodruff PW, Iversen SD, David AS (1997) Activation of auditory cortex during silent lipreading. Science 276:593–596
Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10:2619–2623
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657
Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438
Dieterich M, Bense S, Stephan T, Yousry TA, Brandt T (2003) fMRI signal increases and decreases in cortical areas during small-field optokinetic stimulation and central fixation. Exp Brain Res 148:117–127
Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10:77–83
Gail A, Brinksmeyer HJ, Eckhorn R (2004) Perception-related modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry. Cereb Cortex 14:300–313
Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490
Gobbele R, Schurmann M, Forss N, Juottonen K, Buchner H, Hari R (2003) Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction. Neuroimage 20:503–511
Goldman-Rakic P (1995) Architecture of the prefrontal cortex and the central executive. In: Grafman J, Holyoak K, Boller F (eds) Structure and functions of the human prefrontal cortex. The New York Academy of Sciences, NY USA, pp 71–83
Gonzalez-Burgos G, Krimer LS, Urban NN, Barrionuevo G, Lewis DA (2004) Synaptic efficacy during repetitive activation of excitatory inputs in primate dorsolateral prefrontal cortex. Cereb Cortex 14:530–542
Hadjikhani N, Roland PE (1998) Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomographic study. J Neurosci 18:1072–1084
Hayes EA, Tiippana K, Nicol TG, Sams M, Kraus N (2003) Integration of heard and seen speech: a factor in learning disabilities in children. Neurosci Lett 351:46–50
Heeger DJ, Huk AC, Geisler WS, Albrecht DG (2000) Spikes versus BOLD: what does neuroimaging tell us about neuronal activity?. Nat Neurosci 3:631–633
Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE (2001) Two cortical areas mediate multisensory integration in superior colliculus neurons. J Neurophysiol 85:506–522
King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500
Klucharev V, Mottonen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65–75
Laurienti PJ, Burdette JH, Wallace MT, Yen YF, Field AS, Stein BE (2002a) Deactivation of sensory-specific cortex by cross-modal stimuli. J Cogn Neurosci 14:420–429
Laurienti PJ, Field AS, Burdette JH, Maldjian JA, Yen YF, Moody DM (2002b) Dietary caffeine consumption modulates fMRI measures. Neuroimage 17:751–757
Laurienti PJ, Wallace MT, Maldjian JA, Susi CM, Stein BE, Burdette JH (2003) Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Hum Brain Mapp 19:213–223
Lauritzen M, Gold L (2003) Brain function and neurophysiological correlates of signals used in functional neuroimaging. J Neurosci 23:3972–3980
Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cereb Cortex 10:873–888
Logothetis NK (2003a) The underpinnings of the BOLD functional magnetic resonance imaging signal. J Neurosci 23:3963–3971
Logothetis NK (2003b) MR imaging in the non-human primate: studies of function and of dynamic connectivity. Curr Opin Neurobiol 13:630–642
Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412:150–157
Macaluso E, Frith C, Driver J (2000a) Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. J Neurophysiol 83:3062–3075
Macaluso E, Frith CD, Driver J (2000b) Modulation of human visual cortex by crossmodal spatial attention. Science 289:1206–1208
Macaluso E, Frith CD, Driver J (2001) Multimodal mechanisms of attention related to rates of spatial shifting in vision and touch. Exp Brain Res 137:445–454
Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21:725–732
Margulis M, Tang C-M (1998) Temporal integration can readily switch between sublinear and supralinear summation. J Neurophysiol 79:2809–2813
Mathiesen C, Caesar K, Akgoren N, Lauritzen M (1998) Modification of activity-dependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex. J Physiol 512(Pt 2):555–566
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748
Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391
Meredith MA, Stein BE (1986a) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J Neurophysiol 56:640–662
Meredith MA, Stein BE (1986b) Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res 365:350–354
Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857
Meredith MA, Nemitz JW, Stein BE (1987) Determinants of multisensory integration in superior colliculus neurons I Temporal factors. J Neurosci 7:3215–3229
Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88:181–186
Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14:452–465
Nozawa G, Stanford TR, Vaughan JW, Quessy S, Kadunce D, Stein BE (1997) A factorial approach to modeling multisensory integration in the superior colliculus. Soc Neurosci Abstr 23:451
Ogawa S, Lee TM, Kay AR, Tank DW (1990) Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proc Natl Acad Sci USA 87:9868–9872
Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2003) Neuron-specific response characteristics predict the magnitude of multisensory integration. J Neurophysiol 90:4022–4026
Perrault TJ, Jr, Vaughan JW, Stein BE, Wallace MT (2005) Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiol 93:2575–2586
Populin LC, Yin TC (2002) Bimodal interactions in the superior colliculus of the behaving cat. J Neurosci 22:2826–2834
Quessy S, Sweatt A, Stein BE, Stanford TR (2000) The influence of stimulus intensity and timing on the responses of multisensory neurons in the superior colliculus: comparison to a model’s prediction. Soc Neurosci Abstr 26:1221
Raij T, Uutela K, Hari R (2000) Audiovisual integration of letters in the human brain. Neuron 28:617–625
Rees G, Friston K, Koch C (2000) A direct quantitative relationship between the functional properties of human and macaque V5. Nat Neurosci 3:716–723
Schurmann M, Kolev V, Menzel K, Yordanova J (2002) Spatial coincidence modulates interaction between visual and somatosensory evoked potentials. Neuroreport 13:779–783
Selemon LD, Rajkowska G, Goldman-Rakic PS (1998) Elevated neuronal density in prefrontal area 46 in brains from schizophrenic patients: application of a three-dimensional, stereologic counting method. J Comp Neurol 392:402–412
Shams L, Kamitani Y, Thompson S, Shimojo S (2001) Sound alters visual evoked potentials in humans. Neuroreport 12:3849–3852
Soltysik DA, Peck KK, White KD, Crosson B, Briggs RW (2004) Comparison of hemodynamic response nonlinearity across primary cortical areas. Neuroimage 22:1117–1127
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA
Stein BE, Jiang W, Stanford TR (2004a) Multisensory integration in single neurons in midbrain and cortex. In: Calvert G, Spence C, Stein BE (eds) A handbook of multisensory processes. MIT Press, Cambridge, MA, pp 243–264
Stein BE, Stanford TR, Wallace MT, Vaughan JW, Jiang W (2004b) Cross-modal spatial interactions in subcortical and cortical circuits. In: Driver J (ed) Crossmodal and crossmodal attention. Oxford University Press, Oxford, pp 25–50
Wallace MT, Stein BE (1996) Sensory organization of the superior colliculus in cat and monkey. Prog Brain Res 112:301–311
Wallace MT, Meredith MA, Stein BE (1992) Integration of multiple sensory modalities in cat cortex. Exp Brain Res 91:484–488
Wallace MT, Meredith MA, Stein BE (1993) Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J Neurophysiol 69:1797–1809
Wallace MT, Wilkinson LK, Stein BE (1996) Representation and integration of multiple sensory inputs in primate superior colliculus. J Neurophysiol 76:1246–1266
Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010
Wallace MT, Ramachandran R, Stein BE (2004) A revised view of sensory cortical parcellation. Proc Natl Acad Sci USA 101:2167–2172
Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 13:1034–1043
de Zubicaray GI, McMahon KL, Eastburn MM, Wilson SJ (2002) Orthographic/phonological facilitation of naming responses in the picture-word task: an event-related fMRI study using overt vocal responding. NeuroImage 16:1084–1093
Acknowledgments
Supported in part by Grants NS 42568 (PJL), MH 63861 (MTW), and NS 36916 (BES).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Laurienti, P.J., Perrault, T.J., Stanford, T.R. et al. On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Exp Brain Res 166, 289–297 (2005). https://doi.org/10.1007/s00221-005-2370-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-005-2370-2