Skip to main content
Log in

An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI

Experimental Brain Research Aims and scope Submit manuscript

Abstract

It can be shown empirically and theoretically that inferences based on established metrics used to assess multisensory integration with BOLD fMRI data, such as superadditivity, are dependent on the particular experimental situation. For example, the law of inverse effectiveness shows that the likelihood of finding superadditivity in a known multisensory region increases with decreasing stimulus discriminability. In this paper, we suggest that Sternberg’s additive-factors design allows for an unbiased assessment of multisensory integration. Through the manipulation of signal-to-noise ratio as an additive factor, we have identified networks of cortical regions that show properties of audio-visual or visuo-haptic neuronal convergence. These networks contained previously identified multisensory regions and also many new regions, for example, the caudate nucleus for audio-visual integration, and the fusiform gyrus for visuo-haptic integration. A comparison of integrative networks across audio-visual and visuo-haptic conditions showed very little overlap, suggesting that neural mechanisms of integration are unique to particular sensory pairings. Our results provide evidence for the utility of the additive-factors approach by demonstrating its effectiveness across modality (vision, audition, and haptics), stimulus type (speech and non-speech), experimental design (blocked and event-related), method of analysis (SPM and ROI), and experimenter-chosen baseline. The additive-factors approach provides a method for investigating multisensory interactions that goes beyond what can be achieved with more established metric-based, subtraction-type methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Allman BL, Keniston LP, Meredith MA (2008) Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: sensory-specific versus non-specific coding. Brain Res 1242:95–101

    Article  PubMed  CAS  Google Scholar 

  • Ashby FG (1982) Testing the assumptions of exponential, additive reaction time models. Mem Cogn 10:125–134

    CAS  Google Scholar 

  • Ashby FG, Townsend JT (1986) Varieties of perceptual independence. Psychol Rev 93:154–179

    Article  PubMed  CAS  Google Scholar 

  • Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113

    Article  PubMed  Google Scholar 

  • Beauchamp MS, Lee KE, Argall BD, Martin A (2004) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823

    Article  PubMed  CAS  Google Scholar 

  • Binder JR, Frost JA, Hammeke TA, Bellgowan PS, Rao SM, Cox RW (1999) Conceptual processing during the conscious resting state. A functional MRI study. J Cogn Neurosci 11:80–95

    Article  PubMed  CAS  Google Scholar 

  • Boynton GM, Engel SA, Glover GH, Heeger DJ (1996) Linear systems analysis of functional magnetic resonance imaging in human V1. J Neurosci 16:4207–4221

    PubMed  CAS  Google Scholar 

  • Brown JW, Braver TS (2005) Learned predictions of error likelihood in the anterior cingulate cortex. Science 307:1118–1121

    Article  PubMed  CAS  Google Scholar 

  • Brown JW, Braver TS (2007) Risk prediction and aversion by anterior cingulate cortex. Cogn Affect Behav Neurosci 7:266–277

    Article  PubMed  Google Scholar 

  • Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14:427–438

    Article  PubMed  CAS  Google Scholar 

  • Chudler EH, Sugiyama K, Dong WK (1995) Multisensory convergence and integration in the neostriatum and globus pallidus of the rat. Brain Res 674:33–45

    Article  PubMed  CAS  Google Scholar 

  • Dale AM, Buckner RL (1997) Selective averaging of rapidly presented individual trials using fMRI. Hum Brain Mapp 5:329–340

    Article  Google Scholar 

  • Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150

    Article  PubMed  CAS  Google Scholar 

  • Dolan RJ, Morris JS, de Gelder B (2001) Crossmodal binding of fear in voice and face. Proc Natl Acad Sci USA 98:10006–10010

    Article  PubMed  CAS  Google Scholar 

  • Donders FC (1868) Over de Snelheid van Psychische Processen. In: Onderzoekingen Gedaan in het Psychologisch Laboratorium der Utrechetsche Hoogeschool, pp 92–120

  • Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57:11–23

    Article  PubMed  CAS  Google Scholar 

  • Forman SD, Cohen JD, Fitzgerald M, Eddy WF, Mintun MA, Noll DC (1995) Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): use of a cluster-size threshold. Magn Reson Med 33:636–647

    Article  PubMed  CAS  Google Scholar 

  • Giard MH, Peronnet F (1999) Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. J Cogn Neurosci 11:473–490

    Article  PubMed  CAS  Google Scholar 

  • Glover GH (1999) Deconvolution of impulse response in event-related BOLD fMRI. Neuroimage 9:416–429

    Article  PubMed  CAS  Google Scholar 

  • Heeger DJ, Ress D (2002) What does fMRI tell us about neuronal activity? Nat Rev Neurosci 3:142–151

    Article  PubMed  CAS  Google Scholar 

  • James TW, Servos P, Kilgour AR, Huh E, Lederman S (2006) The influence of familiarity on brain activation during haptic exploration of 3-D facemasks. Neurosci Lett 397:269–273

    Article  PubMed  CAS  Google Scholar 

  • Kanwisher N, Yovel G (2006) The fusiform face area: a cortical region specialized for the perception of faces. Philos Trans R Soc Lond B Biol Sci 361:2109–2128

    Article  PubMed  Google Scholar 

  • Kilgour AR, Kitada R, Servos P, James TW, Lederman SJ (2005) Haptic face identification activates ventral occipital and temporal areas: an fMRI study. Brain Cogn 59:246–257

    Article  PubMed  Google Scholar 

  • Kim S, James TW (2009) Enhanced Effectiveness in visuo-haptic object-selective brain regions with increasing stimulus saliency (under review)

  • Kreifelts B, Ethofer T, Grodd W, Erb M, Wildgruber D (2007) Audiovisual integration of emotional signals in voice and face: an event-related fMRI study. Neuroimage 37:1445–1456

    Article  PubMed  Google Scholar 

  • Lewis JW, Beauchamp MS, DeYoe EA (2000) A comparison of visual and auditory motion processing in human cerebral cortex. Cereb Cortex 10:873–888

    Article  PubMed  CAS  Google Scholar 

  • Macaluso E, George N, Dolan R, Spence C, Driver J (2004) Spatial and temporal factors during processing of audiovisual speech: a PET study. Neuroimage 21:725–732

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Wallace MT, Stein BE (1992) Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection. Exp Brain Res 88:181–186

    Article  PubMed  CAS  Google Scholar 

  • Molholm S, Ritter W, Murray MM, Javitt DC, Schroeder CE, Foxe JJ (2002) Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Brain Res Cogn Brain Res 14:115–128

    Article  PubMed  Google Scholar 

  • Nagy A, Paroczy Z, Norita M, Benedek G (2005) Multisensory responses and receptive field properties of neurons in the substantia nigra and in the caudate nucleus. Eur J Neurosci 22:419–424

    Article  PubMed  Google Scholar 

  • Nagy A, Eordegh G, Paroczy Z, Markus Z, Benedek G (2006) Multisensory integration in the basal ganglia. Eur J Neurosci 24:917–924

    Article  PubMed  Google Scholar 

  • Pieters JPM (1983) Sternberg’s additive factor method and underlying psychological processes: some theoretical consideration. Psychol Bull 93:411–426

    Article  PubMed  CAS  Google Scholar 

  • Puce A, Allison T, Gore JC, McCarthy G (1995) Face-sensitive regions in human extrastriate cortex studied by functional MRI. J Neurophysiol 74:1192–1199

    PubMed  CAS  Google Scholar 

  • Sartori G, Umilta C (2000) The additive factor method in brain imaging. Brain Cogn 42:68–71

    Article  PubMed  CAS  Google Scholar 

  • Saxe R, Brett M, Kanwisher N (2006) Divide and conquer: a defense of functional localizers. Neuroimage 30:1088–1096 discussion 1097–1099

    Article  PubMed  Google Scholar 

  • Schweickert R (1978) A critical path generalization of the additive factor method: analysis of a Stroop task. J Math Psychol 18:105–139

    Article  Google Scholar 

  • Senkowski D, Saint-Amour D, Kelly SP, Foxe JJ (2007) Multisensory processing of naturalistic objects in motion: a high-density electrical mapping and source estimation study. Neuroimage 36:877–888

    Article  PubMed  Google Scholar 

  • Sestieri C, Di Matteo R, Ferretti A, Del Gratta C, Caulo M, Tartaro A, Olivetti Belardinelli M, Romani GL (2006) “What” versus “where” in the audiovisual domain: an fMRI study. Neuroimage 33:672–680

    Article  PubMed  CAS  Google Scholar 

  • Stark CE, Squire LR (2001) When zero is not zero: the problem of ambiguous baseline conditions in fMRI. Proc Natl Acad Sci USA 98:12760–12766

    Article  PubMed  CAS  Google Scholar 

  • Sternberg S (1969a) The discovery of processing stages: extensions of Donders’ method. Acta Psychol 30:315–376

    Google Scholar 

  • Sternberg S (1969b) Memory-scanning: mental processes revealed by reaction-time experiments. Am Sci 57:421–457

    PubMed  CAS  Google Scholar 

  • Sternberg S (1975) Memory scanning: New findings and current controversies. Exp Pshychol 27:1–32

    Article  Google Scholar 

  • Sternberg S (1998) Discovering mental processing stages: the method of additive factors. In: Scarborough D, Sternberg S (eds) An invitation to cognitive science: vol 4, methods, models, and conceptual issues, vol 4. MIT Press, Cambridge, pp 739–811

    Google Scholar 

  • Sternberg S (2001) Seperate modifiability, mental modules, and the use of pure and composite measures to reveal them. Acta Psychol 106:147–246

    Article  CAS  Google Scholar 

  • Stevens SS (1946) On the theory of scales of measurement. Science 103:677–680

    Article  PubMed  Google Scholar 

  • Stevenson RA, James TW (2009) Audiovisual integration in human superior temporal sulcus: inverse effectiveness and the neural processing of speech and object recognition. Neuroimage 44:1210–1223

    Article  PubMed  Google Scholar 

  • Stevenson RA, Geoghegan ML, James TW (2007) Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Exp Brain Res 179:85–95

    Article  PubMed  Google Scholar 

  • Talaraich J, Tournoux P (1988) Co-planar stereotaxic atlas of the human brain. Thieme Medical Publishers, New York

  • Taylor DA (1976) Stage analysis of reaction time. Psychol Bull 83:161–191

    Article  PubMed  CAS  Google Scholar 

  • Thirion B, Pinel P, Meriaux S, Roche A, Dehaene S, Poline JB (2007) Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses. Neuroimage 35:105–120

    Article  PubMed  Google Scholar 

  • Townsend JT (1984) Uncovering mental processes with factorial experiments. J Math Psychol 28:363–400

    Article  Google Scholar 

  • Townsend JT, Ashby FG (1980) Decomposing the reaction time distribution: pure insertion and selective influence revisited. J Math Psychol 21:93–123

    Article  Google Scholar 

  • Townsend JT, Thomas RD (1994) Stochastic dependencies in parallel and serial models: effects on systems factorial interactions. J Math Psychol 38:1–34

    Article  Google Scholar 

  • Wenger MJ, Townsend JT (2000) Basic response time tools for studying general processing capacity in attention, perception, and cognition. J Gen Psychol 127:67–99

    Article  PubMed  CAS  Google Scholar 

  • Yeterian EH, Van Hoesen GW (1978) Cortico-striate projections in the rhesus monkey: the organization of certain cortico-caudate connections. Brain Res 139:43–63

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This research was supported in part by the Indiana METACyt Initiative of Indiana University, funded in part through a major grant from the Lilly Endowment, Inc., the IUB Faculty Research Support Program, and the Indiana University GPSO Research Grant. Thanks to Laurel Stevenson, June Young Lee, and Karin Harman James for their support, to David Pisoni, Luiz Hernandez, and Nicholus Altieri for the speech stimuli; Andrew Butler, Hope Cantrell, and Luiz Pessoa for assistance with Experiment 1; and James Townsend and the Indiana University Neuroimaging Group for their insights on this work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ryan A. Stevenson.

Appendix 1

Appendix 1

The additive-factors method

Sternberg’s (1969a, b, 1975) additive-factors method was originally proposed as a reaction-time paradigm that improved upon Donders’ subtraction method (1868). The subtraction method was originally used to measure the time taken to perform individual ‘stages’ of a given task (suppose the task consists of stages a, b, and c). It supposed that the difference in time between performing a task with and without a given stage (b) was equal to the processing time of that stage:

$$ RT\left( {abc} \right) - RT\left( {ac} \right) = RT\left( b \right) .$$
(7)

The subtraction method assumes that the insertion of processing stage b does not in any way effect processing stages a or c. The additive-factors method avoids this fallacy of ‘pure insertion’ (among others) assumed by the Donderian subtraction method (1868) by instead relying upon the assumption of selective influence, which supposes that if there is an experimental factor that selectively influences process a without influencing process b, then the two processes are independent. Sternberg’s idea was to manipulate such an experimental factor that changed the processing time of a process, let us say process a (where a′ includes a manipulated experimental factor). If two processes (a and b) are not interactive, the manipulation of the experimental factor will have an additive effect: that is, the manipulation of the experimental factor will have the same effect on a condition with processes a and b as it would on process a alone:

$$ RT\left( {ab} \right) - RT\left( a \right) = RT\left( {a^\prime b} \right)-RT\left( {a^\prime } \right) .$$
(8)

This would suggest that the processes a and b were indeed separate processes, with the experimental factor selectively influencing process a. However, if the processes are not selectively influenced, there will be an interaction between the processes:

$$ RT\left( {ab} \right) - RT\left( a \right) \ne RT\left( {a^\prime b} \right)-RT\left( {a^\prime } \right) .$$
(9)

Such interactive findings indicate a lack of selective influence of the experimental factor and thus suggest that processes a and b may not be independent.

Since its inception, the additive-factors method has been employed by a wide range of disciplines, including proposed usage in fMRI (Sartori and Umilta 2000), and has been extended and more rigorously generalized (Taylor 1976; Schweickert 1978; Townsend and Ashby 1980; Ashby 1982; Pieters 1983; Townsend 1984; Ashby and Townsend 1986; Townsend and Thomas 1994; Wenger and Townsend 2000; Sternberg 2001). For a more in-depth overview of the additive-factor method, see “Discovering mental processing stages: the method of additive factors” (Sternberg 1998).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stevenson, R.A., Kim, S. & James, T.W. An additive-factors design to disambiguate neuronal and areal convergence: measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Exp Brain Res 198, 183–194 (2009). https://doi.org/10.1007/s00221-009-1783-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-009-1783-8

Keywords

Navigation