Elsevier

Neuropsychologia

Volume 47, Issue 4, March 2009, Pages 1045-1052
Neuropsychologia

Selective integration of auditory-visual looming cues by humans

https://doi.org/10.1016/j.neuropsychologia.2008.11.003Get rights and content

Abstract

An object's motion relative to an observer can confer ethologically meaningful information. Approaching or looming stimuli can signal threats/collisions to be avoided or prey to be confronted, whereas receding stimuli can signal successful escape or failed pursuit. Using movement detection and subjective ratings, we investigated the multisensory integration of looming and receding auditory and visual information by humans. While prior research has demonstrated a perceptual bias for unisensory and more recently multisensory looming stimuli, none has investigated whether there is integration of looming signals between modalities. Our findings reveal selective integration of multisensory looming stimuli. Performance was significantly enhanced for looming stimuli over all other multisensory conditions. Contrasts with static multisensory conditions indicate that only multisensory looming stimuli resulted in facilitation beyond that induced by the sheer presence of auditory-visual stimuli. Controlling for variation in physical energy replicated the advantage for multisensory looming stimuli. Finally, only looming stimuli exhibited a negative linear relationship between enhancement indices for detection speed and for subjective ratings. Maximal detection speed was attained when motion perception was already robust under unisensory conditions. The preferential integration of multisensory looming stimuli highlights that complex ethologically salient stimuli likely require synergistic cooperation between existing principles of multisensory integration. A new conceptualization of the neurophysiologic mechanisms mediating real-world multisensory perceptions and action is therefore supported.

Introduction

An organism's evolutionary success partially depends on both the ability to reliably detect and discriminate between predators and prey in the environment and also to appropriately respond to them. When encountering an approaching or looming object, one must determine whether to avoid it (a defensive action) or confront it (an aggressive action). Similarly, when encountering a distancing or receding object, one can on the one hand be more assured of one's own safety or can alternatively use this information to determine whether or not pursuit would be worthwhile. In these (and other) ways, simple spatial cues can confer ethologically meaningful information. Given the potentially mortal cost of missing or misinterpreting looming signals, it is unsurprising that ethologists and neuroscientists consider preferential responsiveness to looming signals to be an evolved capacity (Ghazanfar, Neuhoff, & Logothetis, 2002; Graziano & Cooke, 2006; Maier, Chandrasekaran, & Ghazanfar, 2008; Maier, Neuhoff, Logothetis, & Ghazanfar, 2004; Neuhoff, 1998, Neuhoff, 2001, Schiff, 1965; Schiff, Caviness, & Gibson, 1962; Seifritz et al., 2002). Moreover, these situations, like many perceptual events, can likely be facilitated by the integration of multisensory cues to enhance perception and render behavior quicker and/or more accurate (Stein & Meredith, 1993; Welch & Warren, 1980).

Multisensory interactions are a fundamental feature of brain organization (Calvert, Spence, & Stein, 2004; Ghazanfar & Schroeder, 2006; Stein & Meredith, 1993; Stein & Stanford, 2008). Studies are increasingly revealing how the brain achieves such multisensory integration. Anatomic evidence now exists for direct projections between unisensory, even primary, cortices (Cappe & Barone, 2005; Falchier, Clavagnier, Barone, & Kennedy, 2002; Rockland & Ojima, 2003). At a functional level, auditory-visual multisensory interactions occur early in time post-stimulus onset and also within areas typically considered unisensory, again including even primary cortices (e.g. Giard & Peronnet, 1999; Martuzzi et al., 2007, Molholm et al., 2002; Romei, Murray, Merabet, & Thut, 2007). From such findings, new models of brain organization are being developed that incorporate the occurrence of multisensory interactions and integration both at low and high levels of processes and also at early and late time periods following stimulus presentation (Driver & Noesselt, 2008; Ghazanfar & Schroeder, 2006; Stein & Stanford, 2008; Wallace, Ramachandran, & Stein, 2004).

Given this shift in our conceptualization of brain organization, it is increasingly important to understand the functional significance of multisensory interactions as well as the circumstances governing their occurrence. The seminal works of Stein and Meredith (1993) offer several ‘rules’ of multisensory processing based on receptive field properties of single neurons. More recent data nuance these rules by showing that patterns of interactive effects can be impacted developmentally or through experience (Wallace, Carriere, Perrault, Vaughan, & Stein, 2006; Wallace & Stein, 2007) or even by the spatial heterogeneity within single neurons’ receptive fields (Carriere, Royal, & Wallace, 2008). To date, the overwhelming majority of studies have investigated the influences of spatial information on multisensory processing using variation in azimuth or elevation (i.e. 2-dimensional variation in location with respect to the observer). There is comparatively sparse evidence regarding the integration of signals across spatial positions towards versus away from an observer.

Notable exceptions have demonstrated that rhesus monkeys preferentially looked at a looming visual stimulus when presented with a looming, but not receding, sound (Maier et al., 2004). Similarly, 5-month-old infants preferentially looked at matching visual stimuli when presented either with a looming or receding sound (Walker-Andrews & Lennon, 1985). Even though effects were selective for structured sounds instead of noises, the results were only qualitatively suggestive of integrative processes and they did not reveal whether neural response interactions need forcibly be evoked. Likewise, the measurement of looking time cannot differentiate effects occurring at a perceptual level from those driven by biases in attention. Studies of multisensory distance perception by adult humans have predominantly focused on the estimation of time to arrival and remain controversial as whether (and how) auditory and visual distance cues interact and whether or not there is a benefit from multisensory stimulation (Gordon & Rosenblum, 2005; Lewald & Guski, 2004; Sugita & Suzuki, 2003). Moreover, the interpretation of such studies in terms of a neurophysiologic mechanism of either temporal or spatial perception is made complicated by the consistent finding that listeners overestimate the loudness and underestimate the distance of looming sounds (Neuhoff, 1998, Seifritz et al., 2002).

As such, it remains unknown whether multisensory looming/receding signals are integrated to facilitate behavior. Our study addressed this question in humans using a go/no-go motion detection paradigm with unisensory (visual or auditory) and multisensory (simultaneous auditory-visual) stimuli. The perception of visual motion in depth was induced with a central disc that contracted, expanded, or remained constant (i.e. static). The perception of auditory motion in depth was induced with a complex tone that fell or rose in intensity or remained constant (Fig. 1). To ensure that observers used dynamic information in the stimuli, all conditions were initially of the same size/intensity. We assessed multisensory integration of motion perception as measured by reaction times for motion detection (irrespective of its direction or congruence between the senses) and subjective ratings of movement intensity (using a 5-point Likert scale). Performance on multisensory conditions was then compared with that from the constituent unisensory conditions to determine if performance was significantly facilitated to a degree consistent with integrative processes. Finally, the comparison of performance across different multisensory conditions allowed us to determine whether there is selective facilitation for processing multisensory looming signals by humans.

Section snippets

Methods

Sixteen healthy individuals (aged 18–32 years: mean = 25 years; 7 women and 9 men) with normal hearing and normal or corrected-to-normal vision participated. All participants provided written informed consent to the procedures that were approved by the Ethics Committee of the Faculty of Biology and Medicine of the University of Lausanne. The main experiment involved the go/no-go detection of moving versus static stimuli that could be auditory, visual, or multisensory auditory-visual (A, V, and

Multisensory integration of perceived motion in depth

In a first set of analyses, we evaluated if there was evidence for multisensory integration of looming and receding auditory-visual stimulus pairs and if such was affected by the congruence in the direction of perceived motion between the senses. This was done by testing for a redundant signals effect (RSE) (Giard & Peronnet, 1999; Martuzzi et al., 2007, Miller, 1982, Molholm et al., 2002, Raab, 1962, Romei et al., 2007; Schröger & Widmann, 1998) on reaction times (RTs) and movement ratings,

Discussion

Our demonstration of selective integration of multisensory looming signals that affects both reaction times and subjective experience has direct implications for how longstanding principles of multisensory integration, established through parametric variation of position, timing and effectiveness, are to be considered alongside ethologically salient stimuli such as looming signals. The ‘spatial rule’ (Stein & Meredith, 1993) is based on relative superposition of a neuron's excitatory zones,

Acknowledgement

This work has been supported by the Swiss National Science Foundation (grant #3100AO-118419 to MMM) and the Leenaards Foundation (to MMM and GT).

References (44)

  • B.L. Allman et al.

    Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: Sensory-specific versus non-specific coding

    Brain Research

    (2008)
  • D.R. Bach et al.

    Rising sound intensity: an intrinsic warning cue activating the amygdala

    Cerebral Cortex

    (2008)
  • M.S. Beauchamp

    Statistical criteria in fMRI studies of multisensory integration

    Neuroinformatics

    (2005)
  • C. Cappe et al.

    Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey

    European Journal of Neuroscience

    (2005)
  • B.N. Carriere et al.

    Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions

    Journal of Neurophysiology

    (2008)
  • H. Colonius et al.

    The race model inequality: interpreting a geometric measure of the amount of violation

    Psychological Review

    (2006)
  • J. Driver et al.

    Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments

    Neuron

    (2008)
  • A. Falchier et al.

    Anatomical evidence of multimodal integration in primate striate cortex

    Journal of Neuroscience

    (2002)
  • A.A. Ghazanfar et al.

    Auditory looming perception in rhesus monkeys

    Proceedings of the National Academy of Sciences USA

    (2002)
  • A.A. Ghazanfar et al.

    Is neocortex essentially multisensory?

    Trends in Cognitive Sciences

    (2006)
  • M.H. Giard et al.

    Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study

    Journal of Cognitive Neuroscience

    (1999)
  • M. Gondan et al.

    Multisensory processing in the redundant target effect: A behavioral and ERP study

    Perception & Psychophysics

    (2005)
  • M.S. Gordon et al.

    Effects of intrastimulus modality change on audiovisual time-to-arrival judgments

    Perception & Psychophysics

    (2005)
  • M.S. Graziano et al.

    Parieto-frontal interactions, personal space, and defensive behavior

    Neuropsychologia

    (2006)
  • G. Iarocci et al.

    Sensory integration and the perceptual experience of persons with autism

    Journal of Autism and Developmental Disorders

    (2006)
  • J.K. Kern et al.

    Sensory correlations in autism

    Autism

    (2007)
  • J. Lewald et al.

    Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception

    Neuroscience Letters

    (2004)
  • J.X. Maier et al.

    Integration of bimodal looming signals through neuronal coherence in the temporal lobe

    Current Biology

    (2008)
  • J.X. Maier et al.

    Looming biases in monkey auditory cortex

    Journal of Neuroscience

    (2007)
  • J.X. Maier et al.

    Multisensory integration of looming signals by rhesus monkeys

    Neuron

    (2004)
  • R. Martuzzi et al.

    Multisensory interactions within human primary cortices revealed by BOLD dynamics

    Cerebral Cortex

    (2007)
  • Cited by (95)

    • No selective integration required: A race model explains responses to audiovisual motion-in-depth

      2022, Cognition
      Citation Excerpt :

      We were interested in whether such biases also occur in the multisensory processing of audiovisual motion-in-depth. We have here successfully replicated findings by Cappe et al. (2009) showing that RTs to congruent looming motion are faster compared to congruent receding motion and any incongruent combination (Table 2). But our additional analyses lead us to a rather different conclusion.

    View all citing articles on Scopus
    View full text