Selective integration of auditory-visual looming cues by humans
Introduction
An organism's evolutionary success partially depends on both the ability to reliably detect and discriminate between predators and prey in the environment and also to appropriately respond to them. When encountering an approaching or looming object, one must determine whether to avoid it (a defensive action) or confront it (an aggressive action). Similarly, when encountering a distancing or receding object, one can on the one hand be more assured of one's own safety or can alternatively use this information to determine whether or not pursuit would be worthwhile. In these (and other) ways, simple spatial cues can confer ethologically meaningful information. Given the potentially mortal cost of missing or misinterpreting looming signals, it is unsurprising that ethologists and neuroscientists consider preferential responsiveness to looming signals to be an evolved capacity (Ghazanfar, Neuhoff, & Logothetis, 2002; Graziano & Cooke, 2006; Maier, Chandrasekaran, & Ghazanfar, 2008; Maier, Neuhoff, Logothetis, & Ghazanfar, 2004; Neuhoff, 1998, Neuhoff, 2001, Schiff, 1965; Schiff, Caviness, & Gibson, 1962; Seifritz et al., 2002). Moreover, these situations, like many perceptual events, can likely be facilitated by the integration of multisensory cues to enhance perception and render behavior quicker and/or more accurate (Stein & Meredith, 1993; Welch & Warren, 1980).
Multisensory interactions are a fundamental feature of brain organization (Calvert, Spence, & Stein, 2004; Ghazanfar & Schroeder, 2006; Stein & Meredith, 1993; Stein & Stanford, 2008). Studies are increasingly revealing how the brain achieves such multisensory integration. Anatomic evidence now exists for direct projections between unisensory, even primary, cortices (Cappe & Barone, 2005; Falchier, Clavagnier, Barone, & Kennedy, 2002; Rockland & Ojima, 2003). At a functional level, auditory-visual multisensory interactions occur early in time post-stimulus onset and also within areas typically considered unisensory, again including even primary cortices (e.g. Giard & Peronnet, 1999; Martuzzi et al., 2007, Molholm et al., 2002; Romei, Murray, Merabet, & Thut, 2007). From such findings, new models of brain organization are being developed that incorporate the occurrence of multisensory interactions and integration both at low and high levels of processes and also at early and late time periods following stimulus presentation (Driver & Noesselt, 2008; Ghazanfar & Schroeder, 2006; Stein & Stanford, 2008; Wallace, Ramachandran, & Stein, 2004).
Given this shift in our conceptualization of brain organization, it is increasingly important to understand the functional significance of multisensory interactions as well as the circumstances governing their occurrence. The seminal works of Stein and Meredith (1993) offer several ‘rules’ of multisensory processing based on receptive field properties of single neurons. More recent data nuance these rules by showing that patterns of interactive effects can be impacted developmentally or through experience (Wallace, Carriere, Perrault, Vaughan, & Stein, 2006; Wallace & Stein, 2007) or even by the spatial heterogeneity within single neurons’ receptive fields (Carriere, Royal, & Wallace, 2008). To date, the overwhelming majority of studies have investigated the influences of spatial information on multisensory processing using variation in azimuth or elevation (i.e. 2-dimensional variation in location with respect to the observer). There is comparatively sparse evidence regarding the integration of signals across spatial positions towards versus away from an observer.
Notable exceptions have demonstrated that rhesus monkeys preferentially looked at a looming visual stimulus when presented with a looming, but not receding, sound (Maier et al., 2004). Similarly, 5-month-old infants preferentially looked at matching visual stimuli when presented either with a looming or receding sound (Walker-Andrews & Lennon, 1985). Even though effects were selective for structured sounds instead of noises, the results were only qualitatively suggestive of integrative processes and they did not reveal whether neural response interactions need forcibly be evoked. Likewise, the measurement of looking time cannot differentiate effects occurring at a perceptual level from those driven by biases in attention. Studies of multisensory distance perception by adult humans have predominantly focused on the estimation of time to arrival and remain controversial as whether (and how) auditory and visual distance cues interact and whether or not there is a benefit from multisensory stimulation (Gordon & Rosenblum, 2005; Lewald & Guski, 2004; Sugita & Suzuki, 2003). Moreover, the interpretation of such studies in terms of a neurophysiologic mechanism of either temporal or spatial perception is made complicated by the consistent finding that listeners overestimate the loudness and underestimate the distance of looming sounds (Neuhoff, 1998, Seifritz et al., 2002).
As such, it remains unknown whether multisensory looming/receding signals are integrated to facilitate behavior. Our study addressed this question in humans using a go/no-go motion detection paradigm with unisensory (visual or auditory) and multisensory (simultaneous auditory-visual) stimuli. The perception of visual motion in depth was induced with a central disc that contracted, expanded, or remained constant (i.e. static). The perception of auditory motion in depth was induced with a complex tone that fell or rose in intensity or remained constant (Fig. 1). To ensure that observers used dynamic information in the stimuli, all conditions were initially of the same size/intensity. We assessed multisensory integration of motion perception as measured by reaction times for motion detection (irrespective of its direction or congruence between the senses) and subjective ratings of movement intensity (using a 5-point Likert scale). Performance on multisensory conditions was then compared with that from the constituent unisensory conditions to determine if performance was significantly facilitated to a degree consistent with integrative processes. Finally, the comparison of performance across different multisensory conditions allowed us to determine whether there is selective facilitation for processing multisensory looming signals by humans.
Section snippets
Methods
Sixteen healthy individuals (aged 18–32 years: mean = 25 years; 7 women and 9 men) with normal hearing and normal or corrected-to-normal vision participated. All participants provided written informed consent to the procedures that were approved by the Ethics Committee of the Faculty of Biology and Medicine of the University of Lausanne. The main experiment involved the go/no-go detection of moving versus static stimuli that could be auditory, visual, or multisensory auditory-visual (A, V, and
Multisensory integration of perceived motion in depth
In a first set of analyses, we evaluated if there was evidence for multisensory integration of looming and receding auditory-visual stimulus pairs and if such was affected by the congruence in the direction of perceived motion between the senses. This was done by testing for a redundant signals effect (RSE) (Giard & Peronnet, 1999; Martuzzi et al., 2007, Miller, 1982, Molholm et al., 2002, Raab, 1962, Romei et al., 2007; Schröger & Widmann, 1998) on reaction times (RTs) and movement ratings,
Discussion
Our demonstration of selective integration of multisensory looming signals that affects both reaction times and subjective experience has direct implications for how longstanding principles of multisensory integration, established through parametric variation of position, timing and effectiveness, are to be considered alongside ethologically salient stimuli such as looming signals. The ‘spatial rule’ (Stein & Meredith, 1993) is based on relative superposition of a neuron's excitatory zones,
Acknowledgement
This work has been supported by the Swiss National Science Foundation (grant #3100AO-118419 to MMM) and the Leenaards Foundation (to MMM and GT).
References (44)
- et al.
Subthreshold auditory inputs to extrastriate visual neurons are responsive to parametric changes in stimulus quality: Sensory-specific versus non-specific coding
Brain Research
(2008) - et al.
Rising sound intensity: an intrinsic warning cue activating the amygdala
Cerebral Cortex
(2008) Statistical criteria in fMRI studies of multisensory integration
Neuroinformatics
(2005)- et al.
Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey
European Journal of Neuroscience
(2005) - et al.
Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions
Journal of Neurophysiology
(2008) - et al.
The race model inequality: interpreting a geometric measure of the amount of violation
Psychological Review
(2006) - et al.
Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments
Neuron
(2008) - et al.
Anatomical evidence of multimodal integration in primate striate cortex
Journal of Neuroscience
(2002) - et al.
Auditory looming perception in rhesus monkeys
Proceedings of the National Academy of Sciences USA
(2002)
Is neocortex essentially multisensory?
Trends in Cognitive Sciences
Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study
Journal of Cognitive Neuroscience
Multisensory processing in the redundant target effect: A behavioral and ERP study
Perception & Psychophysics
Effects of intrastimulus modality change on audiovisual time-to-arrival judgments
Perception & Psychophysics
Parieto-frontal interactions, personal space, and defensive behavior
Neuropsychologia
Sensory integration and the perceptual experience of persons with autism
Journal of Autism and Developmental Disorders
Sensory correlations in autism
Autism
Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception
Neuroscience Letters
Integration of bimodal looming signals through neuronal coherence in the temporal lobe
Current Biology
Looming biases in monkey auditory cortex
Journal of Neuroscience
Multisensory integration of looming signals by rhesus monkeys
Neuron
Multisensory interactions within human primary cortices revealed by BOLD dynamics
Cerebral Cortex
Cited by (95)
No selective integration required: A race model explains responses to audiovisual motion-in-depth
2022, CognitionCitation Excerpt :We were interested in whether such biases also occur in the multisensory processing of audiovisual motion-in-depth. We have here successfully replicated findings by Cappe et al. (2009) showing that RTs to congruent looming motion are faster compared to congruent receding motion and any incongruent combination (Table 2). But our additional analyses lead us to a rather different conclusion.
Cross-modal integration and plasticity in the superior temporal cortex
2022, Handbook of Clinical NeurologyThe effect of audio cues and sound source stimuli on the perception of approaching objects
2020, Applied Acoustics