Research report
Early auditory–visual interactions in human cortex during nonredundant target identification

https://doi.org/10.1016/S0926-6410(02)00058-7Get rights and content

Abstract

A common finding of behavioral studies is that objects characterized by redundant multisensory cues are identified more rapidly than the same objects presented in either unimodal condition. In a previous electrophysiological study in humans, we have described a network of crossmodal interactions that could be associated with this facilitation effect [M.H. Giard, F. Peronnet, J. Cogn. Neurosci. 11(5) (1999) 473–490]. Here, we sought to determine whether the recognition of objects characterized by nonredundant bimodal components may still induce crossmodal neural interactions. Subjects had to identify three objects defined either by auditory or visual features alone, or by the combination of nonredundant auditory and visual features. As expected, behavioral measures showed no sign of facilitation in bimodal processing. Yet, event-related potential analysis revealed the existence of early (<200 ms latency) crossmodal activities in sensory-specific and nonspecific cortical areas, that were partly dependent on the sensory dominance of the subjects to perform the task. Comparative analysis of the interaction patterns involved in redundant and nonredundant cue processing provides evidence for the robustness of the principle of crossmodal neural synergy that applies whatever the stimulus content (redundant or nonredundant information), and for the high flexibility of the neural networks of integration that are sensitive both to the nature of the perceptual task and to the sensory skill of the individual in that particular task.

Introduction

Multisensory integration is an essential feature of perception. Yet, we are still largely ignorant of the brain processes by which an object characterized by components from several sensory modalities is perceived as a unitary event. Animal studies have described the existence and properties of multisensory neurons in the superior colliculus and in polysensory cortex of mammals, and specified a set of neural ‘principles’ by which crossmodal cues are integrated (reviewed in Ref. [29]).

In humans, a general finding of behavioral studies is that the processing of a stimulus containing redundant bimodal information (e.g. seeing and hearing a barking dog) is more rapid than that of either unimodal stimulus which composes it (seeing the dog or hearing barking). This observation, known as the ‘redundant target effect’ [20], [21] has given rise to several theoretical explanations. The race model assumes that the shorter reaction times are due to triggering the response on the basis of the first detected cue [25]; the independent coactivation model considers that the two unimodal cues induce separate activations that are summed to evoke the response; the interactive coactivation model, in turn, claims that the presence of a target in one modality influences the processing of a target in the other modality [20], with several possible loci of interactions (sensory analysis, response selection or motor execution) [11], [15], [17], [20], [22], [28].

Recent neuroimaging studies in humans have provided neurophysiological support for this last model [4], [5], [14], [16]. In addition, a previous event-related potential (ERP) study in our group showed that the crossmodal interactions could occur already at very early stages of sensory analysis [13]. In that experiment, the subjects’ task was to identify, at each trial, which of two objects was presented by pressing one of two keys. The two objects were defined either by auditory or visual features alone, or by the combination of their respective unimodal features, and were presented randomly in either form. As expected, the subjects were more rapid and accurate to identify the objects in bimodal than in unimodal conditions. Spatiotemporal ERP analysis revealed multiple crossmodal interaction patterns between 40 and 200 ms post-stimulus in sensory-specific visual and auditory cortices and in nonspecific areas (right fronto-temporal scalp regions). Furthermore, it was found that the organization of the interaction networks partly depended on the dominant sensory modality of the subjects to achieve the task, the predominant effects being evident in the sensory cortex of the nondominant modality.

Here, we examine the particular case where the object to be identified is characterized by nonredundant bimodal information, that is, the unimodal cues taken in isolation are not predictive of the target identity. In this case, no facilitation effect is expected in the behavioral measures since the two sensory cues must be completely analyzed (until identification) prior to selecting the response. In line with the cognitive models above, one may therefore predict, a priori, that the two unimodal cues will be processed separately and independently (at least during the early stages of sensory analysis). The present experiment was designed to test this hypothesis. In addition, to compare more rigorously the brain operations involved in redundant and nonredundant bimodal processing, we used the same physical stimuli and protocol as in our previous experiment [13], and added experimental conditions to induce nonredundancy.

Section snippets

Subjects

Twenty-three healthy right-handed subjects (mean age: 21.7, 11 females) participated as paid volunteers. All were free of neurological diseases, had normal hearing and normal or corrected-to-normal vision. Written informed consent was obtained from each subject.

Stimuli

Three objects (A, B and C) were used (Fig. 1). Object A consisted of either the transient deformation of a basic circle to an horizontal ellipse (stimulus V1), or a tone burst of 530 Hz (stimulus A1), or the conjunction of these two

Results

Since our previous experiment had shown that cross-modal neural processes in cortex may partly depend on the dominant sensory modality of the subject for the considered task [13], the interaction patterns were analyzed in all subjects, and separately in the so-called auditory-dominant and visually-dominant subjects (AUD and VIS subjects, respectively; see below).

Discussion

The main result of this study is the finding of early crossmodal neural activities during the recognition of objects characterized by nonredundant auditory and visual components. As expected, recognition was not facilitated by the presence of bimodal cues (no reaction time effect in the behavioral measures) since both sensory features had to be completely analyzed—until identification—to achieve the task. Yet, although the crossmodal activities were fewer, weaker in amplitude and later in

Acknowledgements

We are grateful to J.F. Echallier and P.E. Aguera for technical assistance.

References (33)

  • H.G. Vaughan et al.

    The sources of auditory evoked responses recorded from the human scalp

    Electroencephalogr. Clin. Neurophysiol.

    (1970)
  • K.O. Bushara et al.

    Neural correlates of auditory–visual stimulus onset asynchrony detection

    J. Neurosci.

    (2001)
  • G.A. Calvert et al.

    Response amplification in sensory-specific cortices during crossmodal binding

    NeuroReport

    (1999)
  • V.P. Clark et al.

    Spatial selective attention affects early extrastriate but not striate components of the visual evoked potential

    J. Cogn. Neurosci.

    (1996)
  • J. Downar et al.

    A multimodal cortical network for the detection of changes in the sensory environment

    Nat. Neurosci.

    (2000)
  • M. Eimer et al.

    An event-related brain potential study of cross-modal links in spatial attention between vision and touch

    Psychophysiology

    (2000)
  • Cited by (0)

    View full text