Multisensory conflict modulates the spread of visual attention across a multisensory object

Abstract

Spatial attention to a visual stimulus that occurs synchronously with a task-irrelevant sound from a different location can lead to increased activity not only in the visual cortex, but also the auditory cortex, apparently reflecting the object-related spreading of attention across both space and modality (Busse et al., 2005). The processing of stimulus conflict, including multisensory stimulus conflict, is known to activate the anterior cingulate cortex (ACC), but the interactive influence on the sensory cortices remains relatively unexamined. Here we used fMRI to examine whether the multisensory spread of visual attention across the sensory cortices previously observed will be modulated by whether there is conceptual or object-related conflict between the relevant visual and irrelevant auditory inputs. Subjects visually attended to one of two lateralized visual letter-streams while synchronously occurring, task-irrelevant, letter sounds were presented centrally, which could be either congruent or incongruent with the visual letters. We observed significant enhancements for incongruent versus congruent letter–sound combinations in the ACC and in the contralateral visual cortex when the visual component was attended, presumably reflecting the conflict detection and the need for boosted attention to the visual stimulus during incongruent trials. In the auditory cortices, activity increased bilaterally if the spatially discordant auditory stimulation was incongruent, but only in the left, language-dominant side when congruent. We conclude that a conflicting incongruent sound, even when task-irrelevant, distracts more strongly than a congruent one, leading to greater capture of attention. This greater capture of attention in turn results in increased activity in the auditory cortex.

Introduction

Attentional selectivity, the ability to select one particular source of information to guide action while ignoring others that are irrelevant to the current behavioral goal, is a critical cognitive function for successful navigation in the world. Our attentional selection is enhanced when the behaviorally relevant stimuli belong to the same object. Unisensory visual studies of object-based attentional selection have shown an apparently automatic spread of visual attention through a visual cued object (Egly et al., 1994, Abrams and Law, 2000). Correspondingly, an fMRI study has reported higher activation in human visual cortex representing locations in a cued object relative to equidistant locations on another uncued object (Müller and Kleinschmidt, 2003).

Based on these results, in our recent study, (Busse et al., 2005) we asked if visual attention can spread cross-modally to encompass other parts of a multisensory object. In that study, subjects attended to one of two lateralized streams consisting of rapidly presented simple visual stimuli. Half of these laterally presented visual stimuli were synchronously accompanied by a centrally presented, task-irrelevant tone. Activity associated with the central auditory stimuli occurring synchronously with an attended versus an unattended lateral visual stimulus was extracted by a set of subtraction contrasts. The EEG analysis revealed a late, sustained, frontally distributed pattern of brain activity for the central tones that occurred in the context of an attended relative to an unattended visual stimulus, an activation that resembled attention-related enhancements seen at earlier latencies during intra-auditory selective attention. The corresponding fMRI contrasts showed that this object-related attention effect included enhanced activity in auditory cortex. The pattern of results strongly suggested that attention to the visual modality can spread to encompass simultaneously occurring signals from the auditory modality, even when the auditory stimuli are task-irrelevant and from a different location.

In the Busse et al. (2005) study, the multisensory spread of visual attention across the sensory cortices during attended visual stimulation was revealed with visual–auditory stimulus combinations that were essentially neutral — i.e., they were neither related nor unrelated. In the present work, we asked whether this ‘spread-of-attention’ activity would be modulated by object-related conflict between the relevant visual input and the irrelevant auditory one. To produce object-related multisensory conflict, the stimulus inputs consisted of task-relevant, laterally presented, visual letters (“A” or “X”) to be discriminated that could be accompanied by simultaneous, task-irrelevant spoken letters presented centrally (also “A” or “X”). If the visual letter was an “A”, the spoken letter could be either congruent (“A”) or incongruent (“X”). In previous studies of such multisensory conflict, all the stimuli were typically presented exclusively from the central position, and these have indicated that reading of written letters can be impeded by incongruent letter–sounds and facilitated by congruent ones (e.g. van Atteveldt et al., 2007). Here, the question was whether and by how such multimodal object-related conflict might modulate the cross-modal ‘spread-of-attention’ activity.

When attention spreads across a multisensory object whose sensory components conflict in some way, we might expect to see increased activity in both the ACC and the visual cortex for incongruent compared to congruent letter/sound combinations. Such effects would be expected for several reasons. First, previous neuroimaging studies have implicated the ACC as being involved in object-related conflict processing (van Veen and Carter, 2005, Weissman et al., 2003, among others). Secondly, in a previous multisensory cueing fMRI study in which the relevant visual and irrelevant auditory letter stimuli were simultaneously presented from a central location (Weissman et al., 2004), the results indicated that conflict induced by co-occurring incongruent auditory letters led to increased activity in both the ACC and the visual cortex (Weissman et al., 2004). The authors concluded that during conflict between the modalities, attention toward the relevant input is enhanced by boosting activity in the sensory cortex of the task-relevant modality. In our present study, given that the visual letters were relevant and auditory letters to be ignored, we predicted activity in the visual cortex would be enhanced during conflicting stimulation to help counter the auditory semantic distraction. In the current case, we could assess whether such incongruency-related activity patterns in the ACC and visual cortex will occur even during the multisensory spreading of attention across space.

An additional important question investigated here, however, is whether and how the ‘spread-of-attention’ activity in the auditory cortex would differ for congruent versus incongruent semantic objects. On the one hand, there is evidence (e.g. Molholm et al., 2007) that visual attention would only spread cross-modally if both modalities involved were associated with the same object, and not if they were linked to different objects. In a recent EEG study, Molholm et al. (2007) presented a central stream of alternating pictures and sounds to subjects, with a task to detect a specific picture while ignoring the sounds. Results indicated an increased auditory ERP negativity beginning around 200 ms when the response to the ignored sound corresponded to the attended visual object (e.g. barking sound and dog picture) relative to when the sound that was not a feature of the attended visual object (e.g. barking sound and guitar picture). Thus, the object-based spread of visual attention to the unattended auditory modality was greater when the visual and auditory semantic features matched (Molholm et al., 2007; also see Fiebelkorn et al., 2010).

On the other hand, a task-irrelevant sound that is to be ignored might be more distracting when incongruent with a relevant visual stimulus. Distracting stimuli are known to capture attention (e.g. Sabri et al., 2006, Watkins et al., 2007; cf. Dalton and Spence, 2007 for multisensory distraction). For example, a combined EEG/fMRI auditory study revealed that the more an irrelevant auditory deviant was perceived as distracting, the higher the elicited activity in the auditory cortex (Sabri et al., 2006; see also Watkins et al., 2007).Thus, for the auditory cortex, two opposing predictions could be made. One prediction hypothesizes that there would be greater ‘spreading-of-attention’ activity from the attended visual component to the synchronous task-irrelevant auditory component when they are congruent, due explicitly to their congruence engendering increased activity. Enhanced activity for congruent trials relative to incongruent ones could also be because as soon as the incongruency is detected the brain would respond by quickly sending inhibitory signals to suppress its processing. In contrast, if a conflicting incongruent sound (even when task-irrelevant) distracts more strongly than a congruent one, it may trigger a greater capture of attention, thereby leading to more activity in auditory cortex.

In summary, we specifically asked how the ‘spread-of-attention’ from an attended visual stimulus to a synchronous task-irrelevant auditory stimulus arising from another location would be modulated differentially as a function of whether these components conflicted. Accordingly, we presented congruent and incongruent letter–sound combinations while subjects visually attended one of two lateralized visual letter-streams and ignored the synchronously occurring task-irrelevant central sounds, which could be either semantically congruent or incongruent. Analyses of the interactions of attention by congruency would then provide insight into the mechanisms of the multisensory ‘spread-of-attention’ process, and into multisensory feature processing interactions more generally.