Momentary reductions of attention can have extremely adverse outcomes, but it remains unclear whether increased distraction from irrelevant stimuli contributes to such outcomes. To investigate this hypothesis, we examined trial-by-trial relationships between brain activity and response time in twenty healthy adults while they performed a cross-modal selective attention task. In each trial, participants identified a relevant visual letter while ignoring an irrelevant auditory letter, which was mapped either to the same response as the visual letter (congruent trials) or to a different response (incongruent trials). As predicted, reductions of attention (i.e., increases of response time) were associated not only with decreased activity in sensory regions that processed the relevant visual stimuli, suggesting a failure to enhance the processing of those stimuli, but also with increased activity in sensory regions that processed the irrelevant auditory stimuli, suggesting a failure to suppress the processing of those stimuli. Reductions of attention were also linked to larger increases of activity in incongruent than in congruent trials in anterior cingulate regions that detect response conflict, suggesting that failing to suppress the sensory processing of the irrelevant auditory stimuli during attentional reductions allowed those stimuli to more readily activate conflicting responses in incongruent trials. These findings indicate that heightened levels of distraction during momentary reductions of attention likely stem, at least in part, from increased processing of irrelevant stimuli.