One of the most prevailing views on the functional localization of human cognition is the hemispheric specialization, wherein the left and right hemispheres are implicated primarily in verbal and nonverbal functions, respectively. Cognitive control is known to involve the lateral prefrontal cortex. However, it remains unclear whether the hemispheric specialization in the lateral prefrontal cortex can be observed in cognitive control per se, independent of sensory aspects of stimulus materials. In this functional magnetic resonance imaging study, we tested whether the verbal/nonverbal hemispheric specialization applies to the lateral prefrontal activation by investigating interference suppression, the ability to filter out irrelevant information in the environment. The flanker task was employed using a compound stimulus that contained a target and a flanker. The flanked stimulus was either a color word flanked by a colored patch or a colored patch flanked by a color word, which allowed us to manipulate the modality of the presented flanker stimulus from which interference originates, keeping the total stimulus modality balanced. The inferior frontal gyrus (IFG) showed prominent Modality-by-Hemisphere interaction in interference suppression, the left IFG being activated when a word flanker (plus a patch target) was presented and the right IFG being activated when a patch flanker (plus a word target) was presented. These results suggest that the verbal/nonverbal hemispheric specialization in the IFG can be explained by cognitive control processes per se, independent of sensory aspects of presented materials.