Studies of semantic dementia and repetitive TMS have suggested that the bilateral anterior temporal lobes (ATLs) underpin a modality-invariant representational hub within the semantic system. However, it is not clear whether all ATL subregions contribute in the same way. We utilized distortion-corrected fMRI to investigate the pattern of activation in the left and right ATL when participants performed a semantic decision task on auditory words, environmental sounds, or pictures. This showed that the ATL is not functionally homogeneous but is more graded. Both left and right ventral ATL (vATL) responded to all modalities in keeping with the notion that this region underpins multimodality semantic processing. In addition, there were graded differences across the hemispheres. Semantic processing of both picture and environmental sound stimuli was associated with equivalent bilateral vATL activation, whereas auditory words generated greater activation in left than right vATL. This graded specialization for auditory stimuli would appear to reflect the input from the left superior ATL, which responded solely to semantic decisions on the basis of spoken words and environmental sounds, suggesting that this region is specialized to auditory stimuli. A final noteworthy result was that these regions were activated for domain level decisions to singly presented stimuli, which appears to be incompatible with the hypotheses that the ATL is dedicated (a) to the representation of specific entities or (b) for combinatorial semantic processes.