The role of the anterior temporal lobes (ATLs) in semantic cognition is not clear from the current literature. Semantic dementia patients show a progressive and a specific semantic impairment, following bilateral atrophy of the ATLs. Neuroimaging studies of healthy participants, however, do not consistently show ATL activation during semantic tasks. Consequently, several influential theories of semantic memory do not ascribe a central role to the ATLs. We conducted a meta-analysis of 164 functional neuroimaging studies of semantic processing to investigate factors that might contribute to the inconsistency in previous results. Four factors influenced the likelihood of finding ATL activation: (1) the use of PET versus fMRI, reflecting the fact that fMRI but not PET is sensitive to distortion artifacts caused by large variations in magnetic susceptibility in the area of the ATL; (2) a field of view (FOV) of more than 15 cm, thereby ensuring whole-brain coverage; (3) the use of a high baseline task to prevent subtraction of otherwise uncontrolled semantic activation; (4) the inclusion of the ATL as an ROI. The type of stimuli or task did not influence the likelihood of ATL activation, consistent with the view that this region underpins an amodal semantic system. Spoken words, written words, and picture stimuli produced overlapping ATL peaks. On average, these were more inferior for picture-based tasks. We suggest that the specific pattern of ATL activation may be influenced by stimulus type due to variations across this region in the degree of connectivity with modality-specific areas in posterior temporal cortex.