In this paper we examine a central issue in cognitive neuroscience: are there separate conceptual representations associated with different input modalities (e.g., Paivio, 1971, 1986; Warrington & Shallice, 1984) or do inputs from different modalities converge on to the same set of representations (e.g., Caramazza, Hillis, Rapp, & Romani, 1990; Lambon Ralph, Graham, Patterson, & Hodges, 1999; Rapp, Hillis, & Caramazza, 1993)? We present an analysis of four PET studies (three semantic categorisation tasks and one lexical decision task), two of which employ words as stimuli and two of which employ pictures. Using conjunction analyses, we found robust semantic activation, common to both input modalities in anterior and medial aspects of the left fusiform gyrus, left parahippocampal and perirhinal cortices, and left inferior frontal gyrus (BA 47). There were modality-specific activations in both temporal poles (words) and occipitotemporal cortices (pictures). We propose that the temporal poles are involved in processing both words and pictures, but their engagement might be primarily determined by the level of specificity at which an object is processed. Activation in posterior temporal regions associated with picture processing most likely reflects intermediate, pre-semantic stages of visual processing. Our data are most consistent with a hierarchically structured, unitary system of semantic representations for both verbal and visual modalities, subserved by anterior regions of the inferior temporal cortex.