There is a considerable body of neuropsychological and neuroimaging evidence supporting the distinction between the brain correlates of noun and verb processing. It is however still not clear whether the observed differences are imputable to grammatical or semantic factors. Beyond the basic difference that verbs typically refer to actions and nouns typically refer to objects, other semantic distinctions might play a role as organizing principles within and across word classes. One possible candidate is the notion of manipulation and manipulability, which may modulate the word class dissociation. We used functional magnetic resonance imaging (fMRI) to study the impact of semantic reference and word class on brain activity during a picture naming task. Participants named pictures of objects and actions that did or did not involve manipulation. We observed extensive differences in activation associated with the manipulation dimension. In the case of manipulable items, for both nouns and verbs, there were significant activations within a fronto-parietal system subserving hand action representation. However, we found no significant effect of word class when all verbs were compared to all nouns. These results highlight the impact of the biologically crucial sensorimotor dimension of manipulability on the pattern of brain activity associated to picture naming.