Apart from being increasingly implicated in higher motor control, Broca's area is considered to play an important role in action understanding by coding the motor goal of an action. Moreover, recent findings suggest that parts of Broca's area may be able to code action content in a more abstract fashion, independent of modality, specific movement parameters or effector used. We performed functional magnetic resonance imaging to examine whether in humans processing object-directed hand actions presented either visually as video clips or verbally as spoken sentences relies on the same neural substrates. To control for action specificity, we included videos and sentences depicting inanimate motion events. In order to induce conceptual processing, we asked participants to make judgements about the acceptability of the stimuli. Results show that processing object-directed hand actions presented both visually and verbally leads to common activation of areas in parietal and frontal regions, most prominently in the pars opercularis of Broca's region. We conclude that the pars opercularis of Broca's area is endowed with polymodal capabilities, allowing the processing of higher-level conceptual aspects of action understanding.