To determine how vocally expressed emotion is processed in the brain, we measured neural activity in healthy volunteers listening to fearful, sad, happy and neutral non-verbal vocalizations. Enhanced responses to emotional vocalizations were seen in the caudate nucleus, as well as anterior insular, temporal and prefrontal cortices. The right amygdala exhibited decreased responses to fearful vocalizations as well as fear-specific inhibitory interactions with left anterior insula. A region of the pons, implicated in acoustic startle responses also showed fear-specific interactions with the amygdala. The data demonstrate: firstly, that processing of vocal emotion involves a bilaterally distributed network of brain regions; and secondly, that processing of fear-related auditory stimuli involves context-specific interactions between the amygdala and other cortical and brainstem regions implicated in fear processing.