Functional neuroimaging studies reveal differences in neural correlates of the retrieval of emotional and nonemotional memories. In the present experiment, encoding of emotionally neutral pictures in association with positively, neutrally or negatively valenced background contexts led to differential modulation of neural activity elicited in a subsequent recognition memory test for these pictures. Recognition of stimuli previously studied in emotional compared to neutral contexts elicited enhanced activity in structures previously implicated in episodic memory, including the parahippocampal cortex, hippocampus and prefrontal cortex. In addition, there was engagement of structures linked more specifically to emotional processing, including the amygdala, orbitofrontal cortex and anterior cingulate cortex. These emotion-related effects displayed both valence-independent and valence-specific components. We discuss the findings in terms of current models of emotional memory retrieval.