Reversal learning may conceptually be dissected into acquiring stimulus-reinforcement associations and subsequently altering behavior by switching to new associations as stimulus-reinforcement contingencies reverse (i.e., affective switching). Previous imaging studies have found regions of the ventrolateral and orbitofrontal cortex (OFC) to be involved in both subprocesses. However, these studies did not contain an affectively neutral baseline, which precluded adequate assessment of main effects of reward, punishment, and affective switching. We aimed to determine the neural substrate of these main effects, and of common and dissociable regions for reward and punishment. Furthermore, we aimed to discriminate between stimulus-punishment association and affective switching, i.e., to assess affective switching proper. To this end, we implemented a reversal learning task with an affectively neutral baseline condition that matched the experimental task in visual complexity and motor demands. Interestingly, we found dorsolateral prefrontal cortex (DLPFC) and anterior PFC to be engaged in affective switching, a finding that has not been reported before to our knowledge. Enhanced responses in these areas may represent their involvement in cognitive set shifting per se unrelated to the affective context in a reversal learning design. In addition, OFC, insular and medial prefrontal cortex regions were involved in affective switching. Left medial and lateral OFC were shown to be common areas for feedback processing, whereas left ventral striatum and left lateral OFC were specifically activated by reward and punishment, respectively. These results extend our understanding of the neural substrate of reversal learning in humans.