Episodic recognition can be based on recollection of contextual details, on a sense of recent encounter, or some combination of the two. According to several cognitive models, selectively attending to these distinct aspects of memory may require different retrieval orientations and result in different neural responses depending upon whether or not retrieval is successful. Using event-related fMRI, we examined retrieval orientation by having subjects discriminate between two test words in one of two manners. During source recollection, they selected the member of the pair previously associated with a particular encoding task. In contrast, recency judgment required selection of the most recently encountered item of the pair, regardless of how it had been encoded. Furthermore, successful and unsuccessful trials within each retrieval task were contrasted to determine whether retrieval success effects occurred in overlapping or dissimilar neural populations compared to those associated with each retrieval orientation. The results revealed distinct lateral prefrontal and parietal activations that distinguished attempted source recollection from judgments of relative recency; these orientation effects were largely independent of retrieval success. In contrast, medial temporal lobe structures (hippocampus and parahippocampal gyrus) were differentially more active during successful recollection of encoding context, showing similar reduced responses during failed source recollection and judgments of recency. These results indicate that different memory orientations recruit distinct prefrontal and parietal networks and that the recovery of episodic context is associated with the hippocampus and surrounding medial temporal cortices.