Although numerous neuroimaging studies have examined the functional neuroanatomy supporting episodic memory for verbal material, there have been few investigations of non-verbal episodic encoding and retrieval. We used fMRI to directly compare prefrontal activation elicited by verbal and non-verbal material during encoding and during retrieval. Regardless of the mnemonic operation (encoding/retrieval), inferior prefrontal activation lateralized based on material type. Verbal encoding and retrieval resulted in greater left inferior prefrontal activation, whereas non-verbal encoding and retrieval resulted in greater right inferior prefrontal activation. The similarity between inferior prefrontal activity during encoding and during retrieval indicates that these mnemonic operations depend on shared processes mediated by inferior prefrontal regions.