The function of the anterior-most portion of the temporal lobes, the temporal pole, is not well understood. Anatomists have long considered it part of an extended limbic system based on its location posterior to the orbital frontal cortex and lateral to the amygdala, along with its tight connectivity to limbic and paralimbic regions. Here we review the literature in both non-human primates and humans to assess the temporal pole's putative role in social and emotional processing. Reviewed findings indicate that it has some role in both social and emotional processes, including face recognition and theory of mind, that goes beyond semantic memory. We propose that the temporal pole binds complex, highly processed perceptual inputs to visceral emotional responses. Because perceptual inputs remain segregated into dorsal (auditory), medial (olfactory) and ventral (visual) streams, the integration of emotion with perception is channel specific.