Cognitive impairment, particularly memory disruption, is a major complicating feature of epilepsy. This Review will begin with a focus on the problem of memory impairment in temporal lobe epilepsy (TLE). We present a brief overview of anatomical substrates of memory disorders in TLE, followed by a discussion of how our understanding of these disorders has been improved by studying the outcomes of anterior temporal lobectomy. The clinical efforts made to predict which patients are at greatest risk of experiencing adverse cognitive outcomes following epilepsy surgery are also considered. Finally, we examine the vastly changing view of TLE, including findings demonstrating that anatomical abnormalities extend far outside the temporal lobe, and that cognitive impairments extend beyond memory function. Linkage between these distributed cognitive and anatomical abnormalities point to a new understanding of the anatomical architecture of cognitive impairment in epilepsy. Clarifying the origin of these cognitive and anatomical abnormalities, their progression over time and, most importantly, methods for protecting cognitive and brain health in epilepsy, present a challenge to neurologists.