Memory is systematically affected by temporal lobe epilepsy. Since surgery is a promising alternative to pharmacological treatment the questions which memory system is affected and what the long-term prognosis of memory is are more relevant than ever. We address these issues by cross-sectional and longitudinal analysis of memory performance in large series of patients with temporal lobe epilepsy (TLE). The findings indicate that episodic memory rather than semantic memory is impaired in TLE, in particular in TLE with mesial temporal pathology. With the exception that mesial functions appear increasingly affected by chronic non-mesial TLE, memory decline in TLE is not different from that observed in healthy control subjects. However, since patients perform poorer than controls at any age, normal senescence brings patients to mnesic disability at a younger age. Semantic memory seems unaffected by this process but early cortical lesions appear to interfere with knowledge acquisition. Longitudinal data come to a different conclusion regarding the contribution of epilepsy/seizures to memory decline. Conservative treatment is associated with significant decline in figural memory and 37% of the patients experience some memory decline in the long run. Surgery partly anticipates the decline observed with conservative treatment, but losses are most marked after left temporal lobe surgery. After surgery, quite stable memory or even late recovery from surgery is indicated. Leaving aside the surgical intervention, the data provide evidence that the longitudinal memory outcome in TLE is determined by seizure control, seizure severity, mental reserve capacities, and the retest interval. Thus early and efficient seizure control and the prevention of any cerebral damage from the beginning of epilepsy are demanded.