Hippocampal sclerosis is the most common abnormality associated with medial temporal lobe epilepsy (MTLE). Converging evidence supports that hippocampal sclerosis progresses with time. However, it is unclear whether extrahippocampal atrophy in patients with MTLE, similarly to hippocampal sclerosis, is an unremitting progressive process. In this article, we investigate the relationship between duration of epilepsy and gray matter concentration reduction in patients with MTLE within and outside the hippocampus. We employed a voxel-based morphometry study of MRI of the entire brain of 36 patients with drug refractory MTLE and 49 neurologically healthy age-matched controls. We performed a voxel-based parametric and nonparametric investigation of the association between gray matter concentration, age and duration of epilepsy. We complemented the investigation by extracting the gray matter concentration of regions of interest (ROIs) within the limbic system, and we investigated the association between the gray matter concentration on the ROIs and duration of epilepsy. Patients with MTLE exhibited gray matter concentration reduction that is negatively correlated with the duration of epilepsy within the ipsilateral hippocampus, temporal lobes as well as extratemporal limbic structures that are closely connected with the hippocampus. In conclusion, longer duration of refractory epilepsy was associated with a more intense hippocampal and extrahippocampal atrophy in patients with MTLE. The mechanism of progressive neuronal damage in MTLE may be related to active seizure activity within a limbic network, and early seizure control may prevent further brain atrophy in patients with refractory MTLE.