Brain aging research relies mostly on cross-sectional studies, which infer true changes from age differences. We present longitudinal measures of five-year change in the regional brain volumes in healthy adults. Average and individual differences in volume changes and the effects of age, sex and hypertension were assessed with latent difference score modeling. The caudate, the cerebellum, the hippocampus and the association cortices shrunk substantially. There was minimal change in the entorhinal and none in the primary visual cortex. Longitudinal measures of shrinkage exceeded cross-sectional estimates. All regions except the inferior parietal lobule showed individual differences in change. Shrinkage of the cerebellum decreased from young to middle adulthood, and increased from middle adulthood to old age. Shrinkage of the hippocampus, the entorhinal cortices, the inferior temporal cortex and the prefrontal white matter increased with age. Moreover, shrinkage in the hippocampus and the cerebellum accelerated with age. In the hippocampus, both linear and quadratic trends in incremental age-related shrinkage were limited to the hypertensive participants. Individual differences in shrinkage correlated across some regions, suggesting common causes. No sex differences in age trends except for the caudate were observed. We found no evidence of neuroprotective effects of larger brain size or educational attainment.