Changes in neocortex size were a prominent feature of mammalian brain evolution, but the implications for cortical structure, and consequently for the functional significance of such changes in overall cortical size, are poorly understood. A basic question is whether functionally differentiated cortical areas evolved independently of one another (adaptive specialization) or were allometrically constrained to co-vary tightly with the size of the whole. Here, I provide comparative evidence for adaptive specialization of cortical structure. First, the sizes of individual areas differ significantly between taxa after controlling for overall cortical size. Second, an analysis of separate visual cortical areas reveals that these exhibit statistically correlated evolution, independent of variation in nonvisual areas. Third, visual cortex size exhibits correlated evolution with peripheral visual adaptations (eye morphology and optic nerve size) and with photic niche. Thus, the evolution of mammalian cortical structure was closely associated with specialization for different sensory niches.