Inferences about how the complex sensory and motor systems of the human brain evolved are based on the results of comparative studies of brain organization across a range of mammalian species, and evidence from the endocasts of fossil skulls of key extinct species. The endocasts of the skulls of early mammals indicate that they had small brains with little neocortex. Evidence from comparative studies of cortical organization from small-brained mammals of the six major branches of mammalian evolution supports the conclusion that the small neocortex of early mammals was divided into roughly 20-25 cortical areas, including primary and secondary sensory fields. In early primates, vision was the dominant sense, and cortical areas associated with vision in temporal and occipital cortex underwent a significant expansion. Comparative studies indicate that early primates had 10 or more visual areas, and somatosensory areas with expanded representations of the forepaw. Posterior parietal cortex was also expanded, with a caudal half dominated by visual inputs, and a rostral half dominated by somatosensory inputs with outputs to an array of seven or more motor and visuomotor areas of the frontal lobe. Somatosensory areas and posterior parietal cortex became further differentiated in early anthropoid primates. As larger brains evolved in early apes and in our hominin ancestors, the number of cortical areas increased to reach an estimated 200 or so in present day humans, and hemispheric specializations emerged. The large human brain grew primarily by increasing neuron number rather than increasing average neuron size.