The visual recognition of letters dissociates from the recognition of numbers at both the behavioral and neural level. In this article, using fMRI, we investigate whether the visual recognition of numbers dissociates from letters, thereby establishing a double dissociation. In Experiment 1, participants viewed strings of consonants and Arabic numerals. We found that letters activated the left midfusiform and inferior temporal gyri more than numbers, replicating previous studies, whereas numbers activated a right lateral occipital area more than letters at the group level. Because the distinction between letters and numbers is culturally defined and relatively arbitrary, this double dissociation provides some of the strongest evidence to date that a neural dissociation can emerge as a result of experience. We then investigated a potential source of the observed neural dissociation. Specifically, we tested the hypothesis that lateralization of visual number recognition depends on lateralization of higher-order numerical processing. In Experiment 2, the same participants performed addition, subtraction, and counting on arrays of nonsymbolic stimuli varying in numerosity, which produced neural activity in and around the intraparietal sulcus, a region associated with higher-order numerical processing. We found that individual differences in the lateralization of number activity in visual cortex could be explained by individual differences in the lateralization of numerical processing in parietal cortex, suggesting a functional relationship between the two regions. Together, these results demonstrate a neural double dissociation between letter and number recognition and suggest that higher-level numerical processing in parietal cortex may influence the neural organization of number processing in visual cortex.