Distinct regions within the ventral visual pathway show neural specialization for nonliving and living stimuli (e.g., tools, houses versus animals, faces). The causes of these category preferences are widely debated. Using functional magnetic resonance imaging, we find that the same regions of the ventral stream that show category preferences for nonliving stimuli and animals in sighted adults show the same category preferences in adults who are blind since birth. Both blind and sighted participants had larger blood oxygen-level dependent (BOLD) responses in the medial fusiform gyrus for nonliving stimuli compared to animal stimuli and differential BOLD responses in lateral occipital cortex for animal stimuli compared to nonliving stimuli. These findings demonstrate that the medial-to-lateral bias by conceptual domain in the ventral visual pathway does not require visual experience in order to develop and suggest the operation of innately determined domain-specific constraints on the organization of object knowledge.