Sex differences in visuospatial cognition (VSC) with male advantage are frequently reported in the literature. There is evidence for sexual dimorphisms in the human brain, one of which postulates more gray matter (GM) in females and more white matter (WM) in males relative to total intracranial volume. We investigated the neuroanatomy of VSC independent of general intelligence (g) in sex-separated populations, homogenous in age, education, memory performance, a memory- and brain morphology-related gene, and g. VSC and g were assessed with the Wechsler adult intelligence scale. The influence of g on VSC was removed using a hierarchical factor analysis and the Schmid-Leiman solution. Structural high-resolution magnetic resonance images were acquired and analyzed with voxel-based morphometry. As hypothesized, the clusters of positive correlations between local volumes and VSC performance independent of g were found mainly in parietal areas, but also in pre- and postcentral regions, predominantly in the WM in males, whereas in females these correlations were located in parietal and superior temporal areas, predominantly in the GM. Our results suggest that VSC depends more strongly on parietal WM structures in males and on parietal GM structures in females. This sex difference might have to do with the increased axonal and decreased somatodendritic tissue in males relative to females. Whether such sex-specific implementations of the VSC network can be explained genetically as suggested in investigations into the Turner syndrome or as a result of structural neural plasticity upon different experience and usage remains to be shown.