Brain anatomy is characterized by dramatic growth from the end of the second trimester through the neonatal stage. The characterization of normal axonal growth of the white matter tracts has not been well-documented to date and could provide important clues to understanding the extensive inhomogeneity of white matter injuries in cerebral palsy (CP) patients. However, anatomical studies of human brain development during this period are surprisingly scarce and histology-based atlases have become available only recently. Diffusion tensor magnetic resonance imaging (DTMRI) can reveal detailed anatomy of white matter. We acquired diffusion tensor images (DTI) of postmortem fetal brain samples and in vivo neonates and children. Neural structures were annotated in two-dimensional (2D) slices, segmented, measured, and reconstructed three-dimensionally (3D). The growth status of various white matter tracts was evaluated on cross-sections at 19-20 gestational weeks, and compared with 0-month-old neonates and 5- to 6-year-old children. Limbic, commissural, association, and projection white matter tracts and gray matter structures were illustrated in 3D and quantitatively characterized to assess their dynamic changes. The overall pattern of the time courses for the development of different white matter is that limbic fibers develop first and association fibers last and commissural and projection fibers are forming from anterior to posterior part of the brain. The resultant DTMRI-based 3D human brain data will be a valuable resource for human brain developmental study and will provide reference standards for diagnostic radiology of premature newborns.