Sex steroids exert potent influences on modulating neural development and neural circuit formation in both developing and adult sex steroid-sensitive neuroendocrine brain. During development, estrogen or aromatizable androgen can act as a neurotropic factor on neural tissues, stimulating axonal and dendritic growth and synapse formation. The development of sexual dimorphic synaptic organization may reflect sex steroid-modulating synaptogenesis in the hypothalamus and limbic system during the perinatal period. The onset of puberty also may be due, at least in part, to stimulation of synapse formation by estrogen in the hypothalamus. In adulthood, estrogen has a facilitatory effect on synapse formation in neural structures such as septum, hypothalamus and midbrain with or without brain lesions, and androgen plays a significant role in regulating synaptic remodeling in the androgen-sensitive spinal motoneuron pools. Thus, sex steroids seems to be critical from the developmental period to adulthood for organizing and reorganizing the neuronal circuitry driving neuroendocrine and behavioral functions.