The adult brain is much more resilient and adaptable than previously believed, and adaptive structural plasticity involves growth and shrinkage of dendritic trees, turnover of synapses, and limited amounts of neurogenesis in the forebrain, especially the dentate gyrus of the hippocampal formation. Stress and sex hormones help to mediate adaptive structural plasticity, which has been extensively investigated in the hippocampus and to a lesser extent in the prefrontal cortex and amygdala, all brain regions that are involved in cognitive and emotional functions. Stress and sex hormones exert their effects on brain structural remodeling through both classical genomic as well as non-genomic mechanisms, and they do so in collaboration with neurotransmitters and other intra- and extracellular mediators. This review will illustrate the actions of estrogen on synapse formation in the hippocampus and the process of stress-induced remodeling of dendrites and synapses in the hippocampus, amygdala, and prefrontal cortex. The influence of early developmental epigenetic events, such as early life stress and brain sexual differentiation, is noted along with the interactions between sex hormones and the effects of stress on the brain. Because hormones influence brain structure and function and because hormone secretion is governed by the brain, applied molecular neuroscience techniques can begin to reveal the role of hormones in brain-related disorders and the treatment of these diseases. A better understanding of hormone-brain interactions should promote more flexible approaches to the treatment of psychiatric disorders, as well as their prevention through both behavioral and pharmaceutical interventions.