Head-direction (HD) cells in subcortical areas of the mammalian brain are tuned to a particular head direction in space; a population of such neurons forms a neural compass that may be relevant for spatial navigation. The development of neural circuits constituting the head-direction system is poorly understood. Inspired by electrophysiological experiments about the role of recurrent synaptic connections, we investigate a learning rule that teaches neurons to amplify feed-forward inputs. We simulate random head movements of a rat, during which neurons receive both visual and vestibular (head-velocity) inputs. Remarkably, as recurrent connections learn to amplify exclusively the visual inputs, a neural network emerges that performs spatio-temporal integration. That is, during head movements in darkness, neurons resemble HD cells by maintaining a fixed tuning to head direction. The proposed learning rule exhibits similarities with known forms of anti-Hebbian synaptic plasticity. We conclude that selective amplification could serve as a general principle for the synaptic development of multimodal feedback circuits in the brain.