Binaural cues for spatial localization of complex high-frequency sounds are interaural level and time differences (ILDs and ITDs). We previously showed that cells in the lateral superior olive (LSO) are sensitive to ITDs in the envelope of sinusoidally amplitude-modulated (AM) signals up to a modulation frequency of only approximately 800 Hz. To understand the limitations in this ITD-sensitivity, we here compare responses to monaural modulation in LSO and its input pathways, derived from cochlear nucleus and medial nucleus of the trapezoid body. These pathways have marked functional and morphological specializations, suggestive of adaptations for timing. Afferent cell populations were identified on the basis of electrophysiological signatures, and for each population, average firing rate and synchronization to AM tones were compared with auditory-nerve fibers and LSO cells. Except for an increase in modulation gain in some subpopulations, synchronization of LSO afferents was very similar to that in auditory nerve fibers in its dependency on sound pressure level (SPL), modulation depth, and modulation frequency. Distributions of cutoff frequencies of modulation transfer functions were largely coextensive with the distribution in auditory nerve. Group delays, measured from the phase of the response modulation as a function of modulation frequency, showed an orderly dependence on characteristic frequency and cell type and little dependence on SPL. Similar responses were obtained to a modulated broadband carrier. Compared with their afferents, LSO cells synchronized to monaurally modulated stimuli with a higher gain but often over a narrower range of modulation frequencies. Considering the scatter in afferent and LSO cell populations, ipsi- and contralateral responses were well matched in cutoff frequency and magnitude of delays. In contrast to their afferents, LSO cells show a decrease in average firing rate at high modulation frequencies. We conclude that the restricted modulation frequency range over which LSO cells show ITD-sensitivity does not result from loss of envelope information along the afferent pathway but is due to convergence or postsynaptic effects at the level of the LSO. The faithful transmission of envelope phase-locking in LSO afferents is consistent with their physiological and morphological adaptations, but these adaptations are not commensurate with the rather small effects of physiological ITDs reported previously, especially when compared with effects of ILDs. We suggest that these adaptations have evolved to allow a comparison of instantaneous amplitude fluctuations at the two ears rather than to extract interaural timing information per se.