Interaural time difference (ITD) is a major cue to sound localization in humans and animals. For a given subject and position in space, ITD depends on frequency. This variation is analyzed here using a head related transfer functions (HRTFs) database collected from the literature and comprising human HRTFs from 130 subjects and animal HRTFs from six specimens of different species. For humans, the ITD is found to vary with frequency in a way that shows consistent differences with respect to a spherical head model. Maximal ITD values were found to be about 800 μs in low frequencies and 600 μs in high frequencies. The ITD variation with frequency (up to 200 μs for some positions) occurs within the frequency range where ITD is used to judge the lateral position of a sound source. In addition, ITD varies substantially within the bandwidth of a single auditory filter, leading to systematic differences between envelope and fine-structure ITDs. Because the frequency-dependent pattern of ITD does not display spherical symmetries, it potentially provides cues to elevation and resolves front/back confusion. The fact that the relation between position and ITDs strongly depends on the sound's spectrum in turn suggests that humans and animals make use of this relationship for the localization of sounds.