Vision usually provides the most accurate and reliable information about the location of objects in our environment, and thus serves as a reference for recalibrating auditory spatial maps. Recent studies have shown that recalibration does not require accumulated evidence of cross-modal mismatch to be triggered, but occurs as soon as after one single exposure. Here we tested whether instantaneous recalibration and recalibration based on accumulated evidence represent the same underlying learning mechanism or involve distinct neural systems. Participants had to localize two sounds, a low- and a high-frequency tone, which were paired with opposite directions of audiovisual spatial mismatch (leftward vs. rightward). In accordance with the cumulative stimulus history, localization in unimodal auditory trials was shifted in opposite directions for the two sound frequencies. On a trial-by-trial basis, however, frequency-specific recalibration was reduced when preceded by an audiovisual stimulus with a different sound frequency and direction of spatial mismatch. Thus, the immediate past invoked an instantaneous frequency-invariant recalibration, while the cumulative past invoked changes in frequency-specific spatial maps. These findings suggest that distinct recalibration mechanisms operating at different timescales jointly determine sound localization behavior.