In auditory cortex, activation and subsequent adaptation is strongest for regions responding best to a stimulated tone frequency and less for regions responding best to other frequencies. Previous attempts to characterize the spread of neural adaptation in humans investigated the auditory cortex N1 component of the event-related potentials. Importantly, however, more recent studies in animals show that neural response properties are not independent of the stimulation context. To link these findings in animals to human scalp potentials, we investigated whether contextual factors of the acoustic stimulation, namely, spectral variance, affect the spread of neural adaptation. Electroencephalograms were recorded while human participants listened to random tone sequences varying in spectral variance (narrow vs. wide). Spread of adaptation was investigated by modeling single-trial neural adaptation and subsequent recovery based on the spectro-temporal stimulation history. Frequency-specific neural responses were largest on the N1 component, and the modeled neural adaptation indices were strongly predictive of trial-by-trial amplitude variations. Yet the spread of adaption varied depending on the spectral variance in the stimulation, such that adaptation spread was broadened for tone sequences with wide spectral variance. Thus the present findings reveal context-dependent auditory cortex adaptation and point toward a flexibly adjusting auditory system that changes its response properties with the spectral requirements of the acoustic environment.