Neurophysiological data suggest that the integration of prior information and incoming sensory evidence represents the neural basis of the decision-making process. Here, we aimed to identify the brain structures involved in the integration of prior information about the average magnitude of a stimulus set and current sensory evidence. Specifically, we investigated whether prior average information already biases vibrotactile decision making during stimulus perception and maintenance before the comparison process. For this purpose, we used a vibrotactile delayed discrimination task and fMRI. At the behavioral level, participants showed the time-order effect. This psychophysical phenomenon has been shown to result from the influence of prior information on the perception of and the memory for currently presented stimuli. Similarly, the fMRI signal reflected the integration of prior information about the average vibration frequency and the currently presented vibration frequency. During stimulus encoding, the fMRI signal in primary and secondary somatosensory (S2) cortex, thalamus, and ventral premotor cortex mirrored an integration process. During stimulus maintenance, only a region in the intraparietal sulcus showed this modulation by prior average information. Importantly, the fMRI signal in S2 and intraparietal sulcus correlated with individual differences in the degree to which participants integrated prior average information. This strongly suggests that these two regions play a pivotal role in the integration process. Taken together, these results support the notion that the integration of current sensory and prior average information is a major feature of how the human brain perceives, remembers, and judges magnitude stimuli.