Abstract
Binary classification, an act of sorting items into two classes by setting a boundary, is biased by recent history. One common form of such bias is repulsive bias, a tendency to sort an item into the class opposite to its preceding items. Sensory-adaptation and boundary-updating are considered as two contending sources of the repulsive bias, yet no neural support has been provided for either source. Here, we explored human brains of both men and women, using functional magnetic resonance imaging (fMRI), to find such support by relating the brain signals of sensory-adaptation and boundary-updating to human classification behavior. We found that the stimulus-encoding signal in the early visual cortex adapted to previous stimuli, yet its adaptation-related changes were dissociated from current choices. Contrastingly, the boundary-representing signals in the inferior-parietal and superior-temporal cortices shifted to previous stimuli and covaried with current choices. Our exploration points to boundary-updating, rather than sensory-adaptation, as the origin of the repulsive bias in binary classification.
SIGNIFICANCE STATEMENT Many animal and human studies on perceptual decision-making have reported an intriguing history effect called “repulsive bias,” a tendency to classify an item as the opposite class of its previous item. Regarding the origin of repulsive bias, two contending ideas have been proposed: “bias in stimulus representation because of sensory adaptation” versus “bias in class-boundary setting because of belief updating.” By conducting model-based neuroimaging experiments, we verified their predictions about which brain signal should contribute to the trial-to-trial variability in choice behavior. We found that the brain signal of class boundary, but not stimulus representation, contributed to the choice variability associated with repulsive bias. Our study provides the first neural evidence supporting the boundary-based hypothesis of repulsive bias.
Introduction
We commit to a proposition about a specific world state when making a perceptual decision. One basic form of such commitment is binary classification. It is to decide whether an item's magnitude lies on the smaller or larger side of the magnitude distribution across items of interest (Fig. 1A). For example, when uttering “this tree is tall” while walking in a wood, we are implicitly judging the height of that tree to be taller than the typical height of the trees in the wood (Klein, 1980; Bierwisch, 1989), where “typical height” works as the boundary dividing the “short” and “tall” classes. Like this, binary classification is exercised in our daily language use, whenever modifying a subject with relative adjectives (Rips and Turnbull, 1980; Tribushinina, 2011; Solt, 2015; Lassiter and Goodman, 2017), and has been adopted as an essential paradigm for studying perceptual decision-making (Lages and Treisman, 1998; Grinband et al., 2006; Kepecs et al., 2008; Nahum et al., 2010; Lak et al., 2014; Bosch et al., 2020; Hachen et al., 2021).
Two contending hypotheses on the origin of the repulsive bias in binary classification. A, Task structure (left) and statistical knowledge (right) for binary classification. For any given item, its class is determined by its position relative to the class boundary in the distribution of feature magnitudes relevant to a given task (e.g., a tree is classified as “tall” if its height is in the side greater than the typical height of the trees in the wood of interest). This relativity of binary classification makes the “biased sensory encoding” and the “biased knowledge about boundary position” because of previous stimuli, in principle, have equal footings in inducing the repulsive bias. B, Sensory-adaptation hypothesis. It points to the adaptation of a low-level stimulus-encoding signal to past stimuli (arrow 1) as the origin of the repulsive bias (arrow 2). In the case of visual classification tasks, the task-relevant sensory signals in the early visual cortex (blue patch), which are subject to adaptation, have been hypothesized to mediate the repulsive bias. C, Boundary-updating hypothesis. It points to the attractive shift of a classifier's internal class boundary toward previous stimuli (arrow 3) as the origin of the repulsive bias (arrow 4). Such boundary-representing signals are expected to reside not in the early sensory cortex but in the high-tier associative cortices (red patch).
Humans and nonhuman animals show various forms of history bias in binary classification. One frequent form of such history biases is a tendency to classify an item as the class opposite to its preceding items, dubbed repulsive bias (Lages and Treisman, 1998, 2010; Bosch et al., 2020; Hachen et al., 2021). For instance, we tend to classify a tree of intermediate height as “tall” after seeing a short tree. Currently, it remains unclear why and how repulsive bias occurs.
As one most straightforward scenario for repulsive bias, the previous stimuli may repel away our perception of the current stimulus from themselves because the sensory system adapts to earlier stimuli (Gibson and Radner, 1937; Stocker and Simoncelli, 2006; Clifford et al., 2007; Knapen et al., 2010; Pavan et al., 2012; Morgan, 2014; Nakashima and Sugita, 2017; Fig. 1B). According to this “sensory-adaptation” hypothesis, the current tree is biasedly classified as “tall” since the sensory system's adaptation to the previous short tree makes the current tree appear taller than its physical height. However, there is an alternative scenario, which considers the possibility that the internal class boundary adaptively shifts toward recent samples of property magnitude (Treisman and Williams, 1984; Lages and Treisman, 1998, 2010; Dyjas et al., 2012; Raviv et al., 2014; Norton et al., 2017; Hachen et al., 2021; Fig. 1C). According to this “boundary-updating” hypothesis, the current tree is biasedly classified as “tall” since the shift of the class boundary toward the previous short tree makes the current tree be positioned in the taller side of the boundary.
As discussed previously (Hachen et al., 2021), it is hard to assess which hypothesis is more viable based on behavioral data. This difficulty arises because binary classification is a matter of the relativity between the perceived stimulus and the class boundary: the identical bias in classification can be caused either by sensory-adaptation or boundary-updating. However, the two hypotheses involve distinct neural routes through which repulsive bias transpires. The sensory-adaptation hypothesis predicts that the sensory brain signals subject to adaptation, such as those in the early sensory cortex with substantive adaptation to earlier stimuli, contribute to the choice variability. By contrast, the boundary-updating hypothesis predicts that the brain signals of the shifting boundary, such as those in the high-tier cortices involved in the working memory of previous stimuli, contribute to the choice variability.
Here, we tested these two predictions by analyzing functional magnetic resonance imaging (fMRI) data. We found that the stimulus-encoding signal in primary visual cortex (V1) exhibited adaptation, but its bias induced by adaptation was dissociated from current choices. By contrast, the boundary-representing signals in the posterior-superior-temporal gyrus and the inferior-parietal lobe not only shift to previous stimuli but also covaried with current choices. Our findings contribute to the resolution of the competing ideas regarding the source of repulsive bias by providing the first neural evidence supporting the boundary-updating scenario.
Materials and Methods
The data of experiment 1 (Exp1) and experiment 2 (Exp2) were acquired from 19 (nine females, aged 20–30 years) and 18 (nine females, aged 20–30 years) participants, respectively. Among the participants, 17 of them participated in both experiments. The Research Ethics Committee of Seoul National University approved the experimental procedures. All participants gave informed consent and were naive to the purpose of the experiments. High-spatial-resolution images were acquired only from the early visual cortex in Exp1 while the images in Exp2 were acquired from the entire brain with a conventional spatial resolution. The 17 people who provided the data for both experiments participated in three to six behavior-only sessions for training and stimulus calibration, one fMRI session for retinotopy, and two experimental fMRI sessions (one for each experiment). The remaining people also completed the behavioral and retinotopy fMRI sessions with the same protocols but participated in only one of the two experiments.
The data from Exp1 had been used for our previous work (Choe et al., 2014). The data of Exp2 has never been used in any previous publication. In the current paper, we describe some basic procedures of Exp1. For more details on Exp1, please refer to the original work (Choe et al., 2014).
Experimental setup
MRI data were collected using a 3 Tesla Siemens Tim Trio scanner equipped with a 12-channel Head Matrix coil at the Seoul National University Brain Imaging Center. Stimuli were generated using MATLAB (MathWorks) in conjunction with MGL (http://justingardner.net/mgl) on a Macintosh computer. Observers looked through an angled mirror attached to the head coil to view the stimuli displayed via an LCD projector (Canon XEED SX60) onto a back-projection screen at the end of the magnet bore at a viewing distance of 87 cm, yielding a field of view of 22 × 17°.
Behavioral data acquisition
Figure 2 illustrates the experimental procedures. On each trial, the observer initially viewed a small fixation dot (diameter in visual angle, 0.12°; luminance, 321 cd/m2) appearing at the center of a dark (luminance, 38 cd/m2) screen. A slight increase in the size of the fixation dot (from 0.12° to 0.18° in diameter), which was readily detected with foveal vision, forewarned the observer of an upcoming presentation of a test stimulus. The test stimulus was a brief (0.3 s) presentation of a thin (full-width at half-maximum of a Gaussian envelope, 0.17°), white (321 cd/m2), dashed (radial frequency, 32 cycles/360°) ring that counter-phase-flickered at 10 Hz. After each presentation, participants classified the ring size into small or large using a left-hand or right-hand key, respectively, within 1.5 s from stimulus onset. They were instructed to maintain strict fixation on the fixation dot throughout experimental runs. This behavioral task was performed in three different environments: (1) the training sessions, (2) the practice runs of trials inside the MR scanner, and (3) the main scan runs inside the MR scanner, in the following order.
In the training sessions, participants practiced the task intensively over several (three to six) sessions (∼1000 trials per session) in a dim room outside the scanner until they reached an asymptotic level of accuracy. Note that we opted to train observers with the stimuli that were much larger than those for the main experiments (mean radius of 9°) to avoid any unwanted perceptual learning effects at low sensory levels and to train participants to learn the task structure of classification.
In the practice runs of trials inside the MR scanner, participants performed 54 practice trials and then 180 threshold-calibration trials while lying in the magnet bore. On each of the threshold-estimation trials in which consecutive trials were apart from one another by 2.7 s., one of 20 different-sized rings was presented according to a multiple random staircase procedure (four randomly interleaved one-up-two-down staircases, two starting from the easiest stimulus and the other two starting from the hardest one) with trial-to-trial feedback based on the class boundary with the radius of 2.84°. A Weibull function was fit to the psychometric curves obtained from the threshold-calibration trials using a maximum-likelihood procedure. From the fitted Weibull function, the threshold difference in size (Δ in Fig. 2B) associated with a 70.7% correct proportion of responses was estimated. By finding this threshold for each participant, three threshold-level ring sizes were individually tailored as 2.84
In the main scan runs, one of these rings with threshold-level differences was presented in the order defined by an m-sequence (base = 3, power = 3; nine S and L-rings and eight M-rings were presented; all scan runs started with two M-rings; Buracas and Boynton, 2002) to null the autocorrelation between stimuli. Participants were not informed of the existence of medium-ring. Importantly, participants did not receive trial-to-trial feedback. Instead, only their run-averaged percent correct based on the trials of S-ring and L-ring was shown during a break after each run, to prevent trial-to-trial feedback from evoking any unwanted brain responses associated with rewards (Marco-Pallarés et al., 2007; Carlson et al., 2011) or errors (Carter et al., 1998; Holroyd et al., 2004; Cavanagh and Frank, 2014). Consecutive trials were apart from one another by 13.2 s. In the main scan runs of Exp1 and Exp2, observers performed 156 (six runs × 26 trials) and 208 (eight runs × 26 trials) trials in total, respectively.
MRI equipment and acquisition
We acquired three types of MRI images. (1) 3D, T1-weighted, whole-brain images were acquired at the beginning of each functional session: MPRAGE; resolution, 1 × 1 × 1 mm; field of view (FOV), 256 mm; repetition time (TR), 1.9 s; time for inversion, 700 ms; time to echo (TE), 2.36 ms; and flip angle (FA), 9°. (2) 2D, T1-weighted, in-plane images were acquired at the beginning of each functional session. The parameters for the retinotopy-mapping, the V1 mapping, and the whole brain mapping differed slightly as follows (retinotopy, followed by the V1 mapping, and then by the whole brain mapping): MPRAGE; resolution, 1.078 × 1.078 × 2.0 mm, 1.083 × 1.083 × 2.3 mm 1.08 × 1.08 × 3.3 mm; TR, 1.5 s; T1, 700 ms; TE, 2.79 ms; and FA, 9°). (3) 2D, T2*-weighted, functional images were acquired during each functional session: gradient EPI; TR, 2.7, 2.2, 2.2 s; TE, 40 ms; FA, 77°, 73°, 73°; FOV, 208 mm, 207 mm, 208 mm; image matrix, 104 × 104, 90 × 90, 90 × 90; slice thickness, 1.8 mm with 11% gap, 2 mm with 15% slice gap, 3 mm with 10% space gap; slice, 30, 22, 32 oblique transfers slices; bandwidth, 858 Hz/px, 750 Hz/px, 790 Hz/px; and effective voxel size, 2.0 × 2.0 × 1.998 mm, 2.3 × 2.3 × 2.3 mm, 3.25 × 3.25 × 3.3 mm).
Retinotopy-mapping protocol
Standard traveling wave methods (Engel et al., 1994; Sereno et al., 1995) were used to define V1, to estimate each participant's hemodynamic impulse response function (HIRF) of V1, and to estimate V1 voxels' receptive field center and width. High-contrast and flickering (1.33 Hz) dartboard patterns were presented either as 0.89°-thick expanding or contracting rings in two scan runs, as 40°-width clockwise or counterclockwise rotating wedges in four runs or in one run as four stationary, 15°-wide wedges forming two bowties centered on the vertical and horizontal meridians. Each scanning run consisted of nine repetitions of 27-s period of stimulation. The fixation behavior during the scans was assured by monitoring participants' performance on a fixation task, in which they had to detect any reversal in direction of a small dot rotating around the fixation.
Data preprocessing of V1 images in the retinotopy-mapping session and the main session of Exp1
All functional EPI images were motion-corrected using SPM8 (http://www.fil.ion.ucl.ac.uk/spm; Friston et al., 1996; Jenkinson et al., 2002) and then co-registered to the high-resolution reference anatomic volume of the same participant's brain via the high-resolution in-plane image (Nestares and Heeger, 2000). After co-registration, the images of the retinotopy-mapping scan were resliced, but not spatially smoothed, to the spatial dimensions of the main experimental scans. The area V1 was manually defined on the flattened gray matter cortical surface mainly based on the meridian representations, resulting in 825.4±140.7 (mean±SD across observers) voxels. The individual voxels' time series were divided by their means to convert them from arbitrary intensity units to percentage modulations and were linearly detrended and high-pass filtered (Smith et al., 1999) using custom scripts in MATLAB (MathWorks). The cutoff frequency was 0.0185 Hz for the retinotopy-mapping session and 0.0076 Hz for the main session. The first 10 (of 90; a length of a cycle) and 6 (of 156; a length of a trial) frames of each run of the retinotopy-mapping session and main session, respectively, were discarded to minimize the effect of transient magnetic saturation and allow the hemodynamic response to reach a steady state. The “blood-vessel-clamping” voxels, which show unusually high variances of fMRI responses, were discarded (Olman et al., 2007; Shmuel et al., 2007); a voxel was classified as “blood-vessel-clamping” if its variance exceeds 10 times of the median variance value of the entire voxels. As the final step of data preprocessing, we removed a stimulus-nonspecific (untuned) component from the detrended BOLD time series by subtracting the across-eccentricity-bin average from the individual bins' time series at each time frame t, which resulted in the tuned responses (
Data preprocessing of whole-brain images in the main session of Exp2
The whole-brain images of the participants in Exp2 were normalized to the MNI template in the following steps: motion correction, co-registration to whole-brain anatomic images via the in-plane images (Nestares and Heeger, 2000), spike elimination, slice timing correction, resampling to 3 × 3 × 3-mm voxel size with the SPM DARTEL Toolbox (Ashburner, 2007). Spatial smoothing was not applied to avoid the blurring of the patterns of activity. All the procedures were implemented using SPM8 and SPM12 (https://www.fil.ion.ucl.ac.uk/spm-statistical-parametric-mapping/; Friston et al., 1996; Jenkinson et al., 2002), except for spike elimination, for which we used the AFNI toolbox (Cox, 1996). The first 6 frames of each functional scan, which correspond to the first trial of each run, were discarded to allow the hemodynamic responses to reach a steady state. Then, the normalized BOLD time series at each voxel, each run, and each brain underwent linear detrending, high-pass filtering (0.0076-Hz cutoff frequency with a Butterworth filter), conversion into percent-change signals, and correction for non-neural nuisance signals, which was done by regressing out the mean BOLD activity of CSF.
The anatomic masks of CSF, white matter, and gray matter were defined by generating the probability tissue maps for individual participants from T1-weighted images, by smoothing those maps to the normalized MNI space using SPM12, and then by averaging them across participants. Finally, the masks were defined as respective groups of voxels whose probabilities exceed 0.5.
Unfortunately, in a few of the sessions, functional images did not cover the entire brain. Especially, the lost part was much larger in one participant's session than the others including the orbitofrontal cortex and posterior cerebellum. Thus, not to lose too many of voxels for analysis because of this single session, we relaxed the criterion of voxel selection a bit by including the voxels that were shared by >16 brains in the normalized MNI space. As a result, some voxels in the temporal pole, ventral orbitofrontal, and posterior cerebellum were excluded from data analysis.
Estimation of the eccentricities in retinotopic space for V1 voxels
For each V1 voxel in Exp1, its eccentricity (e, as shown in Fig. 3E,H) was defined by fitting a one-dimensional Gaussian function simultaneously to the time-series of fMRI responses to the expanding and contracting ring stimuli in the retinotopy session, which were also used for the definition of V1. The essence of this procedure is as follows (additional details can be found in the original paper; Choe et al., 2014).
First, the time series of fMRI were extracted only from a relevant group of voxels with SNR > 3 in both of the ring scan runs. Second, an eccentricity-tuning curve (gain over eccentricity, in other words) of a single voxel,
Third, the collective responses of visual neurons within that voxel with a particular
Fourth, the predicted time-series of fMRI responses of that voxel,
Fifth, the eccentricity e and the other model parameters (σ, β, b) were found by fitting
Extraction of the size-encoding signal from V1 voxels
The three different weighting profiles, each representing the contributions of the individual eccentricity bins assessed by the three different schemes (the uniform, the discriminability, and the log-likelihood ratio schemes), were defined as follows. The uniform scheme (Fig. 4B, blue) assigned three discrete values to the eccentricity bins depending on which flanking side of the M-ring (
The discriminability scheme (Fig. 4B, red) defined the weights in proportion to the differential responses of given eccentricity bins to the L (
The log-likelihood ratio scheme (Fig. 4B, yellow) defined the weights by taking the differences between the log-likelihoods of obtaining a given response if the stimulus were the L-ring,
A Bayesian model of boundary-updating (BMBU)
The generative model
The generative model is the observers' causal account for noisy sensory measurements, where the true ring size, S, causes a noisy sensory measurement on a current trial,
The primary purpose of BMBU is to build a generative Bayesian model which allows us to estimate the trial-to-trial latent states of the class boundary variable that are likely to be used by human observers whose class boundary is continually attracted to previous stimuli as posited by the boundary-updating hypothesis on “repulsive bias.” In doing so, we intended to build a parsimonious model with minimal free parameters as long as the model implements the strategy essential to the boundary-updating hypothesis. For this reason, we had to introduce several arbitrary assumptions in building BMBU. For example, although we assumed that memory precision decays exponentially, other forms of decay function are also possible, such as hyperbolic, power, and logarithmic ones. We also assumed that the noisy sensory measurement on a current trial,
Stimulus inference (s)
A Bayesian estimate of the value of S on a current trial,
The posterior
Class boundary inference (b)
The Bayesian observer infers the value of class boundary on a current trial,
Deduction of decision variable (v), decision (d), and decision uncertainty (u)
On each trial, the Bayesian observer makes a binary decision
Then, if
Fitting the parameters of BMBU
For each human participant, the parameters of the generative model,
For each participant, estimation was conducted in the following steps. First, we found local minima of parameters using a MATLAB function, fminsearchbnd.m, with the iterative evaluation number set to 50. We repeated this step by choosing 1000 different initial parameter sets, that were randomly sampled within uniform prior bounds, and acquired 1000 candidate sets of parameter estimates. Second, from these candidate sets of parameters, we selected the top 20 in terms of goodness-of-fit (sum of log-likelihoods) and searched the minima using each of those 20 sets as initial parameters by increasing the iterative evaluation number to 100,000 and setting tolerances of function and parameters to 10−7 for reliable estimation. Finally, using the parameters fitted via the second step, we repeated the second step one more time. Then, we selected the parameter set that showed the largest sum of likelihoods as the final parameter estimates. We discarded (1) the first trial of each run and (2) the trials in which RTs were too short (less than 0.3 s) for parameter estimation for any further analyses because (1) the first trial of each run does not have its previous trial, which is necessary for investigating the repulsive bias, and (2) the response made during the stimulus is shown (0–0.3 s) can be considered too hasty to reflect a normal cognitive decision-making process.
A constant-boundary model
The constant-boundary model has two parameters, bias of class boundary
Estimation of the latent states of the variables of BMBU
Fitting the model parameters separately for each human participant (
Recovery of the true states of the model variables
To ascertain the validity of our procedure of estimating the latent variables of BMBU described above, we checked how accurately it recovers the true states of the variables. This recovery test was conducted in the following procedure.
First, we created 256 different sets of parameter values by taking the possible combinations of the four different values of each of the four model parameters, where the four different values corresponded to the 20th, 40th, 60th, and 80th percentiles of the parameter values fitted to the observers' choices. Second, we acquired the synthetic choices and the true model variables b, s, v, and u by plugging one parameter set into BMBU and simulating it on the actual stimulus sequence presented to the observers. Third, we fitted the parameters of BMBU to the synthetic choices in the same procedure conducted for fitting BMBU to the observed choices. Fourth, we simulated a set of the recovered states of the model variables using the fitted model parameters. Fifth, we calculated the R2 between the true and the recovered variables to assess how reliably our model fitting procedure can recover the true states of the model variables. Finally, we repeated the above procedure for all the remained parameter sets and used the R2 averaged across the 256 parameter sets as the performance measure of the recovery test.
The multiple logistic regression model for capturing the repulsive bias
To capture the repulsive bias in human classification, we logistically regressed the current choice onto stimuli and choices using the following regression model to obtain regression coefficients
To capture the repulsive bias of the Bayesian observers, the Bayesian observers' choices were also regressed with the logistic regression model by substituting
The average marginal effect analysis
Average marginal effect (AME) was calculated by using the R-package “margins” (Leeper et al., 2018). AME quantifies the average marginal effect between an ordinal dependent variable (i.e., binary choice) and an independent variable of a multiple logistic (or probit) regression model (Williams and Jorgensen, 2023). To calculate the AMEs of any given variable on the current choice (
We always included
To test whether the AME of X decreased after controlling
Searching for the multivoxel patterns of activity representing the latent variables of BMBU
We assumed that (1) activity patterns of neural population for representing the latent variables are different between participants, but (2) locations and (3) timings of the activity patterns overlap across participants. Therefore, to identify the brain signals of the latent variables of BMBU in fMRI responses, the support vector regression (SVR) decoding was conducted for each human participant within specific spatial and temporal windows.
As for the spatial window, we implemented a searchlight technique (Kahnt et al., 2011b; Haynes, 2015). A searchlight has a radius of 9 mm (= 3 voxels; Soon et al., 2008) and thus can contain 123 voxels at most. Of the 123 voxels, we excluded the voxels located in CSF or white matter because they reflect non-neural signals. Thus, the effective number of voxels in a searchlight used for the analysis can vary searchlight by searchlight.
As for the temporal windows, we implemented the time-resolved decoding technique in which a target variable is decoded from the BOLD responses at each of the within-trial time points (Fig. 6B). We used the first four time points (out of six in total) because the BOLD responses associated with the action of button press, the last process of the sensory-to-motor decision-making stream, is maximized at the fourth time point (the result is not shown here). In sum, SVR is trained for each participant, each time point, and each searchlight.
The sets of regressions that BMBU requires the brain signals of its latent variables to satisfy
Before training SVR, the BOLD responses in a searchlight and a target latent variable were z-scored across trials. Then, the z-scored variable was decoded for each searchlight using the cross-validation method of leave-one-run-out (eightfold cross-validation). As a result, for each searchlight and at each time point, we acquired a set of decoded latent variables in all trials. In other words, on each time point, we acquired the 4-dimensional map of the decoded variable (i.e., three spatial dimensions and 1 trial dimension). The 3D spatial dimensions of the decoded variables were smoothed with a 5 mm FWHM Gaussian kernel on each trial.
After this subject-wise decoding analysis, we conducted the across-subject analysis to test whether the decoded variables are significantly informative. To do so, for each searchlight locus and each time point, we regressed the smoothed decoded variable onto the regression conditions of the target variable by using a generalized linear mixed effect regression model (GLMM) with a random effect of subjects. The number of regression conditions was 14, 14, and 17 for
SVR was conducted using LIBSVM (https://www.csie.ntu.edu.tw/∼cjlin/libsvm/) with a linear kernel and constant regularization parameter of 1 (Soon et al., 2008; Kahnt et al., 2011b). The brain imaging results were visualized using Connectome Workbench (Marcus et al., 2011) and xjview.
The regression-model test for verifying the brain signals of b ( t ) s ( t ) v ( t )
To identify the brain signals of
The 14 regressions for the brain signal of
The 17 regressions for the brain signal of
Bayesian network analysis
To investigate whether the relationship between decoded b,
We opted to apply this Bayesian network analysis to the three-node networks instead of the six-node network consisting of all the six brain signals identified by the searchlight analysis because the number of possible six-node networks (
Statistics
We used the searchlight technique to look for brain signals related to the latent variables of the BMBU. To make the searchlight analysis statistically powerful by reducing the noise effect in the BOLD signals, we applied a generalized linear mixed effect model (GLMM) with the random effect of observers to calculate the association between the true and the decoded model variables. We applied the mixed effect model only to the searchlight analysis (Fig. 6; Table 1). For the other regression analyses, we conducted the analysis for each individual, respectively, because the mixed effect model was too computationally demanding to be applied to all other analyses. For instance, applying GLMM to the model simulation depicted in Figure 5C requires
Results
Experimental paradigm
Over consecutive trials, participants sorted ring sizes into two classes, small and large, under moderate time pressure (Fig. 2A). To ensure decision-makings with uncertainty, we presented three rings (small, medium, and large) differing by a threshold size (Δ), which was tailored for individuals (Fig. 2B; see Materials and Methods). The ring sizes were presented in m-sequence to rule out any correlation between consecutive stimulus sizes (Buracas and Boynton, 2002). We provided participants with feedback after each scan run by summarizing their performance with the proportion of correct trials.
Binary classification task on ring size. A, Within-trial procedure. With the eyes fixed, human participants were prewarned (2.2 s), with the increase of the fixation dot, to get ready for the upcoming trial after a long intertrial interval (9.5 s), briefly viewed the ring stimulus (0.3 s), and judged its size as large or small in respect to the medium size ring within a limited window of time (1.5 s). B, Ring stimuli with threshold-level differences in size. On each trial, a participant viewed one of the three rings, small (S), medium (M), large (L), the size contrast (Δ) of which was optimized to ensure threshold-level classification performance on a participant-to-participant basis in a separate calibration run inside the MR scanner, right before the main session of fMRI scan runs. The order of ring sizes over trials was constrained with an m-sequence to preclude the temporal correlation among stimuli. Here, the luminance of the rings is inverted here for an illustrative purpose.
To verify the sensory-adaptation hypothesis, we conducted experiment 1, where 19 participants performed the classification task while BOLD measurements with a high spatial resolution were acquired only from their early visual cortices. To verify the boundary-updating hypothesis, we conducted experiment 2, where 18 participants performed the same task while their whole brains were imaged. The data of experiment 1 had been used in our published work (Choe et al., 2014).
Repulsive bias in experiment 1
The participants in experiment 1 displayed a substantive amount of repulsive bias. As anticipated, the proportion of large choices (PL) increased as the ring size on the current trial (
Influences of previous and current stimuli on classification behavior and V1 activity in experiment 1. A–C, Repulsive bias in psychometric curves (A, B) and regression analysis (C). The psychometric curves, where the fractions of large choices are plotted against the current stimulus, are shown separately for the six possible combinations defined by the previous stimulus and choice (A). As the summary of the effects of the previous stimulus on the current choice, the differences in the fractions of large choices between the previous stimuli were L-ring and S-ring (
To ensure this repulsive effect of the previous stimulus on the current choice, we logistically regressed each participant's current choice (
Sensory adaptation in V1
As a first step toward the verification of the sensory-adaptation hypothesis, we defined the size-encoding signal in V1. As our group showed previously (Choe et al., 2014); the eccentricity-tuned BOLD responses in V1 (Fig. 3D) readily resolved the threshold-level differences in ring size, as anticipated by the retinotopic organization of the V1 architecture (Fig. 3E). Thus, the subtraction of the BOLD responses at the voxels preferring S-ring to L-ring from those at the voxels preferring L-ring to S-ring (Fig. 3F) was significantly greater when
Next, having defined the size-encoding signal in V1, which will be referred to as “
In sum, the V1 population activity reliably encoded the ring size and exhibited sensory adaptation.
The variability of V 1
Next, we verified the critical prediction of the sensory-adaptation hypothesis on repulsive bias. Below, we will define what this crucial prediction is and how we empirically examine that prediction.
Above, we confirmed that the ring size, not only on the current trial (
Origin of the covariation between the stimulus-encoding signal of V1 and the current choice. A, The causal structure of the variables implied by the sensory-adaptation hypothesis. The stimulus-encoding signal of V1 (
We can test the critical implication of the sensory-adaptation hypothesis by comparing the average marginal effect (AME; Williams and Jorgensen, 2023) of
In doing so, the trial-to-trial measures of
The AME of
The analyses above were conducted for
Furthermore, the same pattern of AMEs was observed when we used two alternative readout schemes for extracting
Repulsive bias in experiment 2
Having failed to find the evidence supporting the sensory-adaptation hypothesis in experiment 1, we conducted experiment 2 to search the whole brain for the signal representing the class boundary and to test whether that signal relates to the previous stimuli and the current choice in a manner consistent with the boundary-updating hypothesis. As mentioned earlier (see above, Experimental paradigm), the experimental procedure in experiment 2 was the same as in experiment 1, except for the fMRI protocol.
The behavioral performance in experiment 2 closely matched that in experiment 1 (Fig. 3A–C) in many aspects. The PL difference induced by the previous stimulus (
Repulsive bias in experiment 2 and a Bayesian model of boundary updating (BMBU). A–C, Repulsive bias in psychometric curves (A, B) and regression analysis (C). The formats were identical to those in the corresponding figure panels for experiment 1 (Fig. 3A–C), except that the ex post model simulation results (green lines and symbols) are added. In the bottom insets of B, the observed (x-axis) and simulated (y-axis) average differences in the fractions of large choices between the trials in which the previous stimulus was L-ring and those in which it was S-ring are plotted against one another, where the red diagonal demarcates the identity line. In the bottom insets of C, the observed (x-axis) and simulated (y-axis) regression coefficients for the previous stimulus (
Bayesian model of boundary-updating (BMBU)
As we identified
We developed BMBU by formalizing the binary classification task in terms of Bayesian decision theory (Knill and Richards, 1996), a powerful framework for modeling human decision-making behavior under uncertainty. Binary classification is to judge whether the “ring size on the current trial t (
The generative model
On trial t,
Inferring the current stimulus size
On trial t, the Bayesian classifier infers
Inferring the class boundary
On trial t, the Bayesian classifier infers the class boundary (
Making a decision with the inferred current stimulus size and the inferred class boundary
Having estimated
In sum, BMBU models a human decision-maker as the Bayesian classifier who, over consecutive trials, continuously infers the class boundary (b) and the current stimulus size (s), deduces the decision variable (v) from s and b, and makes a decision (d) with a varying degree of uncertainty (u). As shown below, BMBU well predicts human participants' choices and reproduces their repulsive bias.
The prediction and simulation of human choices and repulsive bias by BMBU
We assessed BMBU's accountability for human behavior in the binary classification task in two aspects, comparing its (1) predictability of the choices and (2) reproducibility of repulsive bias to those of the control model which does not update the class boundary (“constant-boundary model”; see Materials and Methods).
We assessed the predictability of BMBU and the constant-boundary model by fitting them to human choices using the maximum likelihood rule (see Materials and Methods). BMBU excels over the constant-boundary model in goodness-of-fit. The average AIC difference across participants is
After equipping the models with their best-fit parameters, we assessed their reproducibility by making them simulate the decisions over the same sequence of ring sizes presented to the human participants (see Materials and Methods). From this simulation, we can also vividly appreciate how BMBU updates its class boundary (
In sum, BMBU's inferences of the class boundary based on past stimuli accounted for a substantive fraction of the choice variability of human classifiers and successfully captured their repulsive bias.
Brain signals of the class boundary and the other latent variables
In the previous section, we demonstrated that BMBU accounted well for the variability of human choices and successfully reproduced the observed repulsive bias. However, such correspondences between the humans' and the models' choices do not necessarily warrant the validity of our procedure of estimating the latent states of the model variables (b, s, and v), which is crucial in testing the boundary-updating hypothesis. To validate our estimation procedure, we tested whether it could accurately recover the true states of the model variables based on the synthetic datasets simulated with 256 ground-truth model parameter sets (see the Materials and Methods). The recovered states of the model variables well matched the corresponding true states (R2 =
Then, with the trial-to-trial states of the simulated latent variables, we identified the brain signals of those variables with the following rationale and procedure. On any given trial t, a classifier makes a decision in the manner constrained by the causal structure of BMBU (Fig. 5G). This causal structure implies two important points to be considered when identifying the neural representations of b, s and v. First, for any cortical activity, its significant correlation with the variable of interest does not necessarily imply that it represents that variable per se but is open to the possibility that it may represent the other variables that are associated with the variable of interest. Second, if any given cortical activity represents the variable of interest, that activity must not violate any of its relationships with the other variables that are implied by the causal structure (Table 1; see Materials and Methods).
We incorporated these two points in our search of the brain signals of b, s and v, as follows. Initially, we identified the candidate brain signals of b, s, and v by localizing the patterns of activities that closely reflect the trial-to-trial states of b, s, and v. For localization, we used the support vector regressor decoding with the searchlight technique (Kahnt et al., 2011a; Hebart et al., 2016), which is highly effective in detecting the local patterns of population fMRI responses associated with the latent variables of computational models (Kriegeskorte et al., 2006). Next, we put those candidate brain signals to a strong test of whether their trial-to-trial states satisfy the causal relationships with the other variables. Specifically, we converted those causal relationships into the empirically testable sets of regression models (Table 1), respectively for b (14 regressions), s (14 regressions), and v (17 regressions) and checked whether all the regressors' coefficients derived from the brain signals were consistent with the regression models (see Materials and Methods). In what follows, we will describe how the regression tests for the brain signal of b (
According to the causal relationship of b with the latent variables,
As a result, the brain signals that survived the exhaustive regression tests clustered in six separate regions (Fig. 6; Table 2). The signal of b appeared in three separate regions at different time points relative to stimulus onset, a region in the left inferior parietal lobe at 1.1s (IPLb1) and two regions in the left posterior superior temporal gyrus at 3.3 and 5.5 s (pSTGb3, pSTGb5). The signal of s appeared in the left dorsolateral prefrontal cortex at 3.3 s (DLPFCs3) and in the right cerebellum at 5.5 s (Cerebs5). The signal of v appeared in the left anterior superior temporal gyrus at 5.5 s (aSTGv5). To ascertain the robustness of the neural representations of the latent variables in these six areas, we repeated the searchlight decoding analysis using a different searchlight size (87 voxels, which is smaller than the original one, 123 voxels). Despite the change in searchlight size, we could detect the clusters that survived all regression tests around the six regions (Table 2).
Brain signals of the latent variables of BMBU. A, Loci of the brain signals. The brain regions where BOLD activity patterns satisfied all the regressions implied by the causal structure of the variables in BMBU are overlaid on the inflated cortex and the axial view of the cerebellum of the template brain. B, Within-trial time courses of the satisfied regressions in number. The within-trial task phases are displayed (top panel) to help appreciate when the brain signals become pronounced, with the hemodynamic delay (4–5 s) in BOLD (bottom three panels). C, The coefficients and the 95% CIs of the generalized linear mixed effect model (GLMM) of the decoded variable averaged across the searchlights of each ROI on the time points on which each ROI was detected. The regression index indicates the index specified in Table 1. B, C, The colors of the symbols and lines correspond to those of the brain regions shown in A. Asterisks indicate the statistical significance (*P,0:05, **P,0:01, ***P,0:001). The 95% CIs of the mean across participants are indicated by the vertical error bars.
Specification of the brain signals of the latent variables of BMBU
Lastly, we investigated whether the probable causal structures between the brain signals of b,
The outcomes of BIC evaluation were consistent with BMBU. First, out of the 162 possible causal graphs, the smallest (best) BIC value was found for “pSTGb5→aSTGv5 ←Cerebs5” (Fig. 7). Second, We found that any graph with the causal arrows between
The probable causal structures between the brain signals of the latent variables in BMBU. For each row, the value in the left indicates the relative BIC scores of the causal structures in reference to the most probable one at the top.
The variability of the class-boundary brain signals associated with previous stimuli contributes to the variability of choice
Finally, with the brain signals that represent the class boundary (IPLb1, pSTGb3, and pSTGb5) in our hands, we verified the boundary-updating hypothesis with the rationale and analysis identical to those for the verification of the sensory-adaptation hypothesis.
We stress that the respective associations of the brain signal of b with the previous stimulus (
Origin of the covariation between the current choice and the brain signals of the latent variables in BMBU. A, The causal structure of the variables implied by the boundary-updating hypothesis. The brain signal of the decision variable (
As anticipated, the AME of pSTGb5 on
The same patterns were also observed for IPLb1 and pSTGb3 (Fig. 8C,D). Especially, the AMEs of pSTGb3 and IPLb1 on
Having found the evidence supporting the boundary-updating hypothesis in the brain signals of b, we also conducted the same AME analysis on the signals of s and v below. Given the causal structure of b,
On the contrary, the contribution of v to
On a separate note, the six loci of the brain signals of b,s, and d were defined by applying the conservative criterion that any given cluster satisfying all the regression tests (Table 1) should be the same or larger than 12. We note that there was a focal region in the right-hemisphere medial visual cortex that survived the regression tests for
To examine the neural loci of the inferred stimulus further, we checked the possibility that VCs3 might carry the signal via which the current stimulus (
Furthermore, to test the sensory-adaptation hypothesis, we examined whether VCs3 carries the stimulus signal via which the previous stimulus (
In sum, the results suggest that neural signals of b and s transferred previous and current stimuli to current decisions, respectively, and the neural signal of v transferred both previous and current stimuli to current decisions as BMBU implies, which is consistent with the boundary-updating hypothesis.
Discussion
Here, we explored the two possible origins of repulsive bias, sensory-adaptation versus boundary-updating, in binary classification tasks. Although
Dissociation between sensory-adaptation in V1 and repulsive bias
What makes sensory-adaptation a viable origin of repulsive bias is not its mere presence but its contribution to repulsive bias. The presence of sensory-adaptation in V1 has been firmly established (Clifford et al., 2007; Kohn, 2007; Solomon and Kohn, 2014; Weber et al., 2019) and is the necessary premise for the sensory-adaptation hypothesis to work. What matters is whether the trial-to-trial variability of V1 because of such adaptation exerts its influence on the current choice. Such an influence was not observed in our data.
From a general perspective, our findings demonstrate a dissociation between the impact of previous decision-making episodes on the sensory-cortical activity and the contribution of that sensory-cortical activity to decision-making behavior. In this regard, V1 in the current work acts like the binocular-disparity-encoding signal of V2 neurons in a recent single-cell study on monkeys (Lueckmann et al., 2018), where, despite the impact of the history on V2 activity, the variability of V2 activity associated with the history failed to contribute to the history effects on decision-making behavior. Similarly, our findings also echo the failure of the sensory-adaptation of V1 in influencing the visual orientation estimation in an fMRI study on human participants (Sheehan and Serences, 2022). There, while sensory-adaptation was evident along the hierarchy of visual areas including V1, V2, V3, V4, and interaparietal sulcus (IPS), the history effect of the previous stimulus on the current estimation behavior was opposite to that expected from sensory-adaptation, which suggests that a downstream mechanism compensates for sensory-adaptation. Such a mechanism was also called for when the single-cell-recording work on monkeys tried to explain their intriguing adaptation effects found along the visual processing hierarchy (McLelland et al., 2009). For instance, static visual stimuli engendered prolonged, on the order of tens of seconds, adaptation in the lateral geniculate nucleus but the adaptation in V1 was paradoxically short-lived, on the order of 100 ms.
The representations of the class boundary in IPL and pSTG
To account for the repulsive bias in binary classification, previous studies proposed descriptive models based on the common idea that the internal boundary continuously shifts toward the previous stimuli (Treisman and Williams, 1984; Lages and Treisman, 1998, 2010; Dyjas et al., 2012; Raviv et al., 2014; Norton et al., 2017; Hachen et al., 2021). However, the neural concomitant of class-boundary updating has rarely been demonstrated.
To our best knowledge, this issue has so far been addressed by one fMRI work (White et al., 2012); which reported the class-boundary signal in the left inferior temporal pole. However, several aspects of this work make it hard to consider the reported brain signal to represent the class boundary inducing repulsive bias. First, they experimentally manipulated the class boundary in a block-by-block manner. Thus, it is unclear whether the reportedly boundary-representing signal was updated by previous stimuli trial-to-trial, which is required to induce repulsive bias. Second, the class boundary size correlated with the average stimulus size block-by-block in their experiments. Because of this confounding factor, one cannot rule out the possibility that the reported brain signal reflects the sensory signal associated with the average stimulus size induced by the current stimulus. By contrast, the brain signal of the class boundary in our work is free from these methodological limitations, because it is updated on a trial-to-trial basis and survived the rigorous set of tests, including those addressing possible confounding variables (Table 1). In this sense, the current work can be considered the first demonstration of the brain signals representing the class boundary that is dynamically updated in such a way that it can account for repulsive bias.
We emphasize that we developed BMBU to infer the trial-to-trial latent states of the class boundary used by human observers for the purpose of verifying the boundary-updating hypothesis on repulsive bias. In this sense, BMBU should not be taken as a unified account of the history effects reported by previous studies. For example, BMBU does not account for the influence of previous decisions on subsequent decision-making, another significant contributor to the history effects (Akaishi et al., 2014; Urai and Donner, 2022). To be sure, we are open to the possibility that there might be a unified mechanism relating the previous, and current, as well, stimuli and previous decisions to the current decision in an integrative manner. To incorporate the previous decisions into such a unified mechanism, it is important to distinguish the influence of the previous choice from that of the previous motor response, which we could not do in the current work because choices and motor responses covaried. In this regard, the weak but significant negative regression coefficient of the previous decision in experiment 1 (Fig. 3C) could have been reflective of the influence of the previous motor response, as previously suggested (Zhang and Alais, 2020).
The representations of inferred stimuli in DLPFC and cerebellum
The brain signals of the inferred ring size (
It is notable that BMBU treats
The residence of the inferred, i.e., subjective or perceived, stimulus representation in DLPFC and cerebellum, instead of the visual cortex, seems consistent with previous reports. DLPFC and cerebellum have been well known for their critical involvement in visual awareness (Gao et al., 1996; Rees et al., 2002; Dehaene and Changeux, 2011; Lau and Rosenthal, 2011; Baumann et al., 2015). By contrast, the visual cortex is likely to be involved more in a faithful representation of physical input than its subjective representation (Renart and Machens, 2014), consistent with the previous findings of our group (Lee et al., 2007; Choe et al., 2014).
The representation of the decision variable in aSTG
Whereas previous single-cell studies have reported that the decision variable is represented in the prefrontal cortex (Kim and Shadlen, 1999; Hanks et al., 2015; Hebart et al., 2016), we identified the brain signal of v only in aSTG but not in PFC. This inconsistency may reflect the poor spatial and temporal resolution of fMRI measurements. For example, if any given signal of interest is encoded in the sequential or dynamical activity patterns across a neural population, as recently demonstrated theoretically (Orhan and Ma, 2019) or empirically (Wutz et al., 2018), such signals cannot be decoded from fMRI responses. Alternatively, the inconsistency may have been a result of the previous studies not taking into account the history effect in defining the decision variable, in contrast to our study which did, given the prevalence of diverse history effects in various decision-making tasks (Fründ et al., 2014; Lak et al., 2020). In this scenario, the brain signal of the inferred stimulus in DLPFC in our study hints at the possibility that the previously reported decision variable signal in PFC could have reflected the inferred stimulus, which is closely associated with the decision variable when the decision boundary is assumed to be fixed (Gold and Shadlen, 2007). Understanding the functional role of DLPFC in perceptual decision-making seems to require further future studies, especially those in which the history effects are considered in decision variable definition while neural responses are probed at a sufficiently high spatiotemporal resolution.
Footnotes
This research was supported by the Seoul National University Research Grant 339-20220013 and by the Brain Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and Information and Communications Technology Grant No. NRF-2021R1F1A1052020.
The authors declare no competing financial interests.
- Correspondence should be addressed to Sang-Hun Lee at visionsl{at}snu.ac.kr
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
