fMRI and EEG Predictors of Dynamic Decision Parameters during Human Reinforcement Learning

Choice proportions and RT distributions are captured by RL-DDM. A, Behavioral RT distributions across the group are shown for each reward condition (red, smoothed with kernel density estimation), together with posterior predictive simulations from the RL-DDM (blue). Distributions to the right correspond to choices of the high-valued option and those to the left represent choices of the low-valued option. The relative area under each distribution defines the choice proportions (i.e., greater area to the right than left indicates higher proportion choices of the more rewarding action). Accuracy is worse, and tails of the distribution are longer, with lower reward probability. B, Model fit to behavior can be more precisely viewed using a quantile-probability plot, showing choices and quantiles of RT distributions. Choice probability is plotted along the x-axis separately for choices involving low, medium, and high differences in values (tertiles of value differences assessed by the RL model). Values on the x-axis >0.5 indicate proportion of choices of the high-valued option, and those <0.5 indicate choice proportions of the low-valued option, each with their corresponding RT quantiles on the y-axis. For example, when value differences were medium, subjects chose the high-valued option ∼80% of the time, and the first RT quantile of that choice occurred at ∼475 ms. Empirical behavioral choices/RT quantiles are marked as X and simulated RTs from the posterior predictive of the RL-DDM as ellipses (capturing uncertainty). Quantiles are computed for each subject separately and then averaged to yield group quantiles. Ellipse widths represent SD of the posterior predictive distribution from the model and indicate estimation uncertainty.

Graphical model showing hierarchical estimation of RL-DDM with trial-wise neural regressors. Round nodes represent continuous random variables, and double- bordered nodes represent deterministic variables, defined in terms of other variables. Shaded nodes represent observed data, including trial-wise behavioral data (accuracy, RT) and neural measures (fMRI and EEG). Open nodes represent unobserved latent parameters. Overall subject-wise parameters are estimated from individuals drawn from a group distribution with inferred mean μ and variance σ. Trial-wise variations of decision threshold a and drift rate v (residuals from the subject-wise values) are determined by neural measures and latent RL value difference/conflict. Plates denote that multiple random variables share the same parents and children (e.g., each subject-wise threshold parameter a_S shares the same parents that define the group distribution). The outer plate is over subjects S while the inner plate is over trials T. Inferred relationships of trial-wise regressors were estimated as fixed effect across the group, as regularization and to prevent parameter explosion. sv = SD of drift rate across trials; t = time for encoding and response execution (“nondecision time”).

Combining fMRI and EEG to estimate DM parameters. A, Trial-to-trial peak estimates of the BOLD signal from pre-SMA and STN ROIs (depicted on sagittal slices) along with trial-to-trial theta power estimates from mid-frontal EEG, were entered in as regressors to DDM parameters. The upper response boundary represents choices of the high-valued option, whereas the lower response boundary represents choices of the suboptimal option. Decision threshold reflects the distance between the boundaries, and the drift rate (speed of evidence accumulation) is proportional to the value difference between options. B, Posterior distributions on model parameter estimates, showing estimated regression coefficients whereby drift rate was found to be proportional to value differences and decision threshold proportional to various markers of neural activity and their interaction with decision conflict. Peak values of each distribution represent the best estimates of each parameter, and the width of the distribution represents its uncertainty. Drift rate was significantly related to the (dynamically varying across trials) value difference between each option (posterior distribution shifted far to the right of zero). Decision threshold was significantly related to STN BOLD; the interaction between pre-SMA BOLD and conflict; and the three-way interaction between mid-frontal theta power, STN BOLD, and conflict.