Neural Correlates of the Divergence of Instrumental Probability Distributions

Task and procedure.

A simple instrumental task was used, in which four action alternatives (i.e., button presses) yielded various food rewards with different probabilities. Specifically, on any given trial, two available actions could differ with respect to the probability with which they produced their respective rewards, with respect to the subjective utility of those rewards and/or with respect to the integrated action value. Thus, by manipulating both probabilities and utilities, we were able to largely decorrelate the different components of the decision problem. To ensure sufficient variance in experienced probabilities and utilities, each subject participated in three consecutive sessions (during a single appearance by the subject in the lab), with the same four actions but with a novel set of food outcomes and outcome probabilities being used in each session. Throughout the task, available actions were indicated by corresponding rectangles on the computer screen, together with images of the food outcomes potentially produced by those actions (Fig. 1A). At the start of the experiment, participants were informed that they would have to remain in the laboratory for 30 min after completing the task, during which they would be allowed to consume any earned treats.

The probabilities with which actions produced their outcomes were generated so as to minimize correlations between our three decision variables (i.e., between outcome probabilities, outcome values, and action values), and thus varied slightly across subjects. Nonetheless, across sessions and action alternatives, probabilities of 0, 0.2, 0.5, 0.7, and 1.0 were used for each subject. In addition, each subject had two probabilities chosen from the set [0.3 0.6 0.8 and 0.9]. The probabilities drawn from this set differed depending on the decorrelation constraints imposed by subjective outcome utilities. A minimum of two and maximum of four distinct probabilities were used in each session.

The subjective values of 36 potential food rewards (listed in Table 1), represented by photographic images, were assessed using evaluative ratings of their pleasantness (on a scale from 0 to 9), as well as a Becker–DeGroot–Marschak (BDM) auction procedure that has been shown to elicit an individual's willingness to pay for a consumer good (Becker et al., 1964). The pleasantness ratings were used to set the utility of food stimuli in subsequent phases of the experiment. The BDM auction was used to obtain convergent evidence for these ratings, providing a measure of inter-rater reliability. Specifically, in the BDM auction, participants were endowed with $5 with which to bid on the various food items. They were instructed that, on each trial, they would have to indicate an amount of money from $0 to $5 that they were willing to pay for the food item displayed on that trial and that, at the end of the experiment, the computer would randomly select one trial from all presented in that phase, as well as randomly draw an amount between $0 and $5. Participants were further told that if the bid that they had indicated on the randomly drawn trial was less than the amount generated by the computer, they would not receive the food item, but would get to keep the $5, and that otherwise they would have to pay the amount generated by the computer and would get to consume the food item.

In each session, participants first went through a “contingency learning” phase, in which each action was sampled 10 times with associated food rewards occurring with respective probabilities. Only one action was available on each trial in this phase, to ensure complete sampling, and outcome occurrences were generated using predetermined sequences such that if, for example, the probability of an outcome was 0.2, that outcome was presented on exactly 2 of the 10 trials. Furthermore, participants were instructed that they would not actually earn any of the food rewards produced by the actions in this phase, but that it was simply meant to expose them to action–outcome relationships. They then proceeded to a “choice phase” in which, on each of 48 trials, they chose between two of the four action alternatives (Fig. 1A). They were instructed that, at the end of the experiment, three trials would be drawn from this phase, and that they would be allowed to consume any rewards earned on those trials upon completion of the task. Following the choice phase, participants provided judgments of the existence and strength of each action–outcome relationship. Active scanning was only performed during the choice phases.

Computational learning model.

We implemented a model-based RL learner, which uses experience with state transitions to update a matrix, T(s,a,s′), of state transition probabilities, where each element of T(s,a,s′) holds the current estimate of the probability of transitioning from state s to s′ given action a. In our task, as illustrated in Figure 1A, on each trial participants were presented with a choice screen displaying two available actions together with the food outcomes potentially produced by those two actions. Thus, each initial state was defined by the particular available actions and their potential outcomes. The two available actions were drawn from a total of four; consequently, in the choice phase of each session, there were six distinct initial states, repeatedly encountered across 48 trials. The state transitions were initialized to the preprogrammed distributions from the contingency learning phase. In each step, leaving state s and arriving in state s′ having taken action a, the FORWARD learner computes a state prediction error[3] (SPE): δ_SPE = 1 − T(s,a,s′), and updates the probability T(s,a,s′) of the observed transition via: T (s,a,s′) = T(s,a,s′) + ηδ_SPE where η is a free parameter controlling the learning rate. Estimated transition probabilities are used together with the rewards at the end states, r(s′) (the magnitude of which were based on pleasantness ratings and taken as given, since potential rewards were displayed together with the actions) to compute state-action values, Q(s,a) as the expectation over the value of the successor state. This is done by defining the state-action values at each level in terms of reward anticipated at the next level: Q(s,a) = Σ_s'T(s,a,s′) *r(s′).

The model additionally assumes that participants select actions stochastically using probabilities generated by a softmax distribution, such that P(s, a) = exp(τ×Q(s, a))∑b=1nexp(τ×Q(s, b)), where the free “inverse temperature” parameter τ controls the degree to which choices are biased toward the highest valued action. To account for the difference in salience between food pictures and the “no outcome” display (which simply consisted of a white line), we used separate learning rate parameters, η and η′, on rewarded and nonrewarded trials, respectively. Free parameters were fit to behavioral data by minimizing the negative log-likelihood, −Σlog(P(s,a)), of obtained choices for each individual.

Information theoretic variables.

We computed the JS divergence of the outcome probability distributions for the two actions available on a given trial. A finite and symmetrized version of the Kullback–Leibler divergence, JS divergence specifies the distance between probability distributions M and N as follows: It is worth noting that, while we have used JS divergence here to quantify the degree to which alternative actions differ with respect to contingent transitions in environmental states, it is not the only computational variable that captures this conceptual point. For example, mutual information (between actions and outcomes) is a highly related information theoretic measure that would be identical to JS divergence for our current purposes, as would be the χ² divergence. A particularly compelling aspect of JS divergence is its remarkable generality: it applies to nominal and numerical, discrete and continuous random variables, and it intuitively generalizes to an arbitrary number of probability distributions. The applicability to multiple distributions (Lin, 1991) is especially important as it eliminates the need for a complex, and presumably computationally costly, process of comparing multiple available actions in a pairwise fashion.

Another important information theoretic variable that can be extracted from the state-transition matrix, and that has been previously shown to profoundly affect decision making (Paulus et al., 2002; Feinstein et al., 2006), is the uncertainty, or entropy, of outcome states. Whereas JS divergence reflects the distance between outcome probability distributions, entropy reflects the degree of uncertainty in an outcome, which is greatest when the probability distribution over outcomes is uniform (i.e., all outcomes are equally likely) and smallest when the probability of a particular outcome is 1 or 0. We computed the Shannon entropy of the outcome variable X conditional on the action variable Y, defined as H(X|Y) = ∑x∈X, y∈Yp(x,y)logp(y)p(x,y), both for the case where the (chosen) action is known (p(y) = [1,0]) and for the case where the two available actions are equally likely (p(y)=[0.5,0.5]). To illustrate the relationship between outcome probabilities and information theoretic variables, a representative set of probability distributions, with corresponding levels of JS divergence and entropy, are shown in Figure 2.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

A representative set of cases used in the experiment. Each case consists of two probability distributions, one for each available action, across three potential outcome states (O1, O2, and O3) including the no-food outcome. A1 and A2 indicate the two actions available on a given trial, drawn from a total of four possible action alternatives. X indicates the levels of JS divergence (blue) and entropy (red) for each case.

Imaging procedure and analysis.

A 3 T scanner (MAGNETOM Trio; Siemens) was used to acquire structural T1-weighted images and T2*-weighted echoplanar images (repetition time = 2.65 s; echo time = 30 ms; flip angle = 90°; 45 transverse slices; matrix = 64 × 64; field of view = 192 mm; thickness = 3 mm; slice gap = 0 mm) with blood oxygenation level-dependent (BOLD) contrast. To recover signal loss from dropout in the medial orbitofrontal cortex (O'Doherty et al., 2002), each horizontal section was acquired at 30° to the anterior commissure–posterior commissure axis. Image processing and statistical analyses were performed using SPM8 (http://www.fil.ion.ucl.ac.uk/spm). The first four volumes of images were discarded to avoid T1 equilibrium effects. All remaining volumes were corrected for differences in the time of slice acquisition, realigned to the first volume, spatially normalized to the Montreal Neurological Institute (MNI) echoplanar imaging template, and spatially smoothed with a Gaussian kernel (8 mm, full-width at half-maximum). We used a high-pass filter with a cutoff of 128 s.

For each subject, we constructed an fMRI design matrix, merged across the three sessions, with two regressors modeling the distinct time periods of each trial. The first, choice-period regressor, modeled a BOLD response from the onset of each trial until the chosen action was performed and a second stick function modeled the onset of the feedback screen. For the choice-period regressor, we entered as parametric modulators, in order, the expected value of the chosen action, the sum of and the absolute difference between the expected values of the two available actions, the utility of the outcome potentially produced by the chosen action, and the sum and absolute difference in utility of the two potential outcomes depicted on the screen. Absolute differences, rather than that between chosen and unchosen, were used to minimize regressor redundancies. Finally, for this trial period, we entered as modulators the entropy conditional on the chosen action, the entropy conditional on both available actions, and the JS divergence of outcome probability distributions of available actions. For the outcome regressor, we entered as modulators, in order, the RPE, the SPE, and the utility of the received outcome. Orthogonalization was applied according to order such that each parametric modulator was orthogonalized to all preceding modulators associated with the same onset regressor. To rule out motor-execution components, the response time on each trial was added as a regressor of no interest, as were two regressors indicating the three sessions and six regressors accounting for the residual effects of head motion. All regressors were convolved with a canonical hemodynamic response function. Group-level random-effects statistics were generated by entering contrasts of parameter estimates for the different modulators into a between-subjects analysis.

We specifically looked for neural effects in areas previously shown to be involved in the implementation of our modeled decision variables. First, in a recent neuroimaging study, Gläscher et al. (2010) assessed neural correlates of SPEs, finding effects in the lateral prefrontal cortex (LPFC) and intraparietal sulcus (IPS). We predicted that activity in these areas would likewise correlate with SPEs in the current study. We also predicted that the dorsolateral prefrontal cortex (DLPFC) would encode the entropy of outcome probability distributions: activity in this area has been shown to scale with the amount of uncertainty associated with a decision, to predict risk aversion (Weber and Huettel, 2008) and, when disrupted by transcranial magnetic stimulation, to increase selection of risky option (Knoch et al., 2006). With respect to our motivational variables, studies of goal-directed performance, which emphasize the casual relationships between actions and outcomes (Tanaka et al., 2008; Liljeholm et al., 2011) or higher-order relational structures (Hampton et al., 2006), have implicated the ventromedial prefrontal cortex (VMPFC) in the encoding of action values. In contrast, activity in the medial orbitofrontal cortex (mOFC), the insula, and ventral striatum (VS), has been shown to correlate with the utility, or value, of a stimulus (Hare et al., 2008; Abler et al., 2009; Schmidt et al., 2012). The anterior insula has also been implicated in the affective evaluation of food pictures (Pelchat et al., 2004; Wang et al., 2004) and in anticipation and experience of appetitive tastes (O'Doherty et al., 2001, 2002). Finally, The VS has been shown, by myriad studies, to encode a RPE (O'Doherty et al., 2003, 2004).

Small volume corrections (SVCs) were performed on a priori regions of interest (ROIs), using a 10 mm sphere with center coordinates obtained by averaging across relevant studies (coordinates are listed in Table 2). All other effects were reported at p < 0.05, using cluster size thresholding (CST) to adjust for multiple comparisons (Forman et al., 1995). AlphaSim, a Monte Carlo simulation (AFNI) was used to determine cluster size and significance. Using an individual voxel probability threshold of p = 0.001 indicated that using a minimum cluster size of 134 MNI transformed voxels resulted in an overall significance of p < 0.05. To eliminate nonindependence bias for plots of parameter estimates, a leave-one-subject-out (Esterman et al., 2010) approach was used, in which 22 general linear models (GLMs) were run with one subject left out in each, and with each GLM defining the voxel cluster for the left out subject. Using rfxplot (Gläscher, 2009), mean β-weights were extracted from spheres (10 mm) centered on the LOSO peaks (identified within ROIs for SVCs) and were averaged across subjects to plot overall effect sizes.