Separate Valuation Subsystems for Delay and Effort Decision Costs

Participants.

Eighteen young, healthy, heterosexual men (mean age: 23.1 ± 1.8 years), participated in the study. All the participants were males because men are generally more responsive to visual sexual stimuli compared with women, both in terms of behavioral arousal and brain response (Hamann et al., 2004; Sescousse et al., 2010), and to avoid potential influence of the menstrual cycle known to have an effect on reward processing in women (Dreher et al., 2007). Two participants initially enrolled were excluded from data analysis because of inappropriate calibration of the task for the preferences of these subjects. The study was approved by the Paris Pitié-Salpêtrière Hospital ethics committee, and written informed consent was obtained from all subjects. All participants were right-handed, as assessed by the Edinburgh Handedness Questionnaire (mean score: 0.83 ± 0.17) and were screened by a psychiatrist for history of psychiatric or neurological illness, as assessed by the Mini-International Neuropsychiatric Interview and the Hospital Anxiety and Depression Scale (HAD) (mean HAD score: 3.32 ± 1.46). None of the subjects showed impulsivity patterns, as assessed by the Barratt impulsiveness scale (Patton et al., 1995) (mean score: 53.78 ± 6.09). Subjects' sexual orientation was assessed using the French analysis of sexual behavior questionnaire (Spira et al., 1993) and sexual arousability was measured with the Sexual Arousability Inventory (SAI) (Hoon and Chambless, 1998) (mean score SAI: 92.76 ± 12.34), which ensured that subjects showed a “standard” sexual arousability. Subjects were asked to avoid any sexual relationship for 24 h before the scanning session.

Delay/effort discounting task.

Subjects were first asked to read the instructions of the task. To ensure that subjects understood the task and to familiarize them with the hand grip, they were trained on a practice version outside the scanner room with fuzzy cues and clear outcome pictures that were different from those subsequently used in the scanner (to avoid any habituation effect). During training and inside the scanner, subjects' maximal strength was measured using a magnetic resonance imaging (MRI)-compatible handgrip (designed by Eric Featherstone, Wellcome Trust Centre for Neuroimaging, London, UK). They were then taken inside the scanner and were invited to find an optimal body position, while lying down with the power grip in their right hand, the arm resting over the belly. The power grip was made up of two molded plastic cylinders, which compressed an air tube when squeezed. The tube led to the control room, where it was connected to a transducer capable of converting air pressure into voltage. Thus, compression of the two cylinders by an isometric handgrip resulted in the generation of a differential voltage signal, linearly proportional to the force exerted. The signal was fed to the stimuli presentation computer (PC) via a signal conditioner (CED 1401, Cambridge Electronic Design). The task was programmed on a PC using the matlab toolbox Cogent 2000 (http://www.vislab.ucl.ac.uk/cogent_2000.php).

There were five sessions (lasting ∼9 min) composed of 48 trials each, leading to a total of 240 trials. The behavioral task was composed of the following two conditions: a delay condition and an effort condition, which were presented in a random order. Each trial started with the presentation of a cue (0.5 s) showing an erotic fuzzy picture of a naked woman. The screen then displayed the instruction “Wait?” or “Squeeze?,” together with a thermometer indicating a proposed level that could be displayed at six different heights (Fig. 1). This level indicated either the proposed delay period, which could last between 1.5 and 9 s (increment: 1.5 s) or the proposed force to exert, varying between 15 and 90% (increment: 15%) of the subject's maximal strength. In the delay condition, subjects had to decide whether they chose the costly option (i.e., wait the longer delay period indicated by the height of the level on the thermometer) to see the cue clearly for 3 s (large reward), or whether they chose the default option (i.e., wait for a fixed short period of time of 1.5 s) to view the picture for 1 s (small reward). Similarly, in the effort condition subjects decided whether it was worth investing in a stronger effort to see the picture clearly for 3 s or simply to invest in a small effort corresponding to 15% of their maximal strength to see the cue for 1 s. Subjects made their choice with a response pad in their left hand (accept costly option, forefinger; reject costly option, middle finger).

After their choice, the thermometer was framed in a red/blue rectangle to visually indicate which option was chosen. Then, subjects were involved in waiting or in exerting an effort proportional to the level indicated on the thermometer. In the delay condition, subjects passively waited until the required time had elapsed and the thermometer was filled up to the indicated level on the thermometer. In the effort condition, subjects squeezed the hand grip until they reach the indicated level on the thermometer. Finally, at the outcome, according to their choice (costly vs default option), they viewed the erotic picture clearly for 3 or 1 s. The duration of the display of the cue plus the proposition (instruction screen) was 4 s plus a jitter of ±1000 ms. If the subject did not make his decision during this time, the trial was aborted and the instruction “Pay attention” was displayed for 2 s.

The trial ended with an intertrial interval of 1.5 s plus a jitter of ±1 s when subjects accepted the proposition and 3.5 s plus a jitter of ±1 s when they rejected it. In this way, the outcome and the intertrial interval lasted for a total of 4.5 s plus a jitter of ±1 s in both options, to avoid that subjects adopt the strategy of choosing more the default option to see more pictures. Note that we chose not to make each trial the same length overall because this would have considerably extended the duration of the experiment (each delay trial would have lasted 17.5 s on average) and because this may have led subjects to no longer accept as many trials as is the case in the current version of the experiment, since short delayed options would lead to waiting overall the exact same duration as the delayed rewarded option. Subjects were explicitly asked to make their choices according to both the fuzzy cue and the proposed level of delay/effort and to weigh the cost and benefit of each option. They were also told that systematically choosing the less costly option would not allow them to see more pictures or finish the experiment earlier and that the sexual intensity of the fuzzy pictures was not linked to the level of proposed delay/effort since these images were presented in a random order. The six different levels were randomly presented across sessions with an average of 20 trials per level and per condition.

Stimuli.

Each picture was presented twice: once with a fuzzy appearance at the beginning of each trial (incentive cue) and once in a clear form at the outcome. Erotic pictures of women were used because—contrary to monetary rewards—they allowed us to include waiting periods in the second range and to have subjects really experience the reward delivery in each trial during scanning. Despite their critical sociobiological importance, erotic stimuli have not been studied as reinforcers, but rather as arousing stimuli in passive viewing paradigms focusing on sexual function.

Five hundred fifty erotic pictures were selected from the World Wide Web according to two objective criteria: women had to be alone and their face visible. These pictures underwent a glass effect in Adobe Photoshop 7.0 using the following parameters: distortion, 10; smoothness, 5; texture type, frosted; scaling, 100; and without invert texture.

The use of fuzzy cues had several advantages. First, displaying them at the beginning of each trial allowed preserving the saliency of the clear picture displayed at the outcome phase and avoided the habituation that could have occurred if only the clear pictures had been repeated (Agmo, 1999). Second, they had the power to motivate the subject by giving an idea of the content of the picture without unveiling the rewarding aspect of the clear picture. Hence, they allowed us to partially guide subjects' choices. Moreover, performing postscan ratings of each fuzzy cue, further used in subjective value computation, allowed us to avoid the assumption that the subjective value of monetary rewards increases linearly with an objective amount of money (Pine et al., 2009).

Five hundred sixteen of these pictures were rated by 30 men in a pilot experiment using the software Presentation 9.9 (Neurobehavioral Systems) (the other 34 pictures were used as examples). The 240 best rated pictures considered as the most rewarding pictures were selected for our delay/effort discounting task. The rationale for using different viewing durations for the clear picture was based on the fact that subjects work harder to see an attractive face longer because it is more rewarding (Hayden et al., 2007).

The duration of the erotic pictures seen clearly and the various delays and efforts used were initially piloted in several behavioral experiments, including ∼50 participants, to ensure that an approximately equal number of immediate and delayed options were chosen by the subjects.

Ratings of the fuzzy cues.

At the end of the scanning session, subjects were asked to rate the 240 fuzzy cues they saw during the experiment. They had to answer the question: “How much would you like to see this fuzzy picture in clear?” by moving a cursor along a scale going from 1 (“I do not want to see the fuzzy picture in clear at all”) to 9 (“I extremely want to see the fuzzy picture in clear'”) with an increment of 0.2. This incentive measure of each picture was used to compute the subjective values (corresponding to the reward intensity, called A in Eq. 1 below). The ratings of the cues were also used in behavioral analysis (see Fig. 2) and in functional MRI (fMRI) region of interest (ROI) analyses (see below, Computational model, and Figs. 5b, 6b), after normalizing and sorting them into one of four categories (category 1 being the lowest rated pictures and category 4 being the highest rated pictures). For the latter analyses, we collapsed the original nine levels of ratings into four bins to ensure a sufficient number of repetitions in each bin and to generate robust statistics. Note that the levels of delays and effort presented on the thermometer were not rated after the scan and were assumed to be perceived in a linear fashion by the subjects.

Computational model.

We used a hyperbolic function to compute the subjective values of delayed rewards because it has previously been shown to offer the best account of delay discounting in both humans and animals (Ainslie, 1975; Frederick et al., 2002). A similar hyperbolic function was used to compute the subjective values of rewards associated with a larger effort. On a trial-by-trial basis, the model estimated the subjective value of the reward associated with the costly option and the default option in the delay condition (SV_D) and in the effort condition (SV_E). These subjective values were computed as follows: Here, A corresponds to the reward intensity (ratings of the fuzzy cues), x is a subject-specific constant corresponding to the ratio between viewing the clear picture for 3 s (large reward) and viewing it for 1 s (small reward), C corresponds to the proposed level of the cost, and k is a subject-specific constant corresponding to the discount factor. This model was then used to create a parametric regressor corresponding to the estimated subjective value of the rewards associated with the costly option in a given condition for analysis of brain images.

The associated probability (or likelihood) of choosing the costly option was estimated by implementing the softmax rule, as follows: This standard stochastic decision rule allowed us to compute the probability of choosing the costly option according to its associated subjective value (supplemental Figs. 4, 5, available at www.jneurosci.org as supplemental material). The temperature β is a free parameter concerning the randomness of decision making. The parameters x, k, and β were adjusted using the least square method to minimize the distance between the behavioral choice and the probability of choice estimated by the model, across all sessions and subjects.

fMRI data acquisition.

Imaging was performed on a 3 tesla TRIO TIM scanner (Siemens Medical Solutions). T2*-weighted echo-planar images (EPIs) were acquired in an interleaved order with blood oxygen-dependent level (BOLD) contrast. Whole-brain functional images were acquired in 35 slices (128 × 128 voxels, 2 mm slice thickness, 2 mm interslice gap, 30° off of the anterior commissure-posterior commissure line at a repetition time of 1.98 s). We used a tilted plane acquisition sequence to optimize functional sensitivity in the orbitofrontal cortex (Weiskopf et al., 2006). T1-weighted structural images were also acquired, coregistered with the mean EPI, segmented, and normalized to a standard T1 template. EPI images were analyzed in an event-related manner, within a general linear model (GLM), using the statistical parametric mapping software SPM5 (Wellcome Department of Imaging Neuroscience, London, UK). The first three volumes of each session were discarded to allow for T1 equilibration effects. Before the analysis, the images were corrected for slice time artifacts, spatially realigned, normalized using the same transformation as structural images, and spatially smoothed with an 8 mm full-width at half maximum Gaussian kernel.

GLM1: main fMRI data statistical analysis.

We used one main linear regression model to account for our data (GLM1). Each trial was modeled as having three different phases, corresponding to the decision-making phase, the cost-enduring phase and the outcome phase. Trials were sorted according to the condition (delay or effort) and distributed into separate regressors. Two regressors were used to account for the decision phase: one for the delay condition and one for the effort condition. The subjective value of the reward associated with the costly option was included as a parametric modulation on these two regressors. Two regressors were used to account for the experienced delay period and the effort investment period (one for each condition). Two additional regressors accounted for the outcome phase, one for the small reward and the other one for the large reward, regardless of the condition. Therefore, the design matrix contained eight regressors of interest, all convolved with a canonical hemodynamic response function (HRF) and modeled with boxcars having the durations of the corresponding event. To correct for motion artifact, subject-specific realignment parameters were modeled as covariates of no interest. Linear contrasts of regression coefficients were computed at the individual subject level and then taken to a group level random-effects analysis (one-sample t test).

We applied a threshold of p < 0.001 (uncorrected) with a cluster-based threshold level of p < 0.05 corrected for multiple comparisons. All the results concerning this model exclusively concern the decision-making phase.

Additional control analyses: GLM2 and GLM3.

Because the subjective value reflects a ratio between the rating of the cue and the level of proposed delay or effort, an increased BOLD response with subjective value of the costly reward could reflect a positive correlation with the ratings of the incentive, a negative correlation with the proposed level of delay or effort, or a combination of both the rating and the level of the cost. Conversely, a decreased BOLD response with subjective value of the costly reward could reflect a negative correlation with the ratings of the incentive, a positive correlation with the proposed level of delay or effort, or a combination of both the rating and the level of the cost. To investigate whether the brain regions showing activity correlating with subjective value of the costly reward were better accounted by the subjective values (Eqs. 1 and 2) than by the rating of the incentive and the level of the cost in a given condition at the time of choice, we performed two additional fMRI analyses (GLM2 and GLM3) with these single parameters as parametric regressors (rating of the cue alone and level of the cost alone modeled in separate regressors for the delay and effort conditions), assessing whether both the strength (peak z-score) and spatial extent (cluster size in millimeters) of the activity of the cluster correlating with subjective value were larger than those of these single parameters (supplemental Fig. 7, supplemental Tables 1–3, available at www.jneurosci.org as supplemental material). These two analyses were similar to our main GLM1 except that the subjective value regressor was replaced by a single regressor containing the ratings of the fuzzy cue in the first analysis (GLM2) and the level of the cost in the second one (GLM3).

ROI analyses.

To gain more insight into the correlational analysis obtained with subjective value, we performed additional ROI analyses with the parameters of the task (rating and level of cost) to plot the respective influences of the proposed level of effort or delay and the influence of the ratings of the incentive in different brain regions. Regions of interest, conducted with the extension of SPM MarsBaR (http://marsbar.sourceforge.net/), were defined functionally from the initial whole-brain analysis (GLM1). Each ROI was created by taking the intersection of the functional cluster of interest and an 8-mm-radius sphere centered on the highest peak voxel of the cluster corresponding to the Montreal Neurological Institute coordinates reported in Table 1. In the delay condition, the functional clusters of interest were the ventromedial prefrontal cortex (vmPFC) and the ventral striatum because of their reported role in coding the subjective value of delayed rewards in the monetary domain (Tanaka et al., 2004; Kable and Glimcher, 2007; Peters and Büchel, 2009). In the effort condition, for the positive correlation with subjective value of the effortful reward, the only ROI was the primary motor cortex (M1) because it was the main activation found in this regression analysis. For the negative correlation with subjective value of the effortful reward, ROIs were defined in the anterior cingulate cortex (ACC) and anterior insula cortex because a previous study in rodents reported effects of ACC lesions on effort-related decision making (Rudebeck et al., 2008), because higher ACC and anterior insula activities were reported during anticipation of higher reward in the context of effort-based cost–benefit valuation in humans (Croxson et al., 2009) and because previous fMRI studies also reported an important role for the ACC–anterior insula network when making decisions under uncertainty (Huettel et al., 2005; Grinband et al., 2006). Figures 3b and 4b show the parameter estimates obtained from GLM1, GLM2, and GLM3 in each ROI. Moreover, to illustrate the activities correlating with subjective value of the high reward in the delay and the effort condition, the β estimates of SV_D and SV_E were plotted (Figs. 3c, 4c). Note that no statistical test was performed on these SVs because the ROI analysis is not independent from the whole-brain analysis.

To further illustrate the shape of the correlational analysis with the subjective value, the rating, and the level of proposed cost, we estimated three additional GLMs (GLM4, GLM5, and GLM6). This allowed us to extract and isolate the percent signal change (averaged across subjects) according to the subjective value of the reward associated with the costly option (GLM4), the rating of the incentive cue (GLM5), and the levels of delay and effort costs (GLM6) (see Figs. 5, 6). In GLM4, each trial was modeled as having three different phases, corresponding to the decision-making phase, the cost-enduring phase, and the outcome phase. Trials were sorted according to the condition (delay/effort). Twelve regressors were used to account for the subjective values at the time of the decision: 6 for the delay condition and 6 for the effort condition. Two regressors were used to account for the experienced delay period and the effort investment period (one for each condition). Two additional regressors accounted for the outcome phase, one for the small reward and the other one for the large reward. Therefore, the design matrix contained 16 regressors of interest, all convolved with a canonical HRF. To correct for motion artifacts, subject-specific realignment parameters were modeled as covariates. For each session of each subject, the subjective values were equally divided in these six categories, from the lowest sixth to the highest sixth (being the high category). This allowed us to extract the percent signal change for each of these categories, which was averaged across sessions for each subject and then averaged across all subjects (Figs. 5a and 6a). To show the effects of the incentive value of the cue (ratings) and the level of proposed delay (costs) in the brain regions correlating positively with the subjective value of the delayed option, we performed GLM5 and GLM6. GLM5 had the same three different phases, with the four categories of rating of the cue as parametric modulation at the time of choice instead of the six categories of subjective values (Figs. 5b and 6b). The other GLM (GLM6) performed to plot the graphs of percentage BOLD change as a function of the levels of costs also had the same three different phases, with the six levels of proposed costs as parametric modulation at the time of choice instead of the four categories of rating of the cue (Figs. 5c, 6c).

Cost-enduring phase and outcome phase.

Finally, we took advantage of our fMRI design, allowing us to distinguish the brain regions involved in the valuation processes concomitant with the decision from the brain regions recruited during the cost-enduring phase and those activated by the large versus small rewards in each of the two conditions. For this analysis, we used a similar model to our main GLM1, except that this model included a parametric modulation by the level of the cost during the delay period and during the effort exerted, and also separated the large and small rewards for the delay and effort conditions. These functional data were analyzed by constructing a set of boxcars having the duration of each corresponding event (i.e., decision phase lasting the response times (RTs), delay/effort phase lasting the delay effectively waited and the effort exerted, and outcome phase lasting 1 s for the small reward and 3 s for the large reward), all convolved with a canonical hemodynamic response function. For the anticipation contrasts, we used a threshold of p < 0.001, uncorrected with a cluster-level threshold of p < 0.05 corrected for multiple comparisons. For the outcome contrasts, we used a threshold of p < 0.005, uncorrected, because of our very specific focus on ventromedial prefrontal cortex and striatum, which have both been robustly implicated in responding to the receipt of reward in a large number of studies (Knutson et al., 2001; Smith et al., 2010).

Finally, in a last analysis, we investigated whether the activation observed at the time of the rewarded outcome was modulated by the level of delay or effort that had just been experienced. We used a similar model to our main GLM1, except that this model included a parametric modulation by the level of the cost during the delay period and during the effort exerted, and also during the outcome period for the delay and effort conditions separately. These functional data were analyzed by constructing a set of boxcars having the duration of each corresponding event (i.e., decision phase lasting the RTs, delay/effort phase lasting the delay effectively waited and the effort exerted, and outcome phase lasting 1 s for the small reward and 3 s for the large reward), all convolved with a canonical hemodynamic response function. For this last analysis, we used a threshold of p < 0.001, uncorrected with a cluster-level threshold of p < 0.05 corrected for multiple comparisons.