Abstract
We examined whether older adults differ from younger adults in how they learn from rewarding and aversive outcomes. Human participants were asked to either learn to choose actions that lead to monetary reward or learn to avoid actions that lead to monetary losses. To examine age differences in the neurophysiological mechanisms of learning, we applied a combination of computational modeling and fMRI. Behavioral results showed age-related impairments in learning from reward but not in learning from monetary losses. Consistent with these results, we observed age-related reductions in BOLD activity during learning from reward in the ventromedial PFC. Furthermore, the model-based fMRI analysis revealed a reduced responsivity of the ventral striatum to reward prediction errors during learning in older than younger adults. This age-related reduction in striatal sensitivity to reward prediction errors may result from a decline in phasic dopaminergic learning signals in the elderly.
Introduction
Human aging is characterized by a substantial decline of presynaptic and postsynaptic markers of the dopamine system (Li et al., 2010). Recent theoretical accounts suggest that these deficits in dopaminergic neuromodulation lead to impairments in reward-based learning in older adults (Eppinger et al., 2011). However, the computational and neurophysiological mechanisms underlying age-related impairments in learning are unclear.
Electrophysiological findings in monkeys suggest that reward-based learning is driven by reward prediction errors—that is, discrepancies between actual and expected outcomes. These reward prediction errors can be captured by reinforcement learning (RL) models and seem to be coded in phasic changes of neuronal activity in the midbrain (Schultz et al., 1997; Niv and Schoenbaum, 2008). Recent studies using combinations of RL models and functional neuroimaging (model-based fMRI) in humans showed correlations between reward prediction errors and BOLD activity in the midbrain, ventral striatum (vStr), and ventromedial PFC (Pessiglione et al., 2006; D'Ardenne et al., 2008; Jocham et al., 2011).
Only a few fMRI studies to date have investigated age differences in reward processing. Two of them reported age differences in striatal activation during reward anticipation, but no age differences or greater striatal activity in older adults during outcome processing (Samanez-Larkin et al., 2007; Schott et al., 2007). However, these studies did not involve learning from outcomes and focused on dissociating reward anticipation from outcome processing. Two other studies looked at age differences in the BOLD response to different outcomes and revealed mixed results (Cox et al., 2008; Mell et al., 2009). Cox et al. (2008) used a probabilistic guessing task and found no substantial age-related changes in outcome-related activity in the striatum when outcomes were unrelated to performance. In contrast, Mell et al. (2009) applied a reversal-learning paradigm in which the outcomes were critical for performance adjustments and found greater learning-related BOLD signal change in the vStr in younger compared with older adults. These findings point to age-related impairments in learning when reward representations need to be updated to adjust behavior. These impairments may be due to a decline of dopaminergic prediction error signaling (Eppinger et al., 2011).
In contrast to previous studies, which used conventional fMRI approaches, here we examined age differences in reward-based learning using model-based fMRI. We applied a task in which participants could learn to choose actions that lead to reward (positive learning) and also learn to avoid actions that lead to aversive outcomes (negative learning; Fig. 1). Consistent with previous findings, we expected that older adults would be impaired in learning from reward (Eppinger and Kray, 2011; Eppinger et al., 2011). Based on the literature in younger adults, we predicted that activity in dopaminergically innervated areas such as the vStr and ventromedial PFC should be correlated with reward prediction error (Pessiglione et al., 2006; Klein et al., 2007; Jocham et al., 2011; Niv et al., 2012). Furthermore, consistent with the dopamine hypothesis of aging, we expected that age-related impairments in learning should be associated with reduced prediction-error-related activity in these areas.
Schematic of the task. The design involved five learning blocks with 48 trials each (24 per learning condition, randomly intermixed). Each block involved a new set of four stimuli (two per learning condition). Feedback was deterministic.
Materials and Methods
Participants
Fifteen younger adults and 15 older adults from the Princeton University (Princeton, NJ) community participated in the study. Two younger adults were excluded because of excessive head motion. One older adult was not able to stay in the fMRI scanner for the entire experiment. Another older participant was excluded because she performed at chance level in both learning conditions. The effective sample consisted of 13 younger adults (mean age = 28.8, SD = 3.3, 7 male) and 13 older adults (mean age = 70.0, SD = 4.6, 7 male), all of whom were right-handed. Participants gave written informed consent and the study was approved by the institutional review board of Princeton University. During the screening session, participants completed a biographical questionnaire and several psychometric tests: (1) the Digit–Symbol Substitution test (DSS; (Wechsler, 1982), (2) Raven's Progressive Matrices (Raven et al., 1998), and (3) the Spot-the-Word test (Baddeley et al., 1992). The psychometric data showed lower scores on the DSS test and Raven's matrices for older compared with younger adults (p < 0.001, η2s > 0.36). In contrast, older adults reached higher scores compared with younger adults on the Spot-the-Word test (p < 0.02, η2 = 0.22). Consistent with previous findings, these results suggest age-related reductions in fluid intelligence and age-related improvements in crystallized intelligence (Li et al., 2004).
Task
The stimuli were 20 colored images of objects (Snodgrass and Vanderwart, 1980). The feedback stimuli indicated a loss of 50 cents, a gain of 50 cents, or a neutral outcome (i.e., 0 cents). If participants missed the response deadline, the words “too slow” were presented. The participants were asked to make a two-choice decision upon presentation of a stimulus using one of two response buttons (left or right). They were instructed to learn the stimulus–response assignments based on the feedback and to maximize their wins and minimize their losses. The experiment consisted of five learning blocks. Each block involved four stimuli (two per learning condition) that were presented 12 times in random order (48 trials per block). Two stimuli (A and B) of each block were associated with the positive learning condition (Fig. 1). If participants responded with a left button press to A, they won 50 cents; if they responded with the right button, they received a neutral outcome (and vice versa for B). The other two stimuli (C and D) were associated with the negative learning condition. If participants responded with a right button press to C, they received a neutral outcome; if they responded with the left button, they lost 50 cents (and vice versa for D). Stimulus-response assignments were counterbalanced across learning blocks and subjects. Feedback was deterministic.
Procedure
The participants performed two sessions. In the first session, they were screened for MR eligibility and performed the psychometric tests and questionnaires. If they were eligible for MR imaging they were invited back for a second session in which they completed the learning task in the MR scanner. Before entering the scanner, participants performed a practice block in which they practiced the task until they reached an average accuracy of 65% or a total number of 150 trials. In the scanner, the participants performed 240 trials (120 per condition). For the fMRI session, subjects received $20.00 as compensation for participation and could win a performance-dependent bonus of $7.50. Participants also performed a delay-discounting experiment in the scanner. These data are reported in Eppinger et al., (2012).
The trial procedure started with the presentation of a fixation cross for 500 ms, followed by presentation of the stimulus for 500 ms. After the stimulus, a blank screen was presented for a variable interval (3.5–7.5 s). After the blank screen, the feedback stimulus was displayed for 500 ms, again followed by a blank screen displayed for a variable interval (3.0–7.0 s). The interstimulus intervals were jittered in 1 s steps according to a long-tailed exponential distribution (λ = 4.0), yielding a mean interstimulus interval (ISI) of 5.0 s (Hagberg et al., 2001). Accordingly, the range of the intertrial interval (ITI) was 8–16 s with a mean of 11.0 s.
As in previous studies, we used an adaptive response deadline procedure in which we adjusted the response deadlines (700–1300 ms) depending on the number of time-out trials (Eppinger et al., 2009; Eppinger and Kray, 2011). Mean response deadlines were 1071 ms (SD = 211 ms) for younger adults and 1139 ms (SD = 194) for older adults. Younger adults (M = 0.06, SE = 0.02) and older adults (M = 0.06, SE = 0.01) did not differ with respect to the number of time-outs (p = 0.90).
The stimuli were projected on a screen mounted at the rear of the scanner bore, which participants viewed through a series of mirrors. EPrime software (PST) was used for stimulus presentation. Manual responses were registered using an MR-safe button box. A pillow and foam cushions were placed inside the head coil to minimize head movements.
fMRI data acquisition
MRI data acquisition was performed on a head-dedicated 3 tesla MRI scanner (Allegra; Siemens) at Princeton University. At the beginning of the fMRI session, high-resolution (1 mm3), T1-weighted structural images were acquired using an MP-RAGE pulse sequence (160 axial slices; FOV, 256 mm; TR, 2500 ms; TE, 4.38 ms; flip angle, 8°). AC-PC-aligned functional images were acquired using a T2*-weighted EPI sequence (33 interleaved slices; voxel size 3 × 3 × 4 mm; FOV, 192 mm; TR, 2000 ms; TE, 30 ms; flip angle, 90°).
Behavioral data analysis
Accuracy and reaction time (RT) data were analyzed using MATLAB (MathWorks) and SAS (SAS Institute) software. Mean accuracy and RTs were averaged for each participant and learning condition into six consecutive equally sized trial bins (five trials per bin, learning condition and block). These data were submitted to 2 (age-group) × 2 (condition) × 5 (trial bin) mixed-effects ANOVAs.
Temporal difference learning model
We modeled subject's choice behavior using a temporal difference (TD) learning approach (Sutton and Barto, 1998) in a two-step process (compare Daw, 2011): First, we used a maximum-likelihood procedure to find the parameters with which an RL model best described the behavior of each subject. In this step, we used a simple RL model in which the reward prediction of action a in condition c was updated according to the difference between the outcome r and the prediction V as follows: V(a) = V(a) + α*(r − V(a)), with α as the learning rate. The value was translated into choice behavior using a softmax function: exp(β*V(a))/[exp(β*V(a)) + exp(β*V(b))], in which β is the inverse temperature parameter regulating competition between actions a and b. The free parameters α and β were determined such that the log-likelihood of the observed choice behavior given the model was maximized. In the second step, we used the average of the estimated α and β parameters (across all subjects) to run a full TD model given the subject's individual choices. We used this model to extract the reward predictions and prediction errors for the fMRI analysis. In the TD model, each trial was split into 11 time steps with stimulus presentation at time step 1 and reward delivery at time step 6. We modeled the value and the prediction error during each time step in each trial (Schultz et al., 1997). The prediction error was then calculated according to the temporal difference rule as follows:
where t is the current time step. The temporal discount parameter γ was set to 0.9. For the fMRI analysis, the prediction error δ and prediction estimates V were extracted from the model at two time points: the time of stimulus presentation (time step 1) and the time of reward delivery (time step 6). The δ and V regressors were modeled in the fMRI analysis as described below.
fMRI data analysis
fMRI data analyses were performed using AFNI (Cox, 1996) and SPM (SPM8; Wellcome Department of Imaging Neuroscience).
Spatial normalization.
To account for age differences in brain morphology, the fMRI images were registered to a study-specific gray matter template using the DARTEL toolbox in SPM8 (Ashburner, 2007). The motivation for this procedure was to avoid a normalization bias toward the anatomy of younger adults (Samanez-Larkin and D'Esposito, 2008). In an initial step, we registered the functional images to the high-resolution T1 image (within each subject) in AFNI using a local Pearson correlation cost function (Saad et al., 2009). Functional and structural images of each participant were then manually aligned with the SPM tissue probability maps and segmented into their tissue components using the unified segmentation procedure (Ashburner and Friston, 2005). The resulting gray and white matter images were used during the DARTEL procedure to create the study-specific gray matter template (Ashburner, 2007; Harris et al., 2009). In addition to the template, the procedure results in so-called flow fields that parameterize the nonlinear deformations that are applied to match each individual image to the template. These flow fields were used to normalize the smoothed (Gaussian FWHM 8 mm) EPI data to an MNI-registered version of the template.
Preprocessing.
The functional data were slice-time corrected to the second volume using Fourier interpolation and realigned using rigid-body 3D motion correction as implemented in AFNI. Transient spikes in the EPI data were removed using the AFNI program 3dDespike. Percentage signal change was calculated for each voxel with respect to the mean activation across the time series.
Statistical analyses.
For condition-based analysis, we performed a general linear model (GLM) analysis for each participant's fMRI data using eight experimental regressors. For the stimulus-locked analysis, two of those regressors modeled the onset of stimuli that were followed by reward (+50 cents) and neutral (−00 cents) feedback (positive learning condition). Another set of two regressors modeled the onset of stimuli that were followed by negative (−50 cents) and neutral (+00 cents) outcomes (negative learning condition). For the outcome-locked analysis, two regressors modeled the onset of reward (+50 cents) and neutral feedback (−00 cents) (positive learning condition). Two other regressors modeled the onset of loss outcomes (−50 cents) and neutral feedback (+00 cents) (negative learning condition). These regressors were convolved with a canonical hemodynamic response function (HRF). To control for potentially confounding effects of RT, we included RT as a regressor of no interest in the analyses. We modeled baseline drifts in the data using a fourth-degree polynomial and included the motion correction parameters into the GLM. For the model-based analysis, we used the parameter estimates from the TD model to examine correlations of BOLD activity with prediction error (δt) and expected value (Vt). To analyze correlations with expected value, we created parametric regressors separately for the two learning conditions by modulating the stimulus-related amplitude of the canonical HRF with the expected value (Vt). To analyze correlations with prediction error, we created parametric regressors separately for the two learning conditions by modulating the outcome-related amplitude of the canonical HRF with prediction error (δt). Motion parameters and RT were added as regressors of no interest to the model.
For the outcome-locked analyses, the β-coefficients from the single-subject GLM analysis were subjected to a whole-brain mixed-effects ANOVA with fixed-effect factors of age group (younger vs older adults) and outcome type (+50, −50, +00, −00) plus the random factor of subjects. To examine age group × learning condition interactions, we focused on two contrasts: reward (+50) > neutral feedback (−00) (positive learning) and loss outcomes (−50) > neutral feedback (+00) (negative learning). The regression coefficients from the parametric prediction error analysis were analyzed using a whole-brain mixed-effects ANOVA with the fixed-effect factors of age group (younger vs older adults) and learning condition (positive, negative) plus the random factor of subjects. Unless stated otherwise, we used the AFNI program AlphaSim to correct for multiple comparisons. We determined that a corrected (family-wise) p-value of 0.05 was achieved with a minimum cluster size of 52 voxels, each significant at p < 0.001.
Results
Behavioral results
Accuracy data
The analysis of the accuracy data revealed a significant interaction between age group and learning condition (F(1,24) = 8.24, p = 0.008, η2 = 0.25). Separate analyses for the two learning conditions revealed significant age differences in accuracy in the positive learning condition (p = 0.01, η2 = 0.24), but not in the negative learning condition (p = 0.35). As shown in Figure 2A, older adults exhibited impairments in learning from monetary rewards, but not monetary losses. The analysis also revealed a significant interaction between age group and trial bin (F(5,120) = 8.13, p < 0.0001, η2 = 0.05, ε = 0.54), which reflects the fact that, across learning conditions, the younger adults learned somewhat faster than the older adults (Fig. 2A). These results show impairments in learning from positive but not negative feedback in older compared with younger adults (Fig. 2A).
A, Accuracy in proportion correct (y-axis) averaged into six equally large trial bins (x-axis) displayed separately for the positive learning condition (left) and the negative learning condition (right). Younger adults are shown in black, older adults are shown in gray. Error bars reflect the SEM. B, RTs in milliseconds (y-axis) averaged into six equally large trial bins (x-axis) displayed separately for the negative learning condition (left) and the positive learning condition (right). Younger adults are shown in black, older adults are shown in gray. Error bars reflect the SEM.
RT data
The analysis of the RT data showed a significant main effect of learning condition, reflecting longer latencies in the negative than the positive learning condition (F(1,24) = 10.04, p = 0.004, η2 = 0.29; Fig. 2B). Furthermore, we obtained a significant main effect of trial bin (F(5,120) = 11.05, p < 0.001, η2 = 0.30, ε = 0.59). As shown in Figure 2B, RT decreases as function of learning. No significant main effect or interactions involving the factor of age group were obtained (p > 0.17). Therefore, across age groups, avoiding actions that lead to aversive outcomes is associated with longer RTs than approaching reward.
fMRI results
Neural systems involved in learning from positive and negative outcomes
For the positive learning condition (positive > neutral outcomes), we found significant activations in the ventromedial PFC (vmPFC), the dorsal anterior cingulate cortex (dACC), and the posterior cingulate cortex (PCC) across groups (all t > 4.5, p < 0.05; Fig. 3A). For the negative learning condition (negative >neutral outcomes), we found significant activations in the dACC, inferior parietal cortex, dorsolateral PFC (dlPFC), the insula, the PCC, the bilateral putamen, and motor areas (all t > 5.4, p < 0.05; Fig. 3B). In the whole-brain analysis, a significant interaction between age group and learning condition was obtained in the vmPFC (F(1,24) = 13.17, p < 0.05). Separate analyses for the two learning conditions showed greater BOLD activity in the vmPFC for younger than older adults in the positive, but not in the negative learning condition (t = 3.6, p < 0.001, cluster size >20 voxels; Fig. 3C). Time courses for activity in the vmPFC, vStr, SMA/ACC, and dlPFC are shown in Figure 4.
A, Significant activations (t statistics) for the positive learning condition across age groups. BOLD activity is time locked to feedback onset. Activations are significant at p < 0.05, corrected for multiple comparisons. B, Significant activations (t statistics) for the negative learning condition across age groups. BOLD activity is time-locked to feedback onset. Activations are significant at p < 0.05, corrected for multiple comparisons. C, Significant main effect of age group in the positive learning condition in the ventromedial PFC (Talairach coordinates: −6, 39, 0, t statistics, significant at p < 0.001, cluster size >20 voxels).
A, Time courses for activity in the vmPFC and vStr (Fig. 3A) for the positive learning condition (+50 and −00 outcomes, left) and the negative learning condition (−50 and +00 outcomes, middle) separately for younger adults (black) and older adults (gray). Right: Difference in percentage signal change between positive and neutral outcomes (positive learning) and negative and neutral outcomes (negative learning) separately for younger adults (black) and older adults (gray). B, Time courses for activity in the SMA/ACC and dlPFC (Fig. 3B) for the positive learning condition (+50 and *00 outcomes, left) and the negative learning condition (−50 and *00 outcomes, middle) separately for younger adults (black) and older adults (gray). Right: Difference in percentage signal change between positive and neutral outcomes (positive learning) and negative and neutral outcomes (negative learning) separately for younger adults (black) and older adults (gray).
Correlations between BOLD activity and reward prediction error
To examine correlations between prediction error estimates and BOLD activity, we performed a parametric regression analysis. As shown in Figure 5A, this analysis revealed significant positive correlations between prediction error estimates and BOLD activity in the positive learning condition (across age groups) in the vmPFC, dACC, PCC, bilateral putamen, bilateral insula, and the thalamus (all t > 4.9, p < 0.05). Therefore, these areas show greater activity the more positive the prediction error. For the negative learning condition, we found significant negative correlations with prediction error in the dACC, the PCC, the left putamen, and the right middle temporal gyrus (all t > 3.6, p < 0.05, Fig. 5B). Therefore, the more negative the prediction error, the higher the activity in these areas. Most interestingly, we also observed a significant interaction between age group and learning condition in the vStr (F(1,24) = 9.46, p < 0.005, cluster size >20 voxels). Separate analyses for the two learning conditions revealed reduced prediction-error-related activity in the positive learning condition for older compared with younger adults in the vStr and vmPFC (all t > 3.6, p < 0.001, cluster size >20 voxels). No significant age differences were obtained for the negative learning condition. As shown in Figure 5C, older adults show reduced correlations between activity of limbic and paralimbic areas (vStr and vmPFC) and prediction error during learning from reward compared with younger adults. No effect of age group was observed for the negative learning condition.
A, Significant within-subject correlations (t statistics) between BOLD activity and prediction error estimates for the positive learning condition across age groups. Activations are significant at p < 0.05, corrected for multiple comparisons. B, Significant within-subject correlations (t statistics) between BOLD activity and prediction error estimates for the negative learning condition across age groups. Activations are significant at p < 0.05, corrected for multiple comparisons. C, Significant age differences in the correlations between BOLD activity and prediction error estimates for the positive learning condition in the ventromedial PFC (Talairach coordinates: −7, 42, −1) and vStr (Talairach coordinates: 12, 5, −1; t statistics, significant at p < 0.001, cluster size >20 voxels).
Discussion
The present behavioral results suggest that older adults learn as well as younger adults to avoid actions that lead to aversive outcomes. In contrast, they perform worse than younger adults when learning to approach reward. Our results are consistent with findings that point to age-related impairments in learning (Nieuwenhuis et al., 2002; Eppinger et al., 2008). Furthermore, the current findings support the idea that a decline in dopamine function with age may lead to an asymmetry in learning from positive and negative outcomes (Frank and Kong, 2008). Similar to the behavioral data, age differences in the fMRI data were only obtained for the reward-based learning condition. As shown in Figure 3C, older adults had reduced activity in the vmPFC for rewarding than neutral outcomes compared with younger adults. Moreover, consistent with impairments in learning, older adults showed reduced reward prediction-error related activity in the vStr and vmPFC than younger adults (Fig. 5C). No such effect was observed for loss prediction errors in negative learning condition. Therefore, the present findings indicate that in older adults, limbic and paralimbic areas (such as the vStr and vmPFC) are less responsive to reward prediction errors. These findings are consistent with the idea of an age-related decline of dopaminergic projections from the midbrain to the vStr and vmPFC (Hämmerer and Eppinger, 2012).
It should be noted that our interpretation does not imply that older adults cannot represent positive prediction errors. In both conditions, positive and negative prediction errors contribute to learning. The distinction we would like to make is between the absolute versus the relative value of outcomes. In the positive learning condition, positive outcomes have both positive absolute value (a reward) and positive relative value (compared with the neutral outcome). In contrast, in the negative learning condition, positive (neutral) outcomes have a positive relative value (compared with a loss), but they do not have absolute positive value (no reward). Insofar as age-related impairments were only observed in the positive learning condition, we interpret this as evidence of a reduced sensitivity to prediction errors during learning from reward, which may result from deficient dopaminergic input to the vStr and vmPFC in older adults (Eppinger et al., 2012). Furthermore, according to this interpretation, the absence of age differences in the negative learning condition is due to the fact that a reduced sensitivity to prediction errors does not affect learning in this condition.
Our findings are consistent with evidence indicating that suboptimal financial decision-making in older adults is associated with increased variability of vStr activity (Samanez-Larkin et al., 2010). The current data add to these results by suggesting that a reduced coupling between reward prediction errors and striatal activity may contribute to age-related decision-making deficits. Whether this effect is due to noisier dopaminergic signaling or if it results from impaired regulatory input from the vmPFC is an open issue (Takahashi et al., 2011). Support for the latter interpretation comes from a diffusion tensor imaging study showing that reduced white matter integrity in corticostriatal pathways accounts for age differences in reward-based learning (Samanez-Larkin et al., 2012).
One alternative interpretation of the current results could be that they reflect age-related deficits in contextualizing neutral outcomes—that is, in considering them relative to the alternative (gain or loss) in the two conditions (Braver et al., 2001). For example, older adults might have perceived the neutral outcomes as positive across conditions (e.g., experiencing relief that nothing worse happened). This would have reduced the difference in their experienced value from reward in the positive learning condition, which could explain age-related learning impairments in this condition. However, it is unclear why such a contextualization deficit would have been restricted to positive learning. By the same logic, neutral outcomes could have induced “regret” because they corresponded to a missed opportunity for reward. More generally, it is unclear why it should be assumed, a priori, that there should have been an asymmetry of these effects and/or why it should have been restricted to older adults. Furthermore, the outcome-related BOLD data do not provide support for this idea. As shown in Figure 4, there is no evidence for a positive BOLD response to neutral outcomes in the vStr and there are no significant differences in BOLD responses to neutral outcomes between age groups.
From a broader lifespan developmental perspective, the current data support the idea of a nonlinear trajectory of dopaminergic function across the lifespan (Cohen et al., 2010; Eppinger et al., 2012). However, it is possible that factors other than a decline in dopamine contribute to our results. For instance, it could be argued that the observed negative learning bias reflects a greater focus on accuracy in the elderly (Starns and Ratcliff, 2010). However, this would imply that older adults should show enhanced (compensatory) activity during negative learning to reach similar performance levels as younger adults, which does not seem to be the case. It could also be argued that older adults approach reward-based learning tasks more strategically and test hypotheses regarding the task structure. This may hurt them in a condition in which the optimal policy is purely reward driven (Worthy et al., 2011). As stated above, the data do not support this idea in terms of enhanced activity during negative learning in the elderly.
Of course, the current findings have to be considered cautiously given the small samples and scarce literature regarding age-comparative model-based fMRI analyses. Furthermore, valence biases are most likely context-dependent. Therefore, in a different experimental context, such as during retrieval of emotionally salient events, older adults may (strategically) focus on positive information (Mather and Carstensen, 2005). In contrast, in situations in which negative feedback is important for behavioral adjustments, as in the current task, they may focus on negative outcomes.
Our data also provide support for a second theoretical proposal, namely that reward learning relies on dopaminergic reward prediction errors, whereas negative learning relies, at least partially, on activity in cognitive control areas such as the dlPFC and dACC (Hämmerer and Eppinger, 2012). During positive learning, we found activity in the vmPFC (Fig. 3A). This is consistent with the role of this area in reward valuation (Grabenhorst and Rolls, 2011). In contrast, for negative learning, we found activity in cognitive control areas such as the dlPFC, the dACC, and the inferior parietal lobule (IPL) (Fig. 3B). These findings support the idea that learning from negative feedback may rely on structures involved in performance monitoring (Botvinick et al., 2004). Consistent with this interpretation, we found increased RT for negative compared with positive learning across groups, indicating that both groups slow down to avoid negative feedback (Fig. 2B).
Further support for the idea of partially separable neural systems involved in learning from gains and losses comes from the model-based fMRI analysis (Fig. 5). This analysis showed positive correlations between reward prediction error and BOLD activity in the putamen, dACC, and vmPFC during positive learning. In contrast, for negative learning, we observed a negative association between loss prediction error and activity in the putamen and dACC. Therefore, similar to results from studies on pain conditioning, our findings suggest an inverse coding of reward and loss prediction errors and point to partially separable systems involved in representing appetitive and aversive prediction errors (Seymour et al., 2004; Seymour et al., 2005). Moreover, also consistent with findings by Seymour et al. (2005), results of a conjunction analysis showed that reward and loss prediction errors correlate (inversely) with activity in the putamen and dACC (Fig. 6). This may indicate that these areas are involved in integrating information about appetitive and aversive prediction errors. Therefore, our results provide evidence for a common neural system involved in prediction error processing, independently of learning condition, involving the putamen and dorsal ACC (Pessiglione et al., 2006; Kim et al., 2006). However, there are also limbic and paralimbic areas that are uniquely activated during reward learning, such as the vmPFC (Yacubian et al., 2006; Liu et al., 2007).
A, Conjunction analysis for learning conditions. Green: Areas activated in the positive learning condition (contrast: +50 > −00). Orange: Areas activated in the negative learning condition (contrast: −50 > +00). Red: areas activated in both learning conditions (conjunction). All activations significant at p < 0.005, cluster size >20 voxels. B, Conjunction analysis for prediction-error related activity. Green: Areas activated in the positive learning condition (contrast: +50 > −00). Orange: Areas activated in the negative learning condition (contrast: −50 > +00). Red: Areas activated in both learning conditions (conjunction). All activations significant at p < 0.005, cluster size >20 voxels.
To conclude, the present findings show age-related impairments in reward-based learning, but not in learning from negative feedback. Across age groups, learning from reward was associated with greater activity of paralimbic areas such as the vmPFC. In contrast, during negative learning, we found greater activity of cortical cognitive control areas, such as the dlPFC, dACC, and IPL. Consistent with the behavioral results, we observed reduced BOLD activity in the vmPFC in older adults during reward-based learning. No age differences were observed during negative learning. The model-based fMRI results show a reduced coupling between reward prediction errors and activity in the vStr and vmPFC in the older adults. No such effect was observed for loss prediction errors during negative learning. These age-related changes in reward prediction error activity may result from a decline of dopaminergic input from the midbrain to the striatum and vmPFC in older adults. Therefore, the present findings indicate that reduced dopaminergic prediction error signaling may be one of the mechanisms that underlie age-related impairments in reward-based learning.
Footnotes
- Received June 21, 2012.
- Revision received March 15, 2013.
- Accepted March 20, 2013.
This research was supported by National Institute on Aging (Grant #AG02436 awarded through the Princeton Center for Health and Wellbeing). N.W.S. was supported by a fellowship from the International Max Planck Research School on the Life Course. We thank Jessica Hughes, Paul Kruger, Peter Foster, and Laura deSouza for assistance with data acquisition and Yael Niv for helpful suggestions regarding RL modelling.
The authors declare no competing financial interests.
- Correspondence should be addressed to Ben Eppinger, Max Planck Institute for Human Development, Lentzeallee 94, 14195 Berlin, Germany. eppinger{at}mpib-berlin.mpg.de
- Copyright © 2013 the authors 0270-6474/13/339905-08$15.00/0