Abstract
Maladaptive decision-making is increasingly recognized to play a significant role in numerous psychiatric disorders, such that therapeutics capable of ameliorating core impairments in judgment may be beneficial in a range of patient populations. The field of “decision neuroscience” is therefore in its ascendancy, with researchers from diverse fields bringing their expertise to bear on this complex and fascinating problem. In addition to the advances in neuroimaging and computational neuroscience that contribute enormously to this area, an increase in the complexity and sophistication of behavioral paradigms designed for nonhuman laboratory animals has also had a significant impact on researchers' ability to test the causal nature of hypotheses pertaining to the neural circuitry underlying the choice process. Multiple such decision-making assays have been developed to investigate the neural and neurochemical bases of different types of cost/benefit decisions. However, what may seem like relatively trivial variation in behavioral methodologies can actually result in recruitment of distinct cognitive mechanisms, and alter the neurobiological processes that regulate choice. Here we focus on two areas of particular interest, namely, decisions that involve an assessment of uncertainty or effort, and compare some of the most prominent behavioral paradigms that have been used to investigate these processes in laboratory rodents. We illustrate how an appreciation of the diversity in the nature of these tasks can lead to important insights into the circumstances under which different neural regions make critical contributions to decision making.
Introduction
Understanding the mechanism by which the brain makes decisions is perhaps one of the most fundamental questions for neuroscientists, psychologists, and economists alike. The demonstration that the humble laboratory rodent can show evidence of complex cost-benefit decision making, subject to similar preferences and biases as those that hallmark human cognition, has opened up numerous exciting possibilities. For example, researchers can apply optogenetic and chemogenetic techniques to dissect neural circuits and manipulate neuronal signaling with unprecedented neuroanatomical and temporal precision. However, for the novice (and even experienced!) behavioral neuroscientist, there may appear to be a bewildering array of different assays to choose from, all of which look superficially similar yet may actually tap into quite distinct cognitive processes and regulated by dissociable neural circuitries. Here, we illustrate this point and the insights gained through comparing task variants within two areas of particular interest, namely, decision making under uncertainty, and models of effort-based choice, before highlighting potential avenues for future research.
Decisions involving uncertainty
Given that nearly all “real-life” decisions involve an element of uncertainty, it is perhaps unsurprising that there is significant interest in understanding the deliberative process by which the brain chooses between probabilistic outcomes. From an economics perspective, risk is used to describe uncertain outcomes associated with known probabilities of occurrence. This concept has also been described as “expected uncertainty,” in which the unreliability of predictive relationships between environmental events is known to the subject (Yu and Dayan, 2005). This can be distinguished from unexpected uncertainty, or ambiguity, in which the degree of uncertainty associated with a particular response or event is unknown to the subject. Expected/risky and unexpected/ambiguous types of uncertain outcomes engage distinct neural circuitries (e.g., Huettel et al., 2006), and the type of uncertainty encountered may recruit different cognitive processes to guide choice (for review, see Winstanley et al., 2012). Most behavioral methodologies involve repeated training sessions or forced choice trials to expose the animals to the contingencies in play, and reduce the chance that animals are responding at random. Once acquired to a stable baseline, most behavioral tasks are therefore assessing risk rather than ambiguity.
Human decision making typically does not obey mathematical norms when choosing between uncertain outcomes, in that our choice is not always governed by the expected utility of different options. As famously described by Prospect Theory, our decision making is instead subject to various heuristics and biases, such as risk aversion and loss aversion (Kahneman and Tversky, 1979; Kahneman, 2011), and such choice patterns are thought to make important contributions to the maladaptive decision making observed in gambling disorder (GD) (Clark, 2010). Whether animals likewise show such mathematically irrational choice is important to consider when assuming that the cognitive processes underlying such decisions may approximate human behavior.
“Risk” is also commonly used to mean the “the possibility that something unpleasant or unwelcome will happen” (Oxford English Dictionary). Thus, the phrase “risky” choice could also imply the chances of not only something good failing to occur, but also something bad happening. In some assays designed to assess risk/reward decision making in animals, the “risk” may merely be a lack of reward, whereas others incorporate the potential for some additional aversive consequence, such as a shock (Simon et al., 2009), unpleasant taste (van den Bos et al., 2006), or prolonged time-out period (Rivalan et al., 2009; Zeeb et al., 2009). As we will describe below, decisions where rewards may potentially come with unpleasant outcomes may recruit different neural circuits to guide choice behavior compared with situations involving choice between some versus no reward. These considerations suggest a number of important ways by which decision-making tasks that require an assessment of uncertainty or risk could be clustered.
Signaling loss: rodent analogs of the Iowa Gambling Task (IGT) and related behavioral tasks
The IGT (Bechara et al., 1994) is one of the best known and widely used decision-making tasks. Subjects choose between four decks of cards, with the goal of accumulating the most points. Each card results in either a net gain, or a net loss. Two decks are considered “risky” or disadvantageous, in that choice can result in large per trial gains but also disproportionately heavy losses, such that continued selection from such decks will result in a reduction in points over time. The optimal strategy is therefore to favor the conservative or advantageous decks, choice from which results in smaller per trial gains and penalties, but gradual accumulation of profit throughout the session. Elevated risky choice has been reported in numerous clinical populations, including bipolar mania (Clark et al., 2001), psychopathy (van Honk et al., 2002), substance use disorders (Bechara and Damasio, 2002), and the presence of suicidal features (Jollant et al., 2005), to name just a few, and has been associated with damage to the amygdala and areas of frontal cortex (Bechara et al., 1999; Fellows and Farah, 2003).
Development of a rodent analog was considered a worthy endeavor by a number of groups, due to the high translational potential of any neurobiological mechanisms found to drive choice. The first obvious challenge is how to model the loss component, given that the food reinforcement typically used with animal tasks is consumed immediately upon delivery and cannot be repossessed. Cues signaling how much reward is cached may not be intuitively recognized by rodents, and the use of conditioned stimuli in the absence of immediate reward leads to issues of extinction over time. Instead, different groups have adopted a range of ingenious ways to either signal loss or substitute for the aversive nature of losses, namely, through one of the following: (1) adding an unpleasant taste to the food, as in the runway-based rat IGT (RIGT) (van den Bos et al., 2006); (2) delivering electric shocks with the reward in what has been termed the risky decision-making task (RDT) (Fig. 1A) (Simon et al., 2009); or (3) through the use of punishing time-outs that reduce the remaining time-to-earn reward to closely approximate “losses,” such as in the rat gambling task (rGT, Fig. 1B) (Zeeb et al., 2009).
Two assays used to probe decision-making involving uncertainty that use different magnitudes of rewards and distinct punishments. A, In the RDT, rats are trained to choose between two levers that delivers either 1 pellet (Small/Safe lever) or 4 pellets (Large/Risky lever), both of which are delivered with 100% certainty. However, selection of the Risky Lever may also deliver a foot shock punisher, and the probability of punishment increases over blocks of trials within a session. B, The rGT was patterned after the IGT used with human subjects. In each 30 min session, rats try to maximize sugar pellet profits by choosing between four options that vary in the number of sugar pellets (1–4) that can be earned on win trials, but also the probability (0.1–0.6) and duration (5–40 s) of punishing time-out periods delivered on losses. “Safe” options associated with smaller per-trial gains yield the most reward per session, whereas “risky” options appear tempting due to the larger reward size but result in less reward over time due to the longer and more frequent time-out punishments they are paired with. In the cued rGT, delivery of reward is accompanied by a 2 s audiovisual cue that increases in complexity with the reward size.
Regardless of other methodological differences between these three tasks variants, some clear similarities start to emerge when considering the neural circuitry underlying performance. Lesions to the BLA made after the animals have already acquired either the rGT (timeouts; Fig. 2A) or RDT (footshock punishment; Fig. 2B) lead to an increase in choice of the risky options (Zeeb and Winstanley, 2011; Orsini et al., 2015a). Thus, in these contexts, the BLA is crucial for mediating how negative outcomes diminish the value of otherwise more preferred rewards. This is in keeping with a broad literature suggesting that the BLA encodes the emotional significance of negative events, and may likewise explain why BLA lesions reduce the ability of a loss to spur further gambling in a rodent model of “loss-chasing” (Tremblay et al., 2014).
A comparison of the effects of BLA lesions/inactivation on three forms of decision-making involving uncertainty. A, On the rGT (see Fig. 1B), subjects choose between options that vary in terms of the probability of obtaining different magnitudes of reward associated with different durations of time-out punishments, with some options being more advantageous (“better”) in the long term. BLA lesions cause disadvantageous patterns of decision making, with rats selecting options that yield larger but less frequent rewards and receiving more time-outs. Adapted with permission from Zeeb and Winstanley (2011). B, In the RDT (see Fig. 1A), rats choose between smaller rewards and larger ones associated with increasing likelihood of foot shock punishment. Again, BLA lesions increase choice of the risky option, such that rats receive more foot shocks, as well as more reward. Adapted with permission from Orsini et al. (2015a). C, In contrast, the PDT entails choice between a small/certain reward and a larger, uncertain one, with the probability of receiving the large reward varying across blocks of trials within a session. Unlike the rGT or RDT, there are no explicit punishments associated with the uncertain reward. Here, BLA inactivations reduce choice of larger, uncertain rewards, most prominently when the uncertain option is of equal or greater utility relative to the certain one. Adapted with permission from Ghods-Sharifi et al. (2009).
Lesions to the orbitofrontal cortex made after extended training fail to alter choice on the rGT yet decrease risky choices associated with shock in the RDT (Zeeb and Winstanley, 2011; Orsini et al., 2015a). Although lesions/inactivations targeting the lateral portion of the OFC have not been performed to our knowledge on the RIGT, cFos levels within the OFC, measured ex vivo following task performance, did not correlate with levels of risky decision making (de Visser et al., 2011). Hence, the use of a painful stimulus (shock) alters how the OFC guides the direction of choice. Interestingly, lesions made to either the OFC or BLA, or disconnecting these two regions before the acquisition of the rGT, results in slower learning about which option is the most beneficial, suggesting that these areas form a functional circuit that enables efficacious development of optimal strategies (Zeeb and Winstanley, 2011, Zeeb and Winstanley, 2013). Whether functional connectivity between the BLA and OFC is necessary for acquisition of the other tasks mentioned here is, as yet, unknown.
With respect to other prefrontal regions, lesions/inactivations of the prelimbic (PrL) and infralimbic (IL) areas of the medial prefrontal cortex (anatomically homologous to anterior cingulate [ACC] areas 32/25) (Uylings et al., 2003; Heilbronner et al., 2016) leads to a suboptimal, risk averse pattern of choice on the rGT, whereby animals increase choice of the option associated with the most frequent reward that is of lowest value, but that is also associated with the lowest penalties (Paine et al., 2013; Zeeb et al., 2015). Inactivating the agranular insula had a similar effect (Pushparaj et al., 2015). However, as yet, no cortical area targeted has resulted in a clear increase in risky choice comparable with that seen after BLA lesions.
Given the pronounced role of striatal regions in mediating the link between affective states and motor output (nucleus accumbens, NAc), goal-directed responding (dorsomedial striatum) and habitual behavior (dorsolateral striatum) (Balleine and O'Doherty, 2010), there has been surprisingly little work done on elucidating the contributions made by these regions to performance of any of these tasks. Likewise, the contribution of dopamine (DA, a major input to the striatum) to decisions involving reward and punishment remains to be clarified. D1-selective ligands do not affect choice on the RDT or the rGT (Zeeb et al., 2009; Simon et al., 2011). Conversely, D2 receptor activation reduces risky choices associated with shock on the RDT, yet neither acute nor chronic administration of D2/3 agonists affect choice on the rGT (Zeeb et al., 2009; Tremblay et al., 2013). However, associating salient audiovisual cues with “wins” that increase in complexity with the magnitude of reward does render choice on the rGT sensitive to modulation by D3 receptors (Barrus and Winstanley, 2016). The picture is further complicated by the findings that systemic blockade of D2 receptors reduces risky choice on the rGT (where the punishment is a timeout) yet has no effect on the RDT using shocks as a punisher (Zeeb et al., 2009; Simon et al., 2011). Clearly, our limited understanding of how striatal and dopaminergic systems contribute to decisions involving rewards and punishment highlight this as an important topic for future research.
Certain versus uncertain rewards: uncertainty discounting tasks
Just as many laboratory-based tasks used with humans frame decision outcomes with respect to gain versus no gain, as opposed to gain versus loss, certain rodent paradigms likewise use probabilistic schedules in which rats can choose to respond for a smaller-certain versus larger-uncertain reward. A paradigm that has been used by a number of groups is one in which the odds of receiving the larger reward shift between blocks of trials in a systematic manner from 100% to 12.5%, either in a descending (“good-to-bad”) or ascending (“bad-to-good”) manner (Cardinal and Howes, 2005; St Onge and Floresco, 2009; Rokosik and Napier, 2011), reminiscent of the structure of some delay-discounting tasks (Evenden and Ryan, 1996).
A series of studies has systematically explored the neural circuitry underlying probabilistic discounting, much of which has recently been reviewed in depth (Orsini et al., 2015b). Inactivation of the BLA (Fig. 2C) or NAc, as well as disconnection of these two regions, decreased choice of the uncertain options (Ghods-Sharifi et al., 2009; Stopper and Floresco, 2011; St Onge et al., 2012). In contrast, inactivating the medial (but not lateral) OFC increased choice of the risky options, whereas inactivation of the insular cortex had no effect (St Onge and Floresco, 2010; Stopper et al., 2014). One key region involved in adjusting decision biases is the PrL PFC. Inactivation of this region increased risky choice when the probability of reward decreased during the session, had the opposite pattern of effects when the order of blocks was reversed, yet had no impact on choice when the reinforcement contingencies remained stable across the session (St Onge and Floresco, 2010). This fascinating pattern of results suggests that this region plays a key role in tracking the probability of reward over time, facilitating adjustments in choice preference in response to contingency shifts. Furthermore, selectively disrupting top-down PFC→BLA connections increased choice of the larger yet increasingly uncertain reward, implying that communication between these two regions serves to modify choice biases driven by BLA→NAc circuits (St Onge et al., 2012).
Dopaminergic transmission plays a key role in modulating this form of decision making. Unlike other tasks that incorporate some form of punishment, D1 receptor activity modifies decision making when risky choices either yield larger rewards or nothing. Reductions in risky choice have been observed when D1 antagonists have been administered systemically (St Onge and Floresco, 2009) or locally within the PFC (St Onge et al., 2011), NAc (Stopper et al., 2013), or BLA (Larkin et al., 2016). Moreover, reducing D1 receptor tone within the forebrain increases lose-shift behavior, suggesting that these receptors reduced the negative impact of unrewarded actions and help a decision-maker keep an “eye on the prize,” maintaining choice biases even when risky choices do not always yield rewards. In comparison, D2 receptor activity within the PFC facilitates shifts in decision biases during probabilistic discounting, as blockade of these receptors increases risky choice when odds shift from “good-to-bad” (St Onge et al., 2011). These findings highlight the complexity through which mesocortical DA transmission regulates different aspects of decision making by acting on different receptors and, presumably, distinct populations of PFC neurons (Gee et al., 2012; Seong and Carter, 2012). Interestingly, D2 receptors within the NAc do not appear to play a role in this form of decision making (Stopper et al., 2013). When comparing this lack of effect with the seemingly more prominent role D2 receptors play in decision making involving some form of punishment described earlier, it is notable that separate populations of striatal neurons expressing D1 or D2 receptors have been proposed to regulate different patterns of behavior. D1-containing cells may be more important for promoting approach behaviors, whereas activation of D2-expressing neurons can be aversive, suggesting their involvement in avoidance behaviors (Lobo et al., 2010; Kravitz et al., 2012). Thus, within corticostriatal circuits, D1 versus D2 receptors may mediate distinct forms of decision making dependent in part on whether the objective is to merely maximize rewards or avoid potentially aversive consequences.
Certain versus uncertain rewards of equal expected value
To address whether rats show decision-making biases similar to human subjects, tasks have been designed in which the options are matched in terms of expected utility. Preference for uncertainty or guaranteed rewards can therefore be ascertained without the potential confounds of differential reinforcement rates, similar to the original studies by Kahneman and Tversky (Tversky and Kahneman, 1974, 1981; Kahneman and Tversky, 1979). In the rat betting task (rBT, Fig. 3A), animals choose between either a guaranteed reward, or double that option or nothing with 50:50 odds. During the course of the session, the number of pellets available on the safe lever (the “bet size”) varies between blocks from 1 to 3 in a pseudo-random fashion, with changes in bet size signaled by forced choice trials and cue lights. Even though the relative utility of the guaranteed and probabilistic options remains equal, choice of the uncertain option decreases as the bet size increases, comparable to escalation of commitment effects in human subjects, whereby individuals are less willing to gamble as the stakes increase even though the odds of success remain constant (Cocker et al., 2012b).
A comparison of the effects of lateral OFC inactivation on two forms of decision making involving uncertainty. A, On a given trial of the Betting task, the number of pellets available on the certain lever (or “bet size”) are signaled by illumination of an equivalent number of cue lights on the opposite wall. A nosepoke at each illuminated aperture turns off the light inside. When all the lights have been extinguished in this way, the two levers are presented. Choice of the certain option results in guaranteed delivery of the bet size, whereas a response on the uncertain lever leads to double that amount or nothing with 50:50 odds. As such, there is no net advantage, with respect to pellets earned, in choosing one option over another: even though the bet size shifts across blocks of 10 trials between 1 and 3 sugar pellets, the relative utility of each option remains equal and constant. B, In the PDT, rats choose between a small/certain reward and larger reward delivered with varying probabilities. In contrast to the betting task, the utility of the two options differs and changes across trial blocks. C, Under control conditions on the rBT, some animals show sensitivity to bet size (wager sensitive, left), and decrease preference for the uncertain option when greater amounts of reward are at stake. Other rats are relatively wager insensitive (right). OFC inactivation increased choice of the uncertain option in wager-sensitive rats when bet sizes were large while having no effect on wager-insensitive rats. D, In contrast to the rBT, OFC inactivation does not affect choice behavior on the PDT. A comparison of these two observations suggests that the OFC may play a more prominent role in directing risk-related decisions influenced by subjective biases, independent of reward utility.
Individual rats vary considerably in their propensity to switch preference to the safe lever as bet size increases, such that rats may be classified as either wager-sensitive or -insensitive (Fig. 3C) (Cocker and Winstanley, 2015). With regards to the brain regions involved in mediating such biases, inactivations or lesions to the BLA, PrL, or IL regions are without effect on choice in either wager-sensitive or -insensitive rats, yet inactivation of the lateral OFC selectively ameliorates wager-sensitive choice patterns (Tremblay et al., 2014; Barrus et al., 2016). Recent data using a somewhat comparable task suggest that selective activation of NAc neurons expressing D2 receptors can ameliorate risky choice through signaling unfavorable outcomes during the decision-making process (Zalocusky et al., 2016). In keeping with such observations, acute and chronic administration of a D2/3 receptor agonist increased risky choice on these different decision-making tasks, whereas a D1 agonist was without affect (Tremblay et al., 2016; Zalocusky et al., 2016). Thus, even though D2 receptors in the NAc may not play as prominent a role in guiding risky choice in situations where the utility of options differ, they may nonetheless shape choice biases in other situations involving subjective preferences for risky versus more certain options.
Comparison of neural circuitries
As is often the case in neuroscience, the same manipulations have not been performed in all the tasks described above. Yet, the involvement of four key systems (the BLA, lateral OFC, mPFC, and DA system) have been assessed in at least the majority of the paradigms under consideration, which permits some comparison of their relative contribution to different aspects of decision making (Table 1). Whereas inactivating the BLA increases risky choice during both the rGT and RDT, it has the opposite effect on the PDT (Fig. 2). The manner in which the BLA guides choice during risky decision making thus depends on the nature of the penalties involved and corresponding response priorities of the decision-maker. When risky choices can incur some form of punishment, the BLA biases action selection to reduce the chances of aversive events. In comparison, when choosing between certain versus larger uncertain reward (but no other punishment), the neural activity in the BLA promotes choices that may maximize the amount of reward obtained. In contrast, neural activity in this nucleus does not appear to contribute to choice when options are matched for expected utility, as in the rBT.
Relationship between task features and neurobiology implicated in decision-making assays involving an assessment of uncertaintya
The lateral OFC has a pronounced role to play in risk-related decisions when there is a chance of a painful aversive stimulus, in which case this region appears to promote risky choice, perhaps through signaling the appetitive qualities of the reward. This manipulation had no effects in either the PDT (Fig. 3C,D) or rGT (at least once task contingencies have been learned), suggesting that the effects on the RDT are not attributable to disruptions in processing of loss signals or contingency shifts. In the rBT, inactivating the OFC selectively ameliorated wager-sensitive response bias, increasing choice of the risky option. A singular theory of OFC function has been notoriously difficult to establish (Stalnaker et al., 2015). However, when attempting to integrate these findings, both increases in choice of uncertain options in the rBT and decreases in risky choice in the RDT following OFC lesions may reflect alterations in the subjective evaluation of different outcomes. Thus, in the rGT and PDT, there are arguably objectively defined “optimal” strategies. In contrast, in the rBT, wager-sensitive rats shift preference away from the risky option as the bet size increases, even though this does not result in any change in overall reward rate; lateral OFC inactivations abolish this effect. In a similar vein, the risky option is always rewarded in the RDT, but the chances of punishment increase during the session. Given that the aversive and appetitive consequences associated with choice in this task are in different modalities, there is no “mathematical” norm by which choice can be judged. As such, silencing the OFC causes animals to behave as if the larger reward is not “worth” the penalty, which may be a reflection of diminished valuation of rewards. Thus, if subjective value is viewed as some internally generated reward assessment, based more on how the outcomes make the subjects feel, then perhaps the effects of lateral OFC inactivation on both tasks can be understood as a decrease in the impact that this value judgment has on choice.
On the other hand, the finding that inactivation of the PrL PFC reduces choice of the best option in the rGT and impairs modifications in choice biases when reward probabilities change in the PDT, suggests that this region promotes efficient choice behavior to ensure gain is maximized under conditions where the objective value of different options are discrepant. More specifically, this region of the frontal lobes serves to optimize reward seeking in situations presenting multiple options (i.e., more than two) or when the relative utility of different options is volatile.
Last, decisions that do not involve explicit aversive outcomes are exquisitely sensitive to manipulations of DA in multiple terminal regions, with D1 receptors playing a prominent role in promoting choice of larger uncertain rewards by reducing sensitivity to nonrewarded actions. In comparison, choice guided in part by the potential of aversive outcomes, as in the rGT or RDT, is relatively insensitive to D1 receptor manipulations, whereas tone on D2 receptors may be more consequential in modifying choice. This being said, DA transmission within the PFC can mediate seemingly opposing patterns of behavior by acting on different receptors. Thus, the principles of operation by which cortical and subcortical DA transmission refines different forms of risky decisions ares likely to be much more complex than simple dichotomies (e.g.; D1 = reward, D2 = aversion), as is typically the case with studies of monoamines.
Decisions involving effort-related costs
Deciding whether a particular outcome is worth the effort required to achieve it is another almost universal factor that influences daily decision making and may contribute to clinically relevant symptoms, such as anhedonia and anergia (Salamone et al., 1994; Treadway et al., 2009). Effortful tasks can generally be conceptualized as taxing physical or mental resources. Although human neuroimaging has started to delineate a somewhat overlapping network of frontostriatal regions implicated in both choice variants (e.g., Botvinick et al., 2009; Croxson et al., 2009; Treadway et al., 2012), it is unclear whether the allocation of cognitive and physical effort is underpinned by the same cognitive or neurobiological processes (Kool et al., 2010; Kurniawan et al., 2010), and preclinical studies may make an important contribution in this regard.
Physical effort
The original assays developed by Salamone and colleagues (Cousins et al., 1994; Salamone et al., 2001) have been highly influential and used a concurrent choice task conducted in an operant chamber, wherein rats choose between pressing a lever a number of times to obtain a more palatable, sweetened food pellet reward, or consuming less palatable laboratory chow that is freely available, with a more recent variant requiring rats to respond on a progressive ratio to obtain the preferred reward (Randall et al., 2012). Critically, this task can be used to delineate whether alterations in choice are due to nonspecific effects on satiety or more evaluative aspects of decision making. A complementary method requires rats to choose between different magnitudes of the same reward. The original T-maze-based task required rats to surmount a scalable barrier to obtain a larger reward in one arm, whereas no barrier is placed in the small reward arm (Salamone et al., 1994). An automated “effort-discounting” task conducted in operant chambers works on a similar principle, in which rats choose between two levers, one delivering a smaller reward after one press and the other a larger reward after a greater number of presses (1–20) that increases over blocks of trials. Advantages of this latter procedure include the ability to assess choice latencies, rates of responding (as an index of response vigor), and being able to equate the relative delay to obtain the two rewards (Floresco et al., 2008; Shafiei et al., 2012; Simon et al., 2013).
Preference for high effort/large reward options on each of the above-mentioned tasks is markedly reduced by manipulations of the NAc, primarily the more lateral “core” region. This includes cell body lesions or inactivations (Hauber and Sommer, 2009; Ghods-Sharifi and Floresco, 2010), neurotoxic lesions of mesoaccumbens DA terminals (Salamone et al., 1991; Salamone et al., 1994), or blockade of either D1 or D2 receptors within the NAc (Nowend et al., 2001). Similarly, lesions or inactivation of the BLA induces a similar shift in preference away from high-cost options (Floresco and Ghods-Sharifi, 2007; Ghods-Sharifi et al., 2009; Ostrander et al., 2011). Notably, these manipulations do not affect preference for larger versus smaller rewards of equal cost. Thus, subcortical circuitry that enables overcoming physical effort costs to obtain more preferred rewards includes the NAc, BLA, and mesoaccumbens dopamine, mirroring their involvement in mediating choice of larger, uncertain rewards. In this regard, it could be argued that decisions involving effort may be related conceptually to those involving uncertainty, in that the decision-maker may be unsure whether the physical challenges may be overcome. This being said, these studies typically involve extensive pretraining, so that subjects are well experienced in overcoming the effort requirements at hand, which would minimize this form of uncertainty. Furthermore, the observations that there are considerable dissociations with respect to the regions of the frontal lobe that mediate decisions involving uncertainty versus effort (as discussed below) suggest that these two types of decisions may be mediated by distinct evaluative processes.
With respect to prefrontal involvement, a number of groups have shown that lesions of the rat dorsal ACC (but not the PrL or OFC) also reduce choice of high effort options on the T-maze barrier task (Walton et al., 2002, 2003; Schweimer and Hauber, 2005; Rudebeck et al., 2006; Floresco and Ghods-Sharifi, 2007; Hauber and Sommer, 2009; Ostrander et al., 2011; Holec et al., 2014). The findings of a seemingly selective involvement of ACC in effort-related choice has spawned a plethora of lesion, neurophysiological, and imaging studies in rats, monkeys, and humans exploring its contribution to these functions. Indeed, direct stimulation of the ACC in humans elicits a feeling of foreboding regarding an impending challenge accompanied by the will to persevere and strive to overcome difficulties, described as if driving into a storm when there is no alternative but to make it to the other side (Parvizi et al., 2013).
However, a deeper inspection of the literature reveals that the involvement of this region in decisions related to physical effort costs only manifests itself under some experimental conditions. For example, similar lesions do not affect performance of the concurrent choice task, where rats chose between pressing a lever for sweetened pellets versus freely available laboratory chow, nor do they affect instrumental responding for food on a progressive ratio (Schweimer and Hauber, 2005). Similarly, ACC lesions did not alter effort-related decisions when rats were required to exert greater physical force by pressing a weighted lever to obtain larger rewards (Holec et al., 2014). Furthermore, in their original study, Walton et al. (2002) noted that that the effects of medial PFC lesions could be remediated if the ratio of reward magnitude was changed from 4:2 to 5:1. Of particular note, a detailed analysis of choice behavior on the T-maze barrier task revealed that, early in a test session, ACC lesion rats were just as likely to initially approach the arm containing the larger reward and the barrier as controls. Yet, upon encountering the barrier, lesioned rats were much more likely to abort their choice and select the easily accessible smaller reward (Holec et al., 2014). This pattern of results suggest that the ACC is not involved in the initial decision to select higher cost/larger reward options, but instead may be more important in overcoming these costs at the point when faced with physical challenge. Thus, even though neurophysiological studies suggest that ACC neural activity encodes information regarding effort cost (Kennerley et al., 2006; Cowen et al., 2012), it is apparent that the ACC does not play a ubiquitous role in effort-related decisions. Rather, it may serve a more specialized function in biasing choice when the discrepancy between the particular costs and benefit associated with different choices is relatively small or when impending effort costs are physically apparent (e.g., a perceptible barrier).
Cognitive effort
The design of the rat cognitive effort task (rCET) takes advantage of a well-validated rodent test of visuospatial attention: the five choice serial reaction time task (5CSRT) (Carli et al., 1983; Robbins, 2002). Rats are trained to attend to an array of five nosepoke holes, and respond in an illuminated aperture when a stimulus light inside is briefly (0.5 s) illuminated to earn reward. The number of premature nosepoke responses provides an index of motor impulsivity, while attentional ability is indicated by the accuracy of target detection. Each trial in the rCET is similar to that in the 5CSRT, with one exception: at the start of each trial, the animal chooses between two levers, and its response determines whether the trial is attentionally demanding, with a stimulus duration of 0.2 s, or much easier, with a stimulus duration of 1.0 s. Successful performance of a hard trial results in 2 pellets, compared with only 1 following an accurate response on an easy trial. Accuracy for hard trials averages ∼65%, whereas for easy trials this improves to 90% (Cocker et al., 2012a).
Notable individual differences emerge when analyzing preference for the hard versus easy trials, such that animals can be classified as either “workers” or “slackers” by whether percentage choice of the harder option falls above or below the group mean (typically 70% choice of the harder option). Critically, this does not depend on animals' ability to perform the harder trials: accuracy is just as good in slackers as in workers. Such observations provide surface-level support for the theory that cognitive ability and mental endeavor are dissociable processes (Naccache et al., 2005). Experiments probing the neurobiological substrates that guide rCET performance also largely point to such a conclusion. As expected from the human literature (and in keeping with some studies of physical effort in animals), ACC inactivations uniformly decreased choice of the more effortful option without impairing accuracy of target detection (Hosking et al., 2014a). However, this manipulation also increased premature responses, indicative of an increase in motor impulsivity. The time taken to make a choice between the easy and hard options was not affected by silencing the ACC, suggesting that the increase in choice of the easier option did not occur more rapidly due to a loss of deliberation or increased reflection impulsivity. On the other hand, PrL inactivation decreased choice of the harder option while also impairing accuracy of target detection (Hosking et al., 2016).
The data described above implicate frontal regions in the evaluation of differentially effortful tasks, yet these findings pose an additional question as to whether perturbations in impulse control or attentional ability induced by frontal inactivations drives the reduced choice of the more cognitively challenging option. That is, are rats shying away from the harder option because they are less likely to succeed at the task, and hence, can choice between differentially effortful cognitive challenges be manipulated independently of their performance? The effects of BLA inactivation on this task would suggest so, in that this manipulation altered choice without affecting accuracy or premature responding, although rats were slower to make a choice (Hosking et al., 2014a). Interestingly, BLA inactivations reduced choice of the more preferred option in both subgroups, increasing choice of the harder option in slackers and the easier option in worker rats. As such, although the BLA may facilitate expression of subjective preferences, other regions may play a more prominent role in determining the most preferred course of action.
In contrast to these more selective regional effects, striatal manipulations have a profound impact on almost every facet of the rCET: inactivations of the DMS decrease choice of the harder option, impair accuracy, increase impulsivity, and slow responding, whereas inactivating the NAc leads to a complete cessation of task performance (Silveira et al., 2016a). Given these pronounced effects, it is somewhat surprising that selective dopaminergic drugs do not affect choice on the rCET, in marked contrast to physical effort-based choice (Hosking et al., 2015). Instead, cholinergic and cannabinoid, but not noradrenergic, signaling appears to be highly influential on decision-making in the rCET (Hosking et al., 2014b; Silveira et al., 2016b). Of particular note is the observation that THC decreased choice of the more demanding option without affecting any other task parameter. Although acute administration of the psychostimulants amphetamine and caffeine decreased high effort choice in workers, these drugs either had the opposite or null effects on slackers' choice, respectively (Cocker et al., 2012a). These observations are also largely contrary to the effects of these compounds in tests of physical effort (e.g., Salamone et al., 2007; Bardgett et al., 2009), although higher doses of amphetamine can reduce choice of the more effortful option in the operant task version (Floresco et al., 2008).
Comparison of neural circuitries
Table 2 provides a simple overview regarding the ways in which the tasks discussed vary in terms of core structural characteristics, and variation in the neural circuitry involved. Some of the most interesting findings from the rCET pertain to individual differences in animals' willingness to select the more cognitively demanding option, and this has yet to be fully explored in physical effort judgments. BLA inactivations decrease choice of the high effort option in both physical and cognitive effort tasks, as long as only “worker” rats are considered in the latter; BLA inactivations have the opposite effect, increasing choice of the harder option, in slacker rats. Baseline choice of the HR option is typically >75% in both the T-maze barrier task, and in the epoch of the effort-discounting task where BLA inactivations had the greatest effect, leading to the suggestion that the increase in choice of the harder option in the cognitively lazy rats may be a result of the difference in baseline choice rather than there being something fundamentally different about the role of the BLA in cognitively effortful decision making. However, close inspection of the data does not support this hypothesis. BLA inactivation still decreased choice of the more physically demanding option even in a subgroup of rats that showed <60% preference for the difficult option at baseline, virtually matching the basal choice profile of slacker rats (Floresco and Ghods-Sharifi, 2007). As such, the role of the BLA in mediating the exertion of cognitive effort may be distinct from its role in promoting physically demanding choice.
Relationship between task features and neurobiology implicated in different decision-making assays involving the assessment of efforta
The frontal cortex is also more widely involved in rCET performance, with PrL, IL, and ACC inactivations causing a shift toward the easier option in all rats, whereas the ACC only plays a critical role in supporting choice of more physically demanding tasks when either a physical barrier is used or when the two options are more difficult to distinguish. When integrating these preclinical findings with those obtained from human subjects, it is interesting to note that ACC stimulation induced a feeling of persevering in the face of adversity (Parvizi et al., 2013). As such, activity within the ACC may influence effort-related choices by enabling the decision-maker to overcome seemingly more daunting mental or physical obstacles, or alternatively, reducing the attractiveness of easily accessible yet inferior rewards, thereby empowering the individual to “stay the course” to obtain more preferred rewards.
The differential involvement of other medial prefrontal regions in mediating performance of the rCET versus physical effort tasks may reflect the nature of the effortful task. Rather than completing a physical challenge, rats are instead performing an attentionally-demanding test requiring accuracy of target detection and behavioral inhibition, processes that are more frontally dependent than scaling a barrier or completing a series of lever-responses. Similarly, the greater involvement of dopamine in physical-effort tasks may reflect the importance of this neurotransmitter in motor control. However, dopaminergic innervation of frontostriatal circuits is clearly implicated in attention and impulsivity (Dalley et al., 2004), yet dopamine antagonists did not alter choice on the rCET. While involvement in the effortful activity itself may recruit neurochemical systems into the decision process, this is clearly not the dominant factor involved. The observation that decisions involving the assessment of physical and cognitive effort cost are mediated by partly dissociable neural and neurochemical systems may offer valuable insight into the nature of individual differences in willingness to engage in physically versus cognitively demanding tasks, and potentially inform any interventions designed to encourage these qualitatively distinct kinds of endeavor.
Summary and future directions
Combining the latest technological advances with behavioral assays of decision making has the potential to increase our knowledge regarding the mechanisms underlying the choice process with unparalleled neural and neurochemical specificity. Current neuroscience research does not support the idea that there is one central “decision-making region,” in which all evaluation and comparative processes occur, but that this process is likely composed of a network of regions that exist in a dynamic network. As per the related field of sensorimotor decision making, activity within the nodes of this network is thought to peak as evidence from internal and external processing accumulates, until selection of option X among competing alternatives comes to dominate (Orsini et al., 2015b). Furthermore, each node and neurotransmitter within the neural circuitry discussed here also contributes to numerous other brain and cognitive processes. Recognizing the duplicity of functions mediated by a region may actually clarify its contribution to choice on a particular paradigm. For example, the conclusion that damage to the OFC results in risky choice, as measured by the original version of the IGT, was refined in light of data demonstrating that this choice deficit could be attributed to a deficit in reversal learning (Fellows and Farah, 2005).
Although most of the data collected to date were generated from male rats, it will be important to validate these behavioral assays in female rats and determine whether performance is moderated by sex. Indeed, whereas female rats make fewer advantageous choices compared with males in the RIGT (van den Bos et al., 2012), recent findings suggest that female rats are more risk-averse when performing the RDT (Orsini et al., 2016). The authors of the latter study discussed the importance of ruling out potential confounds related to task engagement before concluding there could be a sex-dependent difference in cognition. For example, with respect to the RDT, it was critical to control for shock reactivity and motivation for food reward, both of which may be altered in female rats given that they are typically smaller and lighter than males. It is increasingly recognized that a range of contextual factors, including trial rate, motivational state, the “framing” of decisions (gain vs loss), and even the risk preference of probands, can also significantly impact risky choice, and likewise may be important to consider when trying to translate behavioral methods across species (Blanchard et al., 2014; Blanchard and Hayden, 2015; Cocker and Winstanley, 2015; Heilbronner and Hayden, 2016; Suzuki et al., 2016). While we have focused on rodent studies, the importance of nonhuman primate research in delineating key brain mechanisms underlying choice, such as parsing action-value versus stimulus-value computations, cannot be understated (Samejima et al., 2005; Lau and Glimcher, 2008; Morris et al., 2014). As rat decision-making research increases in complexity, its synthesis with nonhuman primate decision making research merits further consideration and may likewise highlight important areas for future research and methodological innovation (e.g., Redish, 2016).
In conclusion, the data reviewed here demonstrate that the cognitive processes taxed by different decision-making tasks can vary significantly, and this impacts the neural circuits that are recruited in guiding the direction of behavior. It is important to keep this in mind when selecting a behavioral assay that will maximize the chances of fulfilling the goals of the experiment. At the simplest level, this could mean choosing a task that is known to be sensitive to manipulations of the area or neurotransmitter system under investigation. For example, if the experimental aim is to delineate the role of dopaminergic projections to the anterior cingulate cortex in mediating decisions involving differential amounts of effort, it would be optimal to select the maze-based test of physical effort, instead of the rCET or concurrent-choice task. If the aim is to look at the role played by subjective, rather than objective, reward value in risky choice, consider choosing a task in which the options are matched for expected utility. Critically, if task choice must be determined by either available behavioral testing equipment or other logistical challenges, being aware of the limitations and potential confounds associated with any particular behavioral assay will enable a more informed discussion of the results, and therefore a more insightful and informative contribution to the literature. As such, deciding which decision-making task is most appropriate in addressing an experimental question may therefore be one of the most critical decisions of all.
Footnotes
This work was supported by Canadian Institutes for Health Research operating grants to C.A.W. and S.B.F. and Canadian Natural Sciences and Engineering Council Discovery Grants to C.A.W. and S.B.F. We thank Dr. Caitlin Orsini for providing data used to generate one of the figures in this article.
C.A.W. received salary support through the Michael Smith Foundation for Health Research and the Canadian Institutes of Health Research New Investigator Award program, and has consulted for Shire Pharmaceuticals on an unrelated matter. The other author declares no competing financial interests.
- Correspondence should be addressed to Dr. Catharine A. Winstanley, Department of Psychology, University of British Columbia, Djavad Mowafaghian Centre for Brain Health, 2215 Westbrook Mall, Vancouver, British Columbia V6T 1Z3, Canada. cwinstanley{at}psych.ubc.ca
