Distinct Neurobehavioral Mechanisms for Expectancy Violation and Value Updating

Decision making has utmost importance in our daily social lives and has been studied across varied fields, such as psychology, neuroscience, economics, management, and machine learning. One of the fundamental aspects of decision making is that the actions are chosen based on the expected value of their outcome. In most theoretical accounts of decision making, value is assumed to be updated primarily based on the discrepancy between the expected and the actual outcome of the behavior (Dayan and Niv, 2008). However, in the real world, the process of updating is much more flexible. It depends on, among other things, the nature of uncertainty. Uncertainty can refer to risk, which is present when there are multiple possible outcomes whose probability distribution is well defined. Uncertainty can also refer to ambiguity when the probabilities of the possible outcomes are unknown or not estimable (Ellsberg, 1961). The two conditions may have different implications for the process of value updating. For instance, when you do not know whether a coin is fair or not, example of an ambiguous condition, 10 heads in a row would make you suspect that it is not, and it would make sense to predict that the next toss will result in a head. But when tossing a fair coin, the odds of getting a head do not change even after 10 heads in a row, so logically your expectations about the result of the coin should not change. In a recent study, Kobayashi and Hsu (2017) delved further into the neurobehavioral distinction between expectancy violation and value updating using behavioral and neuroimaging techniques.

Kobayashi and Hsu (2017) first undertook a behavioral experiment to test the dissociation between expectancy violation and value updating. Subjects participated in a gambling task presented on the computer screen. The task consists of an urn containing balls of up to three different colors. Participants were told before each trial how many balls were in the urn, and how many of those balls had one of the colors (“risky color”). However, how many of the remaining balls had each of the remaining two colors (“ambiguous colors”) was unknown. At the beginning of each trial, a predetermined winning color was shown on the screen. Before the actual gamble (“resolution draw”), a ball was randomly drawn from the urn, revealed its color (“observed draw”), and was returned back. The subjective values of the gambles were assessed both before and after the observed draws as willingness to sell (WTS; i.e., the minimum amount of money for which subjects would be willing to exchange his/her gamble). Because the draw's color is probabilistic, each observed draw is associated with some level of expectancy violation. However, a χ² test of independence showed that the value was updated only when an ambiguous color draw was observed in a gamble with an ambiguous winning color but not in any other condition, thus supporting the dissociation between expectancy violation and value updating.

Kobayashi and Hsu (2017) constructed a model of adaptive decision making that predicts a three-stage process from expectancy violation to belief updating and finally to value updating. They illustrated their model with the same gambling task used in the behavioral experiment. The model proposed that participants must have an internal model reflecting their belief on the urn's content. If a risky color ball is drawn, there is no change in the internal model; hence, the belief remains unaffected. However, the draw of one of the ambiguous color would bias the belief toward this observed draw. For example, when an urn contains four balls, two balls in red and two in either yellow or green, a red color draw adds no new information to what is already known about the urn content. In contrast, a yellow color draw ensures that there is at least one yellow ball and leads to an update of the internal model. However, an ambiguous color draw can lead to value update only when the winning color is also ambiguous. When the winning color is risky, an ambiguous color draw, even though updates the belief toward it, has no effect on the value. Thus, in the given example, the observed yellow draw should not affect the chance of winning and thus the WTS value if the winning color is red but does affect it when the winning color is yellow (or green). To test their model, Kobayashi and Hsu (2017) fitted it to the WTS values obtained from the behavioral experiment in mixed-effects modeling and found that the subjective values were consistent with those predicted by the model. The detailed mathematical formulation of the model is omitted from this commentary.

Kobayashi and Hsu (2017) used the same gambling paradigm in an fMRI experiment to understand the neural underpinnings of the elicited behavior. Unlike the usual “on-off” fMRI paradigm where a condition is either present or absent in each trial, here three experimental conditions (i.e., expectancy violation, belief updating, and value updating) can happen at a variety of levels during the experiment depending upon the winning color and the color of the observed draw in a particular trial. The authors entered the trialwise values denoting three conditions as defined by their quantitative model in a GLM analysis of fMRI data. The resultant analysis found BOLD signal in bilateral anterior insula (AI) to be specifically modulated by variation in levels of expectancy violation but not by the other two variables. Furthermore, regions in posterior middle frontal gyrus, superior frontal sulcus, intraparietal sulcus, and a cluster in precuneus showed unique sensitivity toward belief updating, and ventromedial prefrontal cortex (vmPFC), anterior and middle cingulate cortices, and left superior temporal gyrus toward value updating. To further support the anatomical dissociation, they performed a region-of-interest (ROI) analysis of fMRI data that found expectancy violation ROIs to be correlated solely with expectancy violation but not with the other two variables. Similar observations were made for belief updating and value updating, thus further supporting the dissociation.

Last, the authors tested their hypothesis using dynamic causal modeling analysis. They constructed and compared three families of dynamic causal modeling, each consisting of expectancy violation, belief updating, and value updating ROIs as nodes, with bidirectional connections between each pair. The nodes were selected based on GLM analysis. In the first family of models, connections between belief updating ROIs and value updating ROIs were modulated based on the type of the gamble (ambiguous or risky). In the second model, the type of the gamble modulated the connections from the expectancy violation ROIs to value updating ROIs. Family 3 did not allow for any modulations. Bayesian model comparison found the observed fMRI data to be best fit by the first family of models, consequently supporting the idea that it is belief updating and not mere expectancy violation that drives value updating.

The significance of the Kobayashi and Hsu (2017) study lies in assigning distinct roles to brain regions within a coherent framework of adaptive decision making. The study, for example, proposes different functional roles for AI and ACC, two frequently coactivated brain regions in functional neuroimaging research (Menon, 2015). The proposed role of AI is in accord with its suggested role (Sridharan et al., 2008) in mediating the interaction between executive control network and default mode network, the two large-scale neural networks that have been implicated with externally and internally oriented cognition, respectively. To detect expectancy violation, the brain needs to compare external observations with the internally generated expectancy; AI is a suitable candidate region where this function can be performed. This conceptualization is in line with the hypoactivation of AI in autism spectrum disorder resulting in defective evaluation of emotional salience (Uddin and Menon, 2009). In contrast, the value updating role of ACC conforms to its suggested role in “cognitive control.” Cognitive control refers to, among other things, the ability to rationally ignore prepotent but behaviorally irrelevant information in favor of attending to information that is relevant for the decision in question (McGovern and Sheth, 2017). This function may be behind the deficit in the response shifting in autistic patients (Shafritz et al., 2008), or the misspecification of cognitive control signals in obsessive-compulsive disorder (McGovern and Sheth, 2017), as altered functioning of ACC has been well established in both of these conditions (Yücel et al., 2003; Friedman et al., 2017). Likewise, the finding that vmPFC and superior temporal cortex being significantly activated in OCD patients (Mataix-Cols et al., 2013) can be explained by their proposed role in value updating.

The study by Kobayashi and Hsu (2017) does not elaborate on the functional specialization within a single group of brain regions. For instance, understanding the distinct roles played by ACC and vmPFC in value updating is an interesting open question for future research. Some possible hypotheses, however, can be derived drawing upon extant literature. The plausible role of ACC in cognitive control has already been discussed. vmPFC responses, in contrast, have been postulated to more directly represent the subjective value of the chosen behavior (Rushworth et al., 2011). vmPFC is among the brain regions most consistently reported to show abnormal activity in anxiety disorders (Myers-Schulz and Koenigs, 2012), a condition characterized by a subjective misrepresentation of the value of an action.

One should, however, exercise caution while ascribing specific function to brain regions solely based on BOLD signals. In some cases, it may merely reflect the input to one set of region or output from another region that is probably more directly involved in the function. Moreover, an equally appealing alternative interpretation of the observations could be made based on the concept of prediction error uncertainty (Bossaerts, 2010). The idea is that, instead of having dissociable areas or circuit for expectancy violation and value updating, brain regions may maintain variance around the predictions, such that errors are not recognized unless they are larger than some SEM. Thus, the apparent absence of activation in “value updating ROIs” in response to expectancy violation could be due to the uncertainty being not large enough during expectancy violation.

The model proposed by Kobayashi and Hsu (2017) presupposes a fully rational decision-making process. However, several studies from the broad field of psychology and neuroscience have suggested that automatic, instinctive processes outperform rational processes at solving complex tasks (Dijksterhuis et al., 2006). In a seminal work, Bechara et al. (1997) showed that processes driven by instinctual and affective drives, such as stress associated with a loss, biased the decision in a gambling task much earlier than when the declarative process of value updating became accessible to the subject. Future studies could explore the possibility of the incorporation of affective and instinctual elements into the Kobayashi and Hsu (2017) model. In that regard, the projections from the amygdala, hypothalamus, and other subcortical regions to vmPFC, as a part of the frontosubcortical circuit (Rubia, 2013), may be explored as an alternative nondeclarative pathway for value updating, bypassing the declarative belief updating stage.

Overall, Kobayashi and Hsu (2017) proposed and empirically demonstrated behavioral and anatomical dissociation between expectancy violation, belief updating, and value updating. This dissociation may prove helpful in understanding the neurobiology of decision making as well as the neuropathological bases of several clinical conditions.