Abstract
Economic theories of decision making are based on the principle of utility maximization, and reinforcement-learning theory provides computational algorithms that can be used to estimate the overall reward expected from alternative choices. These formal models not only account for a large range of behavioral observations in human and animal decision makers, but also provide useful tools for investigating the neural basis of decision making. Nevertheless, in reality, decision makers must combine different types of information about the costs and benefits associated with each available option, such as the quality and quantity of expected reward and required work. In this article, we put forward the hypothesis that different subdivisions of the primate frontal cortex may be specialized to focus on different aspects of dynamic decision-making processes. In this hypothesis, the lateral prefrontal cortex is primarily involved in maintaining the state representation necessary to identify optimal actions in a given environment. In contrast, the orbitofrontal cortex and the anterior cingulate cortex might be primarily involved in encoding and updating the utilities associated with different sensory stimuli and alternative actions, respectively. These cortical areas are also likely to contribute to decision making in a social context.
Decision making refers to the process by which the animal chooses a particular response based on its evaluation of the potential costs and benefits associated with alternative actions. Economic theories suggest that this can be based on a set of numerical scores, referred to as utilities, associated with alternative choices (von Neumann and Morgenstern, 1944). This process of decision making is dynamic and continually adjusted to reflect the animal's experience. As described by the reinforcement learning theory (Sutton and Barto, 1998), any discrepancy between the outcome expected by the animal and the actual outcome from its chosen action influences the animal's future decision-making strategies. Throughout this interactive process, the animal also needs to take into consideration the possibility that the mapping between its chosen actions and their outcomes might change depending on the environmental context. Furthermore, the desirability of a particular outcome is dependent on the animal's current biological needs.
Given the complex nature of the decision-making process and the need to integrate information about the animal's external environment and internal milieu, the primate frontal cortex, with its close anatomical connections with high-order sensorimotor cortical areas (Petrides and Pandya, 1999, 2002) and subcortical limbic system (Öngür and Price, 2000), is likely to be involved in various aspects of decision making (Fig. 1). Recently, fertile interactions between the disciplines of economics, reinforcement learning theory, and animal learning psychology have influenced the development of new hypotheses regarding the nature of decision making and these are increasingly being tested in neurobiological studies of the primate frontal cortex. Many findings from these studies have already begun to provide a more complete and coherent perspective on multiple functions of the frontal cortex (Fellows, 2004; Wallis, 2007). Although this review focuses on the role of primate frontal cortex in decision making, it should be noted that decision making is a process distributed in numerous brain areas (Sugrue et al., 2005; Daw and Doya, 2006), and that frontal cortex is likely to be involved in functions other than decision making (Miller and Cohen, 2001; Amodio and Frith, 2006).
A, B, Medial (A) and lateral (B) view of the rhesus monkey's brain showing the lateral prefrontal cortex, the orbitofrontal cortex, and the anterior cingulate cortex.
Lateral prefrontal cortex and state representation
An optimal action, namely an action that is most likely to yield the most desirable outcome, often changes according to the state of the animal's environment. Therefore, if a sensory stimulus informs the animal of a change in the state of its environment, the animal needs to store such information until it produces an appropriate action or until this information can be combined with another stimulus to determine the new state of the environment. For example, a classic working memory task requires the animal to remember a particular state of the environment, as often indicated by a brief sensory stimulus. The animal is rewarded only when it produces an action corresponding to this state after some delay. Therefore, neural activity related to the animal's working memory, as commonly seen in the lateral prefrontal cortex (Fuster and Alexander, 1971; Funahashi et al., 1989), can be characterized as representing the state of the environment. Similarly, many other types of signals found in the prefrontal cortex, such as the abstract properties of a sensory stimulus (Freedman et al., 2001; Nieder et al., 2002) and the rules of the task that specify how to determine a correct action based on sensory stimuli (Sakagami and Niki, 1994; Wise and Murray, 2000; Hoshi et al., 2000; Wallis et al., 2001) can be considered as encoding the state of the environment. Neurons in the lateral prefrontal cortex are also involved in accumulating sensory evidence (Kim and Shadlen, 1999) or transforming perceptual information to motor outputs (Takeda and Funahashi, 2004). In short, neurons in the lateral prefrontal cortex encode specific states of the environment necessary to determine an optimal action unequivocally.
More recently, many investigators have manipulated the motivational context during working memory tasks. In these tasks, the animal is informed of the correct action and its immediate outcome. The results from these experiments have shown that so-called delay activity is often modulated by the outcomes expected from the correct and incorrect actions (Watanabe, 1996; Leon and Shadlen, 1999; Amemori and Sawaguchi, 2006; Kobayashi et al., 2006), indicating that the lateral prefrontal cortex might encode the state of the environment and the properties of the expected reward conjunctively (Watanabe and Sakagami, 2007). In the lateral prefrontal cortex, increasing the value of the expected reward often increased the reliability of the state representation (Kobayashi et al. 2002). Therefore, in addition to encoding the state of the environment, the lateral prefrontal cortex might represent the utilities or values associated with various states of the environment.
Orbitofrontal cortex and value representation
Lesions which encompass the orbitofrontal cortex in human patients often impair their ability to adjust decision making strategies when the previously successful choices are no longer advantageous (Bechara et al., 2000; Rolls, 2000) and to make preference judgments consistently (Fellows and Farah, 2007). Impaired decision making abilities in such psychiatric conditions as substance abuse and frontotemporal dementia may also result from the dysfunction of the orbitofrontal cortex (Rahman et al., 2001). Similarly, orbitofrontal lesions in monkeys impair the ability to modify behavior when the outcomes of decisions change dynamically (Izquierdo et al., 2004) and to assign appropriate values to different objects in the environment (Izquierdo et al., 2005). According to one novel proposal derived from reinforcement learning, this faculty may depend on the ability to represent the statistical interdependence between the values of alternative options (Hampton et al., 2006). Knowledge of such interdependence might allow the decision makers to switch to a better choice immediately, as soon as they experience the reduction of values from a particular choice.
Signals related to the expected reward have been also identified in the primate orbitofrontal cortex. However, in contrast to the neurons in the lateral prefrontal cortex, neurons in the orbitofrontal cortex seldom encode different states of the environment and optimal actions associated with them. Instead, their activity is largely determined by the expected outcome (Tremblay and Schultz, 2000; Wallis and Miller, 2003; Roesch and Olson, 2004), even when the outcome is determined by the animal's own choice (Padoa-Schioppa and Assad, 2006). These results demonstrate that it is possible to design experiments that determine how frontal areas differ in their processing of signals related to the states of the environment and the outcomes that can be expected from those states. Although it is widely assumed that various frontal areas make distinct contributions to decision making and cognition, few studies have directly tested this assumption and compared the contributions of different frontal areas in the same task. Formal accounts of decision making suggest a number of critical axes along which to compare frontal areas in future experiments.
Anterior cingulate cortex and outcome evaluation
Several lines of evidence suggest that the primate anterior cingulate cortex might play a key role in choosing appropriate actions when the environment is uncertain or dynamic. First, many single-neuron recording studies have found that the neurons in the anterior cingulate cortex modulate their activity according to the outcome of the animal's action (Niki and Watanabe, 1979; Ito et al., 2003; Matsumoto et al., 2003, 2007). Second, this outcome-related activity might be required for the animals to update its decision-making strategies after committing an error (Shima and Tanji, 1998; Procyk et al., 2000). Third, a lesion in the anterior cingulate cortex impairs the animal's ability to integrate signals related to the outcomes of the animal's previous choices to make optimal decisions (Kennerley et al., 2006). Finally, the results from rodent studies suggested that the anterior cingulate cortex may also be involved in combining information about the costs and benefits associated with alternative actions (Rudebeck et al., 2006b). Combined with the anatomical finding that much of the primate anterior cingulate cortex projects to cortical areas with motor functions (Dum and Strick, 1991), including the presupplementary motor area and supplementary eye field (Wang et al., 2001; Luppino et al., 2003), these results suggest that the utilities associated with different actions, referred to as action value functions in reinforcement learning theory, may be encoded and updated in the anterior cingulate cortex (Rushworth et al., 2007).
Frontal cortex and social interactions
During social interaction, the outcome of an action can change dynamically depending on the actions of other decision makers in a group. Although choice behaviors of humans and animal in social settings often deviate from the optimal strategies described by game theory, such deviations can be often accounted for by reinforcement learning algorithms (Camerer, 2003; Lee et al., 2004). Given that the different areas of primate frontal cortex are intimately involved in reinforcement learning, this suggests that they might also play an important role in socially interactive decision making. For example, lesions in the orbitofrontal cortex induce a loss of social dominance with increased aversion and reduced aggression in threatening situations (Butter and Snyder, 1972).
During a computer-simulated zero-sum game, the activity in the lateral prefrontal cortex encode the signals related to the animal's previous choices and their outcomes in multiple trials (Barraclough et al., 2004; Seo et al., 2007), and therefore might provide an appropriate context in which the animal's decision-making strategies can be updated during social interaction. For more complex social interactions, such as cooperation, the process of identifying successful decision-making strategies might depend on some cortical areas specialized for processing socially meaningful stimuli and thereby inferring actions expected from other animals. For example, the direction of gaze in other animals might provide information about their probable actions during social interactions (Deaner et al., 2005; Flombaum and Santos, 2005). In fact, a lesion in the primate anterior cingulate gyrus causes the animals to become less interested in gathering information from social stimuli, such as faces (Rudebeck et al., 2006a), implicating an important role for this frontal area, perhaps in conjunction with the amygdala (Bachevalier and Loveland, 2006), in social perception.
Summary and conclusion
Guided by the detailed accounts of connectivity among different subdivisions of the primate frontal cortex and by formal theories of decision making, a large number of recent lesion and physiological studies have begun to explore the contribution of the frontal cortex to the making and assessment of choices. Overall, the results from these studies suggest that the lateral, medial, and ventral aspects of the prefrontal cortex might be specialized for representing the relevant states of the animal's environment, updating the desirability of alternative actions, and predicting the values of rewards expected from different objects in the animal's environment, respectively. How these brain areas interact with one another and whether they are further differentiated to improve the animal's ability to behave optimally during complex social interaction remains an important topic for future research.
Footnotes
-
This work was supported by the National Institutes of Health (D.L.), the Medical Research Council (M.F.S.R., M.E.W.), the Royal Society (M.F.S.R.), Human Frontier Science Program (M.S.), and a Grant-in-Aid for Scientific Research on Priority Areas from the Ministry of Education, Culture, Sports, Science, and Technology of Japan (M.W., M.S.).
- Correspondence should be addressed to Dr. Daeyeol Lee, Department of Neurobiology, Yale University School of Medicine, 333 Cedar Street, Sterling Hall of Medicine C303, New Haven, CT 06510. daeyeol.lee{at}yale.edu
References
