Dopamine and Norepinephrine Differentially Mediate the Exploration–Exploitation Tradeoff

Modulation of beta-noradrenergic receptor activity bidirectionally changed sensitivity to outcome. The effect of beta-noradrenergic modulation on win-stay and lose-shift was also influenced by sex. A, The noradrenergic drug administration schedule and drug dosage used to modulate noradrenergic receptor activity. Left, A beta-noradrenergic receptor agonist isoproterenol (0.3 mg/kg) and, right, a beta-noradrenergic receptor antagonist propranolol (5 mg/kg) were systemically administered to respectively up- and downregulate norepinephrine activity. A total of 0.9% saline was used as the vehicle control. Control and drug sessions were interleaved and repeated for six sessions each. B, Left, Average probability of obtaining reward compared with the chance level probability of reward for each animal under isoproterenol (ISP) and vehicle (dot). Error bars depict mean ± SEM across sessions for each animal. Right, Probability of obtaining reward over chance level across ISP (dark brown) and vehicle (gray). C, Average response time under ISP and vehicle. ISP significantly increased response time. D, Left, Average probability of obtaining reward compared with the chance level probability of reward for each animal under propranolol (PRO) and vehicle (dot). Error bars depict mean ± SEM across sessions for each animal. Right, Probability of obtaining reward over chance level across PRO (light brown) and vehicle (gray). E, Average response time under PRO and vehicle. PRO significantly decreased response time. F, Average probability of win-stay under ISP and vehicle. G, Average probability of lose-shift under ISP and vehicle. Inset, Probability of lose-shift across treatments across sexes. Decrease in lose-shift under ISP was primarily driven by changes in females (interaction term). H, Average probability of win-stay under PRO and vehicle. Inset, Probability of win-stay across treatments across sexes. Increase in win-stay under PRO was primarily driven by changes in males (interaction term). I, Average probability of lose-shift under PRO and vehicle. Inset, Probability of lose-shift across treatments across sexes. Decrease in lose-shift under PRO was primarily driven by changes in males (interaction term). J, Average outcome sensitivity across sessions under ISP and vehicle. ISP decreased outcome sensitivity. K, Outcome sensitivity across sexes under ISP and vehicle. L, Average outcome sensitivity across sessions under PRO and vehicle. PRO increased outcome sensitivity. M, Outcome sensitivity across sexes under PRO and vehicle. * indicates p < 0.05. Graphs depict mean ± SEM across animals unless specified otherwise.

Up- and downregulation of dopamine activity had bidirectional effects on the level of exploration. The neuromodulatory effect on exploration across sexes is demonstrated in Extended Data Figure 3-1. A, Left, Structure of a HMM that modeled exploration and exploitation as latent goal states underlying observed choices. This model incorporates two exploit states for two choices and one explore state where choices were uniformly distributed across options. Right, Reward probabilities, choices, and HMM labels for an example session of 300 trials for a given mouse. Shaded areas demonstrated HMM-labeled exploratory choices. B, Probability of choice as a function of value differences between choices for exploratory and exploitative states under vehicle (left) and flupenthixol (FLU; right). C, D, Difference in response time between explore and exploit choices under vehicle and flupenthixol (FLU). E, Probability of choice as a function of value differences between choices for exploratory and exploitative states under vehicle (left) and apomorphine (APO) (right). F, G, Difference in response time between explore and exploit choices under vehicle and apomorphine (APO). H, Distribution of the percentage of HMM-labeled exploratory choices under flupenthixol (FLU)/vehicle (top) and apomorphine (APO)/vehicle (bottom). I, Probability of exploration for dopamine antagonist (FLU) and agonist (APO), normalized by their vehicle control. Decreasing dopamine activity increased exploration and increasing dopamine activity decreased exploration. J, Probability of exploration by session with vehicle and drug session interleaved (flupenthixol, top; apomorphine, bottom). Drug administration sessions are in colored shades. * indicates p < 0.05. Graphs depict mean ± SEM across animals.

Downregulating norepinephrine activity influenced exploration but the effect was modulated by sex. A, Probability of choice as a function of value differences between choices for exploratory and exploitative states under vehicle (left) and propranolol (PRO) (right). B, C, Difference in response time between explore and exploit choices under vehicle and propranolol (PRO). D, Probability of choice as a function of value differences between choices for exploratory and exploitative states under vehicle (left) and isoproterenol (ISP) (right). E, F, Difference in response time between explore and exploit choices under vehicle and isoproterenol (ISP). G, Distribution of the percentage of HMM-labeled exploratory choices under propranolol (PRO)/vehicle (top) and under isoproterenol (ISP)/vehicle (bottom). H, Probability of exploration for beta-noradrenergic antagonist (PRO) and agonist (ISP), normalized by their vehicle control. Propranolol decreased the level of exploration. I, Probability of exploration by session with vehicle and drug session interleaved. Drug administration sessions are in colored shades. Top, Propranolol (PRO) condition; bottom, isoproterenol (ISP) condition. J, Probability of exploration under PRO and vehicle across sexes. The effect of PRO on exploration was primarily driven by a significant decrease in exploration under PRO in males. K, Probability of exploration under ISP and vehicle across sexes. ISP significantly decreased exploration in females but not males. * indicates p < 0.05. Graphs depict mean ± SEM across animals.

Up- and downregulating dopamine receptor activity bidirectionally modulated decision noise parameter in a reinforcement learning (RL) model. The correlation between HMM parameter and RL model parameters are shown in Extended Data Figure 5-1. Choice kernel updating rates from RL model were not affected by modulation and are shown in Extended Data Figure 5-2. A, A diagram of RL parameters that capture learning rate (α), asymmetric learning (γ), choice bias (α_c), and inverse temperature/decision noise (β). The RL models tested included one or more of the above parameters. B, Model likelihood for six RL models using log likelihood for vehicle and apomorphine (APO) condition (top) and vehicle and flupenthixol (FLU) condition (bottom). The three-parameter RLCK model was the best fit and most parsimonious model for behaviors. C, Model agreement for the best fit model (RLCK), which measures the probability of the model predicting the actual choices of animals for vehicle and APO (top) and vehicle and FLU condition (bottom). D, E, Animal choices (purple) and simulation (green) from the best fit model (RLCK) for the same animal under vehicle/APO and vehicle/FLU. The gray line represents the reward probability of the left choice. F, Average inverse temperature (β) across APO and vehicle. Inset, Inverse temperature across conditions after removing two outliers. G, Average inverse temperature (β) across FLU and vehicle. Inset, Inverse temperature across conditions after removing two outliers. H, Agonizing (APO) and antagonizing (FLU) dopamine activity revealed bidirectional effect on inverse temperature. I, Average learning rate (α) across FLU and vehicle. Flupenthixol increased learning rate. J, Average learning rate (α) across APO and vehicle. * indicates p < 0.05. Graphs depict mean ± SEM across animals.