A Comparison of Rapid Rule-Learning Strategies in Humans and Monkeys

Monkeys are slower rule learners than humans

We compared the ability of monkeys and humans to rapidly adapt to changes in task contingencies in a rule-switching task inspired by the WCST. On each trial, subjects were presented with four objects and received feedback upon selecting one of them (Fig. 1a, middle). Each object was composed of one unique feature from each of three dimensions—visual pattern, shape, and color (Fig. 1a, top). Each of the 12 possible features appeared in one of the objects on each trial, but object compositions changed across trials. On a given block of trials, subjects received positive feedback (monkeys, food reward; humans, the word “CORRECT” displayed on the screen) for selecting the object which contained the feature defined by the current hidden rule (e.g., red) and negative feedback (monkeys, timeout; humans, the word “INCORRECT” displayed on the screen) otherwise. After subjects demonstrated that they had learned the current rule by reaching criterion performance on the current block, a new block was initiated through an uncued switch to a randomly chosen new rule (Fig. 1a, bottom). Similarities and differences in task parameters and learning criteria across the monkey and two human datasets are outlined in Extended Data Tables 1-1 and 1-2. The second human dataset was collected to match the task parameters in the monkey dataset, enabling a better comparison between species (see Materials and Methods).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Monkeys rapidly learn rules in a rule-switching task but are slower than humans. a, Rule-switching task structure. Each trial is composed of fixation, decision, response, feedback, and ITI epochs. After fixation, the subject is presented with four objects that are pseudorandomly composed of three features—a pattern, shape, and color. The features composing each object are mutually exclusive with respect to other objects. Each block of continuous trials is governed by a rule (1 of the 12 features). The subject receives positive feedback only for choosing the object with that feature. The identity of the rule is hidden and must be discovered. An uncued rule switch to a random new feature occurs when the subject demonstrates they have learned the current rule. b, Distribution of trials-to-learning criteria in four monkey subjects (brown) and 10 human subjects (5 subjects, Dataset 1, green; 5 subjects, Dataset 2, blue). All subjects rapidly learn the rule, but on average, monkeys are over four times slower than humans. c, Decision process for the WSLS learning strategy in two-armed bandit problems. The decision to choose an arm can be in one of two states: persist when it is chosen and avoid when it is not. The decision to choose an arm stays in the persist state as long as positive feedback is received (win–stay) and switches to the avoid state otherwise. It then stays in the avoid state as long as positive feedback is received and switches to the persist state when negative feedback is received (lose–shift). See Extended Data Figure 1-1 for more details.

Remarkably, well-trained monkeys learned new rules within only tens of trials. Yet, they were over three times slower than humans (Fig. 1b; monkeys, 27.84 ± 2.92 trials (mean ± SEM); Human Dataset 1, 5.98 ± 0.52 trials; Human Dataset 2, 6.47 ± 0.19 trials). This learning deficit in monkeys was significant in comparison with that in subjects in Human Dataset 2 (bootstrap test with t statistic, p < 0.01), as well as in comparison with subjects in Human Dataset 1 following a correction for the difference in the learning criterion between the two datasets (Extended Data Fig. 1-1; monkeys, 20.61 ± 1.52 trials; Human Dataset 1, 5.98 ± 0.52 trials; bootstrap test with t statistic; p < 0.005). We then sought to explain the interspecies computational differences that produce this rule learning slowing in monkeys. Specifically, we focused on inferring individuals’ rule-learning strategies from behavior and identifying the species differences that contribute to the learning speed difference. One possible strategy, WSLS, is widely reported during rule learning in many species, particularly in the two-armed bandit problem where the identity of the more rewarding arm must be learned and can change over trials. Here, one of the arms is repeatedly chosen as long as this produces positive feedback (win–stay). When negative feedback is received, the other arm is chosen on the next trial (lose–shift). This strategy can be cast as a decision process comprised of two behavioral states—persist and avoid—where the choice of the currently rewarded arm is in the persist state and transitions to the persist or avoid states subject to positive or negative feedback, respectively, while the choice of the other arm is in the avoid state and transitions to the persist state when that arm was not chosen on the previous trial and negative feedback was received (Fig. 1c).

The WSLS strategy is computationally efficient and requires that the subject attend to and maintain only a single arm's identity in working memory. By replacing arm identity with feature identity, the approach is readily adapted to solve problems in our rule-switching task and always finds the rule. However, feedback is equally informative about all three features in the chosen object, not just the attended one. Due to this neglect of information about unattended features, a simulated WSLS agent learns rules much slower than optimal: indeed humans learn more rapidly (Extended Data Fig. 1-1; WSLS agent mean, 13.31 trials; SD, 12.85 trials; Human Dataset 1, 5.98 ± 0.52 trials). This underscores a trade-off between computational and information efficiency in multidimensional environments. Simultaneously maintaining and updating beliefs about multiple features is more information efficient but increases computational complexity and working-memory demands. In contrast, attending to a single feature at a time is computationally simpler but inefficient in its integration of trial outcomes. In the following sections, we address how the two species solve this trade-off.

Dynamic model uncovers hidden states during rule learning

Prior cognitive model comparisons of human behavior in rule-switching tasks provide evidence for rule-learning strategies wherein subjects selectively attend to and learn about individual features or dimensions, rather than feature configurations (i.e., objects; Bishara et al., 2010; Wilson and Niv, 2012; Niv et al., 2015). It is argued that such a mental representation of stimuli in terms of features resolves the curse-of-dimensionality which impairs learning efficiency in high-dimensional environments. For example, it is more efficient to learn the value of 12 features than the dozens of objects they can be combined into. Drawing on these findings, we developed a behavioral model to predict the probability of a subject choosing individual features given their choices and outcomes on previous trials. However, in contrast to earlier work, our model does not postulate a specific internal belief structure and update rule, thus making fewer assumptions regarding the learning algorithm underlying a subject's behavior. Instead, it aims to discover in an unbiased manner how the decision-making process evolves as a function of feedback. Recently, this approach has been successful at revealing previously unobserved behavioral states underlying human, monkey, rodent, and fruit fly decision-making (Ebitz et al., 2018; Calhoun et al., 2019; Roy et al., 2021; Bolkan et al., 2022).

For each feature, we model whether the feature is chosen or not (denoted as c) as a function of past choices and their outcomes (h) via a Bernoulli GLM (Fig. 2a; see Materials and Methods). The choice outcome on an earlier trial is represented by a four-dimensional binary vector where the dimensions represent whether positive feedback was received after choosing the feature on the trial (C⁺), negative feedback was received after choosing the feature (C⁻), positive feedback was received after not choosing the feature on the trial (NC⁺), or negative feedback was received after not choosing the feature (NC⁻). This allows us to assess separately how the present choice depends on past choice outcomes both when that feature was chosen (direct feedback) and when it was not (indirect feedback). Furthermore, the model permits dynamic changes in how past choices and outcomes are transformed into a present choice via hidden states (s). A feature's associated hidden state also undergoes a transition at the end of each trial depending on past choices and outcomes, which may reflect updates to the feature's value based on past choice outcomes, or a change in the level of attention to the feature or even a shift in strategy (i.e., how a feature's history is weighted in determining its choice). Note that while the model permits these possibilities and others, it does not prescribe the nature and function of the states. Rather, the states and their dynamics emerge upon fitting the model to behavioral data. These hidden state dynamics are modeled as an input-dependent or IOHMM (Bengio and Frasconi, 1994).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

IOHMM-GLM model fits uncover dynamic changes in choice behavior during rule learning. a, IOHMM-GLM model architecture fit to data. The model predicts the choice of a feature c at each trial t from the choice–outcome trial history h via a GLM. Hidden states s determine the GLM's parameters. These states can transition at each trial also based on the choice–outcome trial history via a separate state transition GLM. b, Model fit log-likelihoods on training and test datasets for each human (Dataset 1, green; Dataset 2, blue) and monkey (brown) subject in models with varying numbers of states and that use choice outcomes from varying numbers of previous trials (lag) to determine the feature choice and state transition probabilities. Each point represents a single subject's mean over a fivefold cross-validation and over 5 (monkey) or 10 (human) different model initializations. Each subject's best-fit model with four states and lag 1 (dashed red box) was chosen for further analysis. c, Probability of selecting the chosen object produced by a model extension based on feature choice probabilities predicted only from choice outcomes on earlier trials (left) and on feature choice probabilities computed from most-likely state estimates derived from past, present, and future choice outcomes (right). The probability on each trial was binned according to the trial's relative position in the rule block and averaged across blocks. Line and shading represent the mean and SD across subjects for each species. Dots represent block percentiles at which the average object selection probability is significantly above chance (bootstrap test with t statistic, p < 0.05). d, Most-likely states estimated by the model for 300 trials in an example human (top) and monkey (bottom) subject. The rule on each block is outlined in black. See Extended Data Figures 2-1 and 2-2 for more details.

The IOHMM-GLM's goodness of fit to behavior depends both on the number of previous trials determining a subject's choice (lag) and on the number of possible hidden states. Accordingly, we fit IOHMM-GLMs to each subject's behavior while systematically varying these two parameters (Fig. 2b). Across subjects in both species, model accuracy showed a stronger dependence on the number of states than on the lag. Crucially, accuracy plateaued as the number of states increased and exhibited overfitting at higher lags. In this task, subjects choose objects rather than individual features. Therefore, we extended our model to compute the probability of choosing each object in a trial, based on the model's predicted probability of choosing individual features on that trial (see Materials and Methods). Fits of this model extension to each subject's behavior based on each of the IOHMM-GLMs in Figure 2b revealed a qualitatively similar relationship between model accuracy and the underlying parameters (Extended Data Fig. 2-1a). For each subject, the best-fit model comprised of four states and with lag 1 (history from the previous trial only) does not overfit the data while producing prediction accuracies at or very close to the performance plateau. Therefore, we selected these models (Fig. 2b, dashed red box; Extended Data Fig. 2-1a) for further analysis.

Figure 2c (left) shows the choice probability predicted by these four-state lag 1 models for the chosen object at each trial after a rule switch, averaged over rule blocks; averaging across blocks is achieved by normalizing the trial number by the block length. The results show that the model's prediction of the chosen object is significantly above chance (0.25) in both species (monkeys, 0.47 ± 0.02; Human Dataset 1, 0.63 ± 0.02). Also, prediction accuracy improves as the rule is learned over the block's time course. Our primary goal, however, is to find the most accurate explanation for each subject's rule-learning behavior rather than predict their future choices based on past choice outcomes. For this, we consider the most-likely sequence of states across trials inferred by the model for each feature. This is given by the maximum a posteriori probability (MAP) estimate of the sequence of states across all trials in an experimental session. In this formulation, each estimated or inferred state best explains not only the present choice but also past and future choices subject to the model's choice probabilities in the inferred past/future states and its state transition probabilities between the inferred present and past/future states (see Materials and Methods; Fig. 2d). The model is generally quite confident in its MAP estimates of the most-likely sequence of states, as measured by the cumulative density of their posterior probabilities (Extended Data Fig. 2-1b). Moreover, since the inferred states for each feature are estimated from past, present, and future choices, they yield more accurate estimates of the choice probabilities for chosen objects (Fig. 2c, right). For this reason, we rely on the inferred states to identify the rule-learning strategy in each species and to interpret the interspecies differences therein.

To gain insight into the interpretation of our model fits, we similarly analyzed the choices of a simulated WSLS agent (Extended Data Fig. 1-1). By construction, we know that the model's choices only rely on the previous trial. As expected, higher lag models tend to overfit the agent's choices (Extended Data Fig. 2-2a). While the agent's true behavior has only two states (Fig. 1c), we find that a three-state model provides a better fit. Our model splits the WSLS algorithm's “avoid” state into two states—a random state in which a feature is selected with chance probability and an avoid state in which a feature is selected below chance. This is due to a combination of the task's structure and our model setup. By picking one feature consistently across trials, the agent necessarily avoids other features in the same dimension. However, the agent's choices of features in other dimensions appear random to the model, since an object is composed of one feature from each dimension and objects compositions are generated randomly on each trial. Thus, the appearance of this additional state results from our model's treatment of each feature independent of its relationship to other intradimensional features, a simplifying assumption that allows for tractable fitting. Nevertheless, the model largely recovers the hidden states and state transitions that drive the agent's behavior—it correctly identifies when a feature is associated with the persist state 57.6% of the time and accurately determines the underlying decision process (Extended Data Fig. 2-2c,d). Collectively, these results show the reliability of this modeling approach to explain rule learning in both species.

Hidden states reflect feature-based attention and reveal qualitatively similar strategies in the two species

Learning is often conceptualized as updates to a decision-making schema based on past decisions and their outcomes (Behrens et al., 2007; Niv et al., 2015). We sought to identify hidden states that capture this decision-making process and to explore what they reveal about the dynamics of human and monkey rule learning in our task. We compute the choice probability of features associated with each state by marginalizing the model's predicted choice probability under each state and history (Extended Data Fig. 3-1a) over the choice–outcome histories (Extended Data Fig. 3-1b). In both humans and monkeys, a comparison of these choice probabilities revealed that the model determines states based on distinct probabilities of choosing the associated feature, ranging from below chance (avoid) to chance (random) to above chance (preferred) to very high (persist; Fig. 3a). That is, the model states correspond to levels of attention paid to each feature. Moreover, this result was consistently observed in the majority of the models fit to the behavior in both species, as well as in a simulated WSLS agent (Extended Data Fig. 2-2). Since features associated with the preferred or persist state are favored during rule learning, we refer to them as being under exploration. We will show that the estimation of the attentional state toward each feature at each trial permits a systematic analysis of when features are selected for or withdrawn from exploration and how the choice–outcome history informs these decisions. This exercise fosters an exposition of the decision-making process that describes the rule-learning strategy in both species, the resulting learning dynamics between rule switch and rule learning, and the identification of the differences in the decision-making process that most prominently explain the learning performance difference between the two species.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Model describes rule-learning dynamics in terms of changes in feature-attentional states. a, Choice probability of features associated with each state in human (Dataset 1, green; Dataset 2, blue) and monkey (brown) subjects computed directly from model parameters and measured empirically based on most-likely state estimates. Choice probabilities order feature states akin to levels of attention. b, Decision process describing how humans, monkeys, and the WSLS agent start, continue, and stop exploring a feature, derived from their history-dependent state transition probabilities. Process is decomposed based on outcome-dependent transitions when the feature is chosen (direct effect) or not chosen (indirect effect). Arrow thickness indicates probability of the transition. Dashed lines highlight deviations from the WSLS strategy. c, Probability of demoting the state of a chosen feature to a lower probability state when an unchosen intradimensional feature is promoted from the preferred to persist state (left). Probability of promoting a chosen feature to a higher probability state when an unchosen intradimensional feature is demoted from the preferred to the random/avoid state (right). Measurements test to what extent indirect effects of promoting or demoting features in the preferred state result from changing the state, and therefore the choice probability, of a chosen intradimensional feature. Perfect causality would coincide with a probability of 1.0. d, Change in the choice probability of a feature in the preferred state after receiving negative (left) or positive (right) feedback for choosing a different feature. The indirect effect significantly increases the feature choice probability in the former situation and decreases it in the latter. See Extended Data Figures 3-1–3-3 for more details.

Since these analyses rely heavily on the most-likely state estimates, we validated the consistency of these estimates with the model's parameters. First, we compared the feature choice probability per history and state computed directly from the fit parameters (model) and measured based on the state estimate for each feature on each trial (empirical). The two measurements yield consistent results, demonstrating that the estimated most-likely states not only best explain the sequence of choices but also conform with the model's parameters. Next, we similarly compared state transition probabilities per history computed directly from the fit parameters (model) and measured based on the state estimate for each feature on each trial (empirical; Extended Data Fig. 3-2). Here again, we find that the transition probabilities computed from the fit parameters (model) are consistent with empirical measurements of the transition statistics based on the most-likely state estimates. Extended Data Figure 2-2b schematizes the decision process in the two species derived from their state transition probabilities. The thickness of an arrow indicates the probability of the respective transition; extremely rare transitions have been pruned. Similar to the WSLS agent, we find that a feature is most often associated with the avoid state, while an intradimensional feature is simultaneously under exploration (Extended Data Fig. 3-1c). Since the avoid state likely emerges due to this interdependence between the choices of interdimensional features, which our model forgoes for tractability, we do not treat it as distinct from the random state.

We compared the observed behavior in our task with a WSLS strategy taking into account task structure differences between our task and the two-armed bandit task (Fig. 1c). The composition of a chosen object by three features forces the choice of features in the avoid state of the WSLS strategy. Thus, a WSLS decision process for our task must define transitions for such features when they are chosen. Moreover, the multidimensional environment of the task offers multiple alternatives for a subject to shift attention to during lose–shift, compared with the two-armed bandit task. An updated WSLS strategy that accounts for these differences is depicted in Extended Data Figure 2-2b (right). We can now compare the decision process inferred by the model for the two species (Extended Data Fig. 2-2b, left-middle) to this WSLS decision process, revealing salient differences that are delineated by dashed lines. Key among these is the existence of a preferred state where items are not chosen with certainty (or near-certainty) as in the persist state but above chance. The effect of direct feedback (as a result of choosing the feature) on these states and the random/avoid states is similar to those in the WSLS decision process. For example, both species select a feature in the random/avoid state for exploration (by promoting it to the preferred state) seemingly at random after receiving negative feedback for choosing other features. However, an interesting exception is that humans sometimes choose to explore such a feature upon receiving positive feedback for choosing it.

Larger differences emerge with regard to indirect effects of feedback. A feature may not be chosen on a trial when it is associated with the preferred state (feature choice probability in preferred state <1, Fig. 3a). However, its state may still transition subject to the feedback received at the end of the trial—an indirect effect. For example, humans and, to a lesser extent, monkeys demote features from the preferred to the random/avoid state upon receiving positive feedback for choosing a different feature. Consequently, their probability of subsequently choosing an unchosen feature that was associated with the preferred state decreases (Fig. 3d, right). They also promote features from the preferred to the persist state upon receiving negative feedback for choosing a different feature. Consequently, their probability of subsequently choosing an unchosen feature that was associated with the preferred state increases (Fig. 3d, left). This is striking because it is the only way a feature can transition into the persist state, which appears to be reserved mainly for a feature that the subject determines to be the rule (Fig. 2d). Receiving positive feedback for choosing a feature in the preferred state does not definitively confirm that it is the rule, since the rule may be among the other two features in the chosen object. Confidence in the rule's identity may be increased based on the consistency of receiving such direct positive feedback across many trials. Alternatively, it may be done by ruling out other candidates, that is, after receiving negative feedback for choosing an object with a different candidate feature. Consistent with this interpretation, measurements show that when a feature under exploration is not chosen, the object that is chosen often contains a different feature that is also under exploration (Fig. 4d).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Monkeys and humans explore multiple features for several trials in a row to evaluate them. a, The percentage of all trials where at least one feature is under exploration by humans (Dataset 1, green; Dataset 2, blue) and monkeys (brown). b, Distribution of the number of features explored by each monkey and human subject in a block. c, Distribution of number of continuous trials with a feature in an exploration state. d, Probability of choosing an object with all features in the random or avoid state, while at least one other feature is in the preferred or persist state. e, Distribution of number of features simultaneously explored by each monkey and human subject in trials where at least one feature is under exploration.

This approach of promoting a feature to the persist state as an inferred consequence of ruling out an alternative candidate, rather than integrating direct positive feedback across trials in favor of the feature, may be favored by both species due to its computational simplicity—it relies on the outcome of just the previous trial rather than multiple trials and thereby reduces working-memory demands. However, it is possible that these inference-like computations are not deliberate but an inadvertent consequence of demoting or promoting an intradimensional chosen feature. For example, given that the probability of choosing all shapes must sum to 1, when one shape is demoted after its choice produces negative feedback, the probability of choosing another shape that was associated with the preferred state may automatically increase, forcibly promoting it to the persist state. Measurements of the probability of demoting or promoting the chosen feature while promoting or demoting, respectively, an unchosen intradimensional feature in the preferred state are mixed: monkeys do so at chance levels; humans always demote the chosen feature while promoting the unchosen feature but seldom promote the chosen feature while demoting the unchosen feature (Fig. 3c). Nevertheless, these indirect-effect transitions directly and significantly alter the subsequent choice probability of the unchosen feature (Fig. 3d). In summary, the best-fit models discover feature-based attentional states whose dynamics show marked deviations from a WSLS strategy.

Both species simultaneously evaluate multiple features over several trials during rule learning

The explore–exploit dilemma pits the benefit of continuing to select a recently rewarded option (exploit) against the benefit of selecting a different and potentially more rewarding (but possibly less rewarding) option (explore). While much work has been done to determine how humans and other animals navigate this dilemma (Hills et al., 2015; Gershman, 2018; Wilson et al., 2021), how they deal with it in a multidimensional environment with transiently overlapping options remains unclear. Which of the three features of a chosen and rewarded object should be exploited on the next trial, given that they are unlikely to appear colocated in the same object on the following trial? How should the trade-off between the computational complexity and information efficiency of exploring several features at once be resolved?

The model finds that both species continuously explore one or more features (Fig. 4a). In the process, they explore multiple features over the course of a block before ultimately identifying the rule (Fig. 4b). Moreover, each feature is often explored for a series of several trials in both species (Fig. 4c). But the number of these trials is substantially larger in monkeys, a finding we analyze more closely in the following sections. The model also indicates that both species often explore multiple, but not all, features at a time (Fig. 4e). This is consistent with the theory of selective attention (Driver, 2001; Corbetta and Shulman, 2002) wherein objects are selectively attended to (or filtered for higher processing) subject to an internally maintained set of relevant perceptual features (or attentional filters). This also underscores how both species solve the computational complexity–information efficiency trade-off. Since it is computationally challenging to simultaneously attend to and evaluate all 12 features over several trials but inefficient to attend to one feature at a time, both species evaluate a small subset of all features at a time. Indeed, during exploration, it is uncommon for either species to select an object where none of the features is in the preferred or persist states (Fig. 4d). However, monkeys do engage in such random exploration much more frequently than humans.

From these results, we conclude that both species exhibit a deliberate form of exploration to address the challenges inherent in the task environment. Features, often more than one at a time, are selected for exploration via promotion to the preferred state after choices of other features produce negative feedback (Fig. 3b). They are then continuously explored so long as they produce positive feedback, until alternatives are ruled out. At this point, they are then promoted to the persist state or are ruled out after choosing them produces negative feedback (or, in the case of humans, choosing other features produces positive feedback).

Categorization of feature-attentional states characterizes learning dynamics

Rule learning proceeds through a sequential process that progressively reduces ambiguity regarding the rule's identity until it is ultimately determined. Our model reveals elementary feature-specific computations that individuals in each species apply to maintain and update a small subset of candidate features, of which one may be the rule. To elucidate the over-arching learning dynamics that governs the inferred sequential rule-learning process, we developed a simple approach to categorize individual trials based on the features under exploration and the true rule (Fig. 5a). The categories are mutually exclusive and exhaustive—each trial falls into one and only one category. These are as follows:

1. “Perseveration”: a continuous series of trials following a rule switch when the feature governing the previous rule is associated with the persist state

2. “Random search”: trials when none of the features are under exploration (i.e., associated with the preferred or persist states)

3. “Nonrule exploration”: trials when one or more features are under exploration not including the rule feature

4. “Rule-favored exploration”: trials when one or more features including the rule are under exploration

5. “Rule preferred”: trials when only the rule is associated with the preferred state

6. “Rule exploitation”: trials when only the rule is associated with the persist state

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Exploration-based trial categories reveal learning dynamics and identify causes for monkey learning performance deficit. a, Definition of the six trial categories based on whether the features under exploration during the trial include the rule feature. b, Distribution of trial categories at each percentile of rule block. Lines and shaded areas reflect mean values and SEM, respectively, across subjects. c, Trial category summary statistics (top, mean number of trials; middle, variance of number of trials; bottom, reward probability) across rule blocks for human (Dataset 1, green; Dataset 2, blue) and monkey (brown) subjects. Interspecies comparisons of the mean number of trials per category reveal significant differences in the perseveration, random search, rule-favored exploration, rule preferred, and rule exploitation categories (bootstrap test with t statistic); n.s., not significant; *p < 0.1; **p < 0.01; ***p < 0.001. See Extended Data Figure 5-1 for more details.

We compare the distribution of categories for trials across the course of a rule block between species (Fig. 5b). Humans show a progression from perseveration to nonrule exploration, where nonrule features are explored and ruled out, to rule-favored exploration, where the rule feature is simultaneously explored with nonrule features, to rule exploitation, once other candidates are ruled out and the rule is identified. Monkey rule learning is described by similar dynamics except for a much higher incidence of the rule preferred category for most of the block. Also, while nearly all blocks in Human Dataset 1 end in the rule exploitation category, a large proportion of the blocks in the monkey and the second human datasets end in the rule-favored exploration category. A closer analysis of the learning criterion trials, which occur at the end of the block, revealed different reasons for this result in the latter two datasets. The stronger learning criterion (i.e., larger number of trials-to-criterion) in these datasets makes it more likely that the rule and a nonrule feature colocate in the same object on two or more trials in close temporal proximity. Consequently, even though the human subject is exploiting the rule feature in these trials, the model concludes that the nonrule feature is simultaneously under exploration. In contrast, monkeys continue exploring nonrule features even when they do not colocate with the rule feature. This distinction is evidenced by the reward probability, which approaches 1 much sooner in Human Dataset 2 than in the monkey dataset (Extended Data Fig. 5-1b). These results show that our categorization approach expresses human and monkey rule-learning dynamics in terms of behaviorally interpretable learning stages, for example, an increase in the reward rate following a rule switch in both species is marked by the onset of rule exploration with the rule-favored exploration category (Fig. 5c, bottom).

Examining the number of trials spent in each category determined bottleneck categories that produce the rule-learning performance deficit in monkeys (Fig. 5c). Specifically, monkeys spend much longer perseverating on the previous rule, in disambiguating the rule feature from nonrule features (rule-favored exploration) and demonstrating that they have learned the rule (rule preferred or exploitation). The latter two sources of the learning performance deficit in monkeys also explain a majority of the variance in their performance across blocks (Extended Data Fig. 5-1a, bottom). In contrast, the number of trials humans spend exploring nonrule features before selecting the rule feature for exploration (nonrule exploration) largely determines the variance in their rule-learning performance.

Random exploration prolongs the expression of learning in monkeys

A key difference between the two species identified via this trial categorization is that monkeys spend many more trials than humans in the rule preferred or exploitation categories. These extra trials spent demonstrating or expressing that the rule has been learned significantly increase both the block length mean and variance (Fig. 5c; Extended Data Fig. 5-1a). A comparison of monkey learning performance and human performance in Dataset 2 shows that this interspecies difference is not caused by a difference in task parameters or learning criteria. Instead, we hypothesized that the larger mean and variance of the duration of time spent in the rule exploitation category by monkeys compared with that by humans (Fig. 6a) may result from their random exploration of other features when a feature is already associated with the persist state (Fig. 3a). This behavior is unique to monkeys and is prevalent even during rule exploitation trials (Fig. 6b)—after they have identified the rule, monkeys occasionally choose objects that do not include the rule feature.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Random exploration and perseverative errors prolong monkey rule learning. a, Mean number of trials spent by human (Dataset 1, green; Dataset 2, blue) and monkey (brown) subjects in the rule exploitation category per rule block. b, Probability of selecting an object with the rule feature across trials in the rule exploitation category. Monkeys occasionally explore other objects compared with humans. c, Distribution of the number of trials spent by human and monkey subjects in the rule exploitation category per rule block (left) and by simulated agents that select the rule feature with probabilities in b until they reach a learning criterion (right). d, Mean number of trials spent by human and monkey subjects in the perseveration category per rule block. e, The probability of humans and monkeys choosing the previous rule feature at each trial after a rule switch (left) is commensurate with the probability of the previous rule feature being associated with the persist state (right). f, The probability of the previous rule feature transitioning back into the persist state after its selection produces negative feedback is higher in monkeys (bootstrap test with t statistic); n.s., not significant; *p < 0.1.

To test our hypothesis, we simulated agents that select the rule feature with the same probability as monkeys and humans do during the rule exploitation category and asked how many trials it would take these agents to reach a learning criterion. The results revealed that the trial count distributions of the simulated agents were nearly identical to the corresponding subjects (Fig. 6c), thus confirming our hypothesis. Similar “random errors” have been observed in humans with focal lateral prefrontal lesions on the WCST (Barcelo and Knight, 2002), where they were attributed to distraction or a failure to maintain the rule in working memory. However, it remains unclear whether the monkeys in our experiments were more distractable than their healthy human counterparts or deliberately adopted occasional random exploration as part of their strategy—for example, to prolong a highly rewarded state.

Reduced sensitivity to negative feedback increases perseverative errors in monkeys

Perseverative errors occur when the rewarded feature on the previous block continues to be chosen following the rule switch despite receiving negative feedback for the choice. Such errors are characteristic of frontal lobe damage and dysfunction in the WCST (Milner, 1963; Gläscher et al., 2019), where they are believed to reflect a cognitive deficit in adapting to changes in task contingencies. Pronounced perseveration error rates in the WCST are also observed in patients with neuropsychiatric (Sullivan et al., 1993; Ozonoff and McEvoy, 1994; Everett et al., 2001) and substance abuse disorders (Sullivan et al., 1993; Bishara et al., 2010). Interestingly, our model's association of the persist state with the previous rule feature during consecutive trials immediately following a rule switch suggests that monkeys persevere on the previous rule for several more trials than humans (Fig. 6d). Indeed, direct measurements showed that the probability of choosing the previous rule after a rule switch is consistent with such a state estimate in both species (Fig. 6e).

To determine the cause of the elevated perseveration in monkeys, we asked which choice outcome(s) best explained the difference in continued persistence with the previous rule between the two species. Our analysis showed that humans were far more likely to demote the chosen previous rule feature from the persist state in response to negative feedback compared with monkeys (Fig. 6f). Monkeys’ weaker sensitivity to negative feedback parallels that of humans with substance abuse disorders and prefrontal lesions performing the WCST, who also perseverate more than healthy controls (Bishara et al., 2010; Gläscher et al., 2019).

Reduced negative feedback sensitivity compromises efficient credit assignment and prolongs rule learning in monkeys

The largest contribution to the interspecies difference in rule-learning performance is from trials in the rule-favored exploration category where the rule feature is concurrently explored with one or more nonrule features (Fig. 7a,b, left). While it is reasonable to explore the rule for several consecutive trials as it produces rewards, what must be explained is why nonrule features are concurrently explored for many more trials by monkeys. Indeed, monkeys continuously explore individual nonrule features for many more consecutive trials during the rule-favored exploration category (Fig. 7b, right). This explains the lengthier duration of this category in monkeys, resulting from a higher probability of a nonrule feature transitioning back into an exploration state during rule-favored exploration trials (Fig. 7c). Analysis further showed that this interspecies difference in transition probability is explained by a lower sensitivity of monkeys to either form of negative feedback—direct, when the nonrule feature is chosen and negative feedback is received, and indirect, when it is not chosen and positive feedback is received (Fig. 7d).

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Diminished negative feedback sensitivity prolongs concurrent exploration of rule and nonrule features. a, Mean number of trials spent by human (Dataset 1, green; Dataset 2, blue) and monkey (brown) subjects in the rule-favored exploration category per rule block. b, Distribution across rule blocks of the number of trials spent in the rule-favored exploration category by each subject (left) and of the number of consecutive trials spent by them exploring individual nonrule features during this category (right). c, Probability of a nonrule feature transitioning back into an exploration state during rule-favored exploration trials. d, The probability of a nonrule feature transitioning back into an exploration state upon receiving negative feedback for choosing it (direct negative feedback) or positive feedback for choosing a different feature (indirect negative feedback) during rule-favored exploration trials is higher in monkeys (bootstrap test with t statistic). e, Probability of a nonrule feature transitioning back into an exploration state during rule-favored exploration trials and nonrule exploration trials in monkeys. f, Joint probability of a nonrule feature transitioning back into an exploration state and each choice–outcome history occurring during rule-favored exploration trials and nonrule exploration trials in monkeys (left); probability of each choice–outcome history occurring during either category (middle); probability of a nonrule feature transitioning back into an exploration state in response to each choice–outcome history during either category (right). The higher probability of a nonrule feature transitioning back into an exploration state during rule-favored exploration trials compared with that during nonrule exploration trials is explained by a higher incidence of direct positive feedback for choosing the nonrule feature in the former category (bootstrap test with t statistic); *p < 0.1; **p < 0.01. See Extended Data Figure 7-1 for more details.

While both species also explore nonrule features during nonrule exploration trials, time spent in this category is relatively short in both humans and monkeys (Fig. 5c). So what explains the difference in duration between the two categories in monkeys? Since the nonrule exploration category is followed by rule-favored exploration trials, one possibility is that it is cut short by the onset of exploration of the rule feature as the nonrule feature continues to be concurrently explored for many more trials. However, measurements in monkeys showed that nonrule feature exploration only occasionally spans the two categories (probability = 0.27 ± 0.04). Therefore, the number of trials during which a nonrule feature is explored by monkeys is usually much smaller when it happens in the nonrule exploration category than in the rule-favored exploration category.

This difference in duration is reflected in a higher probability of a nonrule feature transitioning back into an exploration state during rule-favored exploration trials (Fig. 7e). Since the probability of transitioning back into the explore state is a marginalization of its joint probability with the choice outcome it follows, we asked what choice–outcome history best explains the transition probability difference between the two categories. Measurements showed that receiving positive feedback for choosing the nonrule feature (C⁺) is the key differentiator between the joint probabilities for the two categories (Fig. 7f, left). This could either be because the transition probability in response to this history is different under the two categories or because the frequency of the history is different under them. We found that while the transition probabilities are similar (Fig. 7f, left), monkeys are more likely to have received positive feedback for choosing a nonrule feature under exploration during the rule-favored, exploration category (Fig. 7f, middle). When the rule feature is concurrently explored with a nonrule feature (rule favored, exploration), the probability of selecting them both when they colocate in an object is higher. This increases the probability of receiving positive feedback for choosing the nonrule feature, which makes appropriately assigning credit to the rule feature challenging. This underscores the importance of negative feedback sensitivity in demoting nonrule features from exploration states, in the absence of which the duration of concurrent exploration of the rule and nonrule feature(s) is prolonged.