Abstract
Learning includes the ability to generalize to new situations and respond to similar, yet not identical stimuli. We use stimulus generalization in humans to show that tones that were negatively reinforced induce wider generalization curves than tones that were positively reinforced, and these in turn induce wider curves than neutral memory. Importantly, these wider generalization curves persist even if outcomes for all tones are made identical, indicating that the learning induced a perceptual change, and not merely a decision bias. Moreover, it persists after taking into account loss-aversion, suggesting it is a result of valence per se, and not intensity that reflects overweighting of the aversive stimuli. This effect of emotional valence on learning suggests different locations of plasticity and network mechanisms in the brain. Particularly, it suggests that brain areas that mediate reinforcement and emotions are involved during the learning process to induce a neural representation that can support this broader behavioral generalization. In addition, these findings highlight a model for anxiety and trauma disorders in which aversive experiences affect more than they should, sometimes even in seemingly irrational situations.
Introduction
If you are a dog, mistaking another dog bark for a lion roar is a shame—you might have missed a friend; but mistaking a roar for a bark can be fatal. In signal detection theory, for example, a different penalty for misses or false alarms can induce decision bias (Green and Swets, 1989; Wickens, 2001). Because a miss in the above-mentioned example is much more “costly” than a false-alarm, a more liberal bias for the loss-related stimulus can be expected. However and importantly, in real-life, it is rarely the exact specific roar or specific bark you once heard, and stimuli rarely repeat with the exact physical properties with which they were first encountered. This means that the same logic used for immediate response should generalize to similar, yet not identical, sounds. We therefore hypothesized that valence associated with a stimulus during learning would influence the scope of generalization, and specifically, that stimuli that were previously associated with loss would generalize more than stimuli that were previously associated with gain.
To test this, we used stimulus generalization, the transfer of a learned response from one stimulus to another similar (but not identical) stimulus (Guttman and Kalish, 1956; Pearce, 1987; Shepard, 1987; McLaren and Mackintosh, 2002; Bouton, 2006). To look for evidence for the long-term effect of valence on the generalization, we tested generalization separately from acquisition. We designed a two-stage instrumental paradigm, where subjects first learn to associate loss or gain of monetary outcomes with different auditory stimuli. Later on, we tested the generalization to different tones surrounding the two conditioned tones, but in a task that now has equal rules and outcomes for all stimuli. Our hypothesis predicts that generalization would be affected by the valence in the initial conditioning, rather than by the current equal conditions and outcomes.
Materials and Methods
Paradigm, experiments, and subjects.
Participants were healthy volunteers (median age of 25 years, 20 males and 19 females). Experiments were conducted under a Helsinki approval with signed consents. Subjects were paid a minimum for their participation, but were informed they could gain more (up to 2.5-fold) by correct responses.
The experiment consisted of two iterated parts, an acquisition stage—designed to assign valence of gain versus loss to two different tones; and a generalization stage—designed to test differences in the generalization to the two tones. For the acquisition stage, subjects were instructed that in each trial, they would hear one of three tones (300, 500, or 700 Hz, for 200 ms); one tone would gain them money—if they pressed one of two keys after it (termed “positive” tone); one tone would lose them money—unless they pressed the other key after it (termed “negative” tone, because it is a negative reinforcer), and one tone had no outcome independent of what was pressed or not (“neutral”). The key assignment was counterbalanced across subjects. Subjects had to learn by trial-and-error which tone is the positive and which is the negative, and had 2.5 s following the tone presentation to press a key, and then received visual feedback telling them if they earned or lost in the trial. In addition, there were pavlovian trials (classical conditioning), in which the positive tone resulted in gain independent of what was pressed, and the negative tone resulted in loss independent of what was pressed. Subjects were informed by the appearance of the word “helpless” on the screen. The acquisition stage comprised of 24% instrumental negative, 24% instrumental positive, 10% pavlovian negative, 10% pavlovian positive, and 32% neutral. Negative and positive tones were 300 and 700 Hz, counterbalanced across subjects, and the neutral tone was 500 Hz. There were 100 trials overall in the acquisition stages. Three subjects were excluded from subsequent analysis because they did not understand the instructions, as evidenced from their extremely poor behavior. Each loss and gain was of 0.5 Israeli shekels (∼15 US cents).
Participants were informed when the acquisition stage was over and that a new different stage begins. They were not informed in any way that this is a generalization stage or that this is the purpose of the study, and moreover, we questioned the subjects after the experiments and none had an idea that generalization was the task. In this stage, subjects were instructed that there would be many more tones. In each trial, if they hear the positive/negative tone, they should press the same key that was previously associated with it. However, if they hear a tone that is not the exact same tone as either the positive or the negative tone, they should press the “middle” key. They were informed that they would gain money for a correct press (either pressing the appropriate key for a negative or positive tone, or the middle key for a different tone), or lose the same amount for any mistake (pressing the positive/negative keys for a different tone, or the middle key for a positive/negative tone). In this stage, we forced the subject to choose a key and press by imposing a big loss (10 times the amount of regular loss) for no response. There was no feedback reported to the subjects in this generalization stage, to avoid any changes in valence of the tones that was acquired during the acquisition stage.
Notice that although we keep calling the original tones positive/negative tones, these two tones have completely identical requirements and outcomes in this stage and therefore should entail identical decision-making policies. We keep these names for ease and clarity of terminology because they were previously associated with positive/negative outcomes.
The generalization stage comprised of 14% of the positive tone, 14% of the negative tone, and 72% of other tones (i.e., that required a press on the middle key). Of these other tones, 14% were “control” tones that are far away from the original tones (80, 100, 100, 120, 480, 500, 500, 520, 880, 900, 900, 920 Hz), and 58% were tones that were closer to the original (positive and negative) tones, [−100, −60, −20, −5, +5, +20, +60, +100] from 300 and 700 Hz. There were 84 trials overall in the generalization stage.
The acquisition and generalization stages appeared three times consecutively in a loop (Fig. 1), to keep the original tones reinforced and avoid extinction effects. Indeed, there was little difference in the response rate for the conditioned tones in the three iterations of the generalization stage (p > 0.1, two-way ANOVA), and no difference in the response rate between each acquisition stage and its consecutive generalization stage (p > 0.1 for all, Wilcoxon rank tests), showing that there were little, if any, extinction effects.
To find the “neutral” generalization curve, we re-called the same participants and performed the same experiment, with an identical acquisition stage, but this time the generalization stage had tones that are [−100, −60, −20, −5, +5, +20, +60, +100 Hz] around the 500 Hz neutral tone, without the tones surrounding the positive/negative tones.
Loss-aversion paradigm.
We repeated the experiment with a new set of participants (n = 14), but this time compensating for possible loss-aversion behavior. We tested our subjects for loss-aversion before the experiment (Tom et al., 2007). Each subject was presented with a series of choices of “gaining X money in 0.5 probability and losing Y money in 0.5 probability,” and had to accept/reject the offer. Subjects were told that soon after the test is over the computer would randomly pick a single offer they had accepted and that, following a “virtual coin toss,” they would actually gain or lose the amount of money noted in that particular offer. X and Y were varied over a range of 5–20 to create a matrix of 256 binary choices. We then fitted logistic regressions with accept/reject choices as dependent variables and the size of potential gain and loss as independent variables. The loss aversion was computed as λ= −βloss/βgain, which is similar to loss-aversion in prospect theory with the simplifying common assumptions of linear value function and identical weights for 0.5 probabilities. The median in our subjects (n = 11) was found to be 1.5 and the mean was 1.7, similar to other reports in the literature (Tom et al., 2007; Sokol-Hessner et al., 2009). In the experiment, we chose to use a robust loss-aversion factor of 2 (λ = 2, a gain of twice the loss, i.e., 1 shekel of gain and 0.5 shekel of loss).
Analysis.
We used two approaches to quantify the generalization rates for individual subjects, and both yielded highly similar results. First, we measured the drop in response rate per 1 Hz of tone; this is the overall slope of the generalization curve in the range tested in our paradigm (±100 Hz). These slopes were then compared using paired t tests (paired within a subject). We further verified that all comparisons that are reported as significant, were also significant at the same level using nonparametric Wilcoxon signed rank test. We used three-way ANOVA to check the effects and interactions of reinforcement type (negative/positive/neutral), the frequency distance (in Hz) from the original tone [0, 5, 20, 60, 100] and the side/direction of the frequency change [higher/lower than the original tone].
Second, because the shape of the generalization curve is nonlinear and is better described by a logistic function, we fitted a multinomial logistic regression for each subject using all single trial (0/1—correct/incorrect) responses. This makes sense because the logistic function reaches a peak plateau at one side (the maximum response at the original tone), and a low plateau (i.e., zero) at the other side (the lowest response at “infinite” distance from the original tone). In this model, the coefficient β1 describes the contribution of the reinforcement type to the response. The higher β1 is, the narrower the generalization curve is. The β1 coefficients of positive and negative data were compared using t tests for each subject, and the p-values were incorporated in Fisher's combined p-value calculation to obtain an overall p-value for the population.
The two approaches yielded significant results for all mentioned tests, except one: in the slope approach, the positive curves were wider significantly than the neutral ones; in the logistic approach, they were wider but not significantly. In both approaches the negative curves were significantly wider than both positive and neutral curves, and this was our original hypothesis.
Results
The paradigm consisted of two different stages: an acquisition stage—where subjects learned to associate one tone with a gain and another tone with a loss of the same amount; and a generalization stage—where subjects were tested for how much they classify a tone that is different from these two original tones as an original tone (Fig. 1A) (see Materials and Methods for full description). Subjects acquired the associations within few trials (Fig. 1B, p < 0.001, trials factor in two-way ANOVA), reached plateau performance after 4–6 trials and well before the end of the acquisition stage (Fig. 1B, p > 0.1, post hoc t tests), and reached similar performance for positive and negative tones (Fig. 1B, p > 0.1, valence factor in two-way ANOVA over 10 last trials). Thus, subjects associated two different tones with different outcomes, and we now refer to these original two tones as the “positive tone” and “negative tone,” to reflect their association with gain and loss, respectively.
Behavioral paradigm. A, Subjects learned by trial-and-error which of two pure tones (300/700 Hz) can result in gain (if followed by a press on one of two keys, e.g., the “left” key) and which can result in loss (if not followed by a press on the other key, e.g., the “right” key). They could use this acquired information to gain more funds, but we introduced new tones in addition; some were completely different from the positive and negative tones (e.g., 100, 500, 900 Hz), and some were in distances of [−100, −60, −20, −5, +5, +20, +60, +100] Hz around the positive and negative tones. Subjects were instructed to press the same key as in the acquisition stage if the tone heard matched exactly either the positive or the negative tone, and a third middle key if it did not match either two. Notice that in this stage, the potential outcomes are identical for all tones: subjects would gain more money for a correct press, i.e., either if after hearing a positive or negative tone, they pressed the original key that was associated with it; or if after hearing a tone that was different from the positive/negative tones, they would press the middle key. They would lose an equal amount of money for any type of error, i.e., either for pressing one of the original keys for a tone that is not one of the positive or negative tones, or for pressing the middle key when it was one of the positive or negative tones. The dashed lines in the generalization stage are the main concern of this paper: how much the subjects mistake a tone for the original tone, although they lose money for it. B, Correct response rate for both positive and negative reinforcement in the acquisition stage. Shown are the raw response rates in windows of two trials averaged over subjects. There was significant above chance-level performance already in the first 2 trials (p < 0.01, χ2) which improved fast further (p < 0.01, ANOVA) and reached plateau already at the ∼6 trial (p > 0.1, post hoc t tests). Thus, subjects learned to discriminate between the 300 and 700 Hz tones and associate them with the correct key press.
In the generalization stage that followed immediately (Fig. 1), we introduced new tones in addition; some were completely different from the positive and negative tones (e.g., 100, 500, 900 Hz), and some were in distances of [−100, −60, −20, −5, +5, +20, +60, +100] Hz around the positive and negative tones. Subjects were required to identify which tone was the original tone in the acquisition stage and which is a new tone which was not encountered in the acquisition stage. In short, they were required to press the same key as in the acquisition stage if the tone heard matched exactly either the positive or the negative tone, and a third middle key if it did not match either two. Subjects (n = 22) followed this simple rule as evidenced from their responses: they pressed the appropriate key for the positive tone (p < 0.001, χ2) and for the negative tone (p < 0.001, χ2), and pressed the middle key for the tones that were clearly distinguished from them (e.g., 100, 500, 900 Hz, p < 0.001, χ2).
We compared the subjects' responses to the tones surrounding the positive tone, versus those surrounding the negative tone. We emphasize that although we refer to them as positive and negative, it is because they were previously conditioned so during the acquisition stage, but they yield identical outcomes in the generalization stage, and therefore should entail similar decision-making policies. We found that patterns of generalization, defined as mistaking a different tone to be a positive/negative tone, were significantly different for these two tones (p < 0.001, ANOVA). There was a significant interaction between the frequency and the type of reinforcement (p < 0.001, ANOVA), indicating that it is not merely an overall change in the amount of learning but a change in the shape of the generalization curve. We focus on absolute distance in Hz from the original tone (Fig. 2A,C), because there was no effect of side/direction (Fig. 2B inset, p = 0.44, ANOVA) or interaction with original frequency (300 vs 700, p > 0.1). We quantified the individual gradients for generalization curves and found that loss induced a much wider generalization curve than gain (Fig. 2B, p < 0.001, paired t tests). We further verified this using individual logistic regressions (p < 0.01, Fisher's combined), and found that the coefficients for loss were a few fold higher than for gain (average ratio 8.3, p = 0.02, t test).
The amount of generalization for negative (aversive), positive (rewarding), and no (neutral)—reinforcement. A, Average response rate in the generalization stage for tones in different distances from the tones that were reinforced during the acquisition stage (n = 22). B, Average slopes (drop in response per 1 Hz) in the three conditions. Inset shows separately generalization for lower frequencies and for higher frequencies than the original tone (main effect of reinforcement type, p < 0.01, two-way ANOVA; no effect of side, p = 0.4; no interaction p = 0.2). C, Same data and results as in A, but presented as percentage difference in hertz from the original reinforced tones (300 and 700 Hz, counterbalanced across subjects).
Both the negative and positive generalization curves are much wider than what could be expected for standard auditory discrimination tasks, where the just-noticeable-difference (JND) for humans ranges around 1–10 Hz. However, our task has a memory component which could influence the generalization, and we therefore repeated the experiments but this time tested generalization on the tone that was not reinforced during the acquisition (500 Hz). The generalization curves obtained for this neutral auditory memory were narrower than the curves obtained for positive learning (Fig. 2A,B, p < 0.05, paired t test), and from those obtained for negative learning (Fig. 2A,B, p < 0.01, paired t test).
There are two possible reasons for the wider curves: that it is a result of valence per se; or that it is a result of loss-aversion, i.e., loss is weighted more than gain if they have similar absolute value, and therefore it is an indirect result of different magnitudes of stimuli. To differentiate between the two alternatives, we repeated the experiments (n = 14) with a factor of 2, i.e., with gain that is twice the size of the loss (see methods). We found similar generalization curves in this set of experiments (Fig. 3A,B, p < 0.01, t tests for linear slopes, p < 0.01, Fisher's combined for logistic regressions). A full two-way ANOVA model that includes both experiments (n = 36) revealed a significant effect of valence (positive vs negative, p = 0.007), nonsignificant effect of intensity (matched in absolute value in the first experiment, corrected for loss-aversion in the second, p = 0.06), and no interaction (p = 0.3). In addition, there was no difference between the ratio of individual negative slopes that are lower than positive slopes in the first and second experiments (18%—4/22 and 28%—4/14, respectively; p > 0.1, χ2, Fig. 3C). Finally, there was no correlation between the individual loss aversion and the width of the individual generalization curves (r = 0.09, p = 0.7, n = 11).
The scope of generalization after considering loss-aversion behavior. A, Average response rate in the generalization stage (same format as in Fig. 2A) but for the second experiment (n = 14), where the sum for negative reinforcement was half the sum than for positive reinforcement. This compensates for loss-aversion behavior of factor 2 (λ = 2). B, Average slopes as in Fig. 2B for the second experiment. C, The distribution of individual rates (slopes) of generalization for all subjects, plotted as the slope in the positive condition versus in the negative condition (n = 36; asterisks for first experiment and diamonds for the second). In the positive condition, 78% (28/36) had a larger slope (p < 0.01, χ2). The inset shows the histogram of individual differences between the positive and negative conditions (mean 0.23; median 0.18; both different from zero at p < 0.01).
Discussion
Our findings that negative reinforcers induce wider generalization than positive or neutral ones make intuitive evolutionary sense: for survival purposes, it is wiser to generalize more loss-related stimuli and thus “not take the risk”. From classical detection theory we know that valence of different outcomes can bias decisions (Green and Swets, 1989; Wickens, 2001). Here we extend this to show that valence can affect not only active decision-making process, but also the process of learning so that it later generalizes more to less similar situations. Notice that this is an apparent “irrational” choice of the subjects, because they lose money due to the wider generalization. The much narrower frequency-selectivity we found for the neutral memory suggests that subjects generalize from one tone to another not only because they cannot tell the difference and discriminate them, but because they judge them likely to belong to a set of situations with similar consequence (Shepard, 1987).
There are two major contributions to how valence can affect generalization. One possibility relates to the loss-aversion effect described in prospect-theory (Kahneman and Tversky, 1979). In this scenario, loss is overweighed and therefore has a higher “magnitude” than gain of the same absolute size. This means that the wider curves we observed can be a result of higher emotional intensity evoked by the stimuli, and that valence is an indirect contributor. An alternative is that valence per se affects the learning process differently and hence the generalization. We think this option is more likely. First, because loss-aversion probably does not apply for relatively small magnitudes like we used (Harinck et al., 2007). Second, we tested this option directly and repeated the experiments with gains twice the absolute size of losses (which is the common loss-aversion factor described and even higher than found for our subjects). We found that the width of generalization curves was still higher with the negative reinforcement than with the positive reinforcement. Therefore, although we do not rule out the contribution of intensity/magnitude, we conclude that there is an effect of valence per se on the generalization.
That valence changes patterns of generalization suggests different brain mechanisms. Neurons with different properties contribute differently to the learning process, and as a result, patterns of generalization point to the location of plasticity in the brain and the underlying network mechanisms (Ahissar, 2001; Paz et al., 2004). Whereas neutral auditory memory and discrimination is most likely mediated by neurons in the auditory pathway with many types of tuning curves (Weinberger, 2007; Bitterman et al., 2008), learning processes that are guided by reinforcement and attention involve other networks as well (Roelfsema et al., 2010). Specifically, negative reinforcers and emotions are mediated by brain areas like the amygdala (Phelps and LeDoux, 2005; LaBar and Cabeza, 2006). Loss-aversion, for example, was indeed shown to depend on the amygdala and connected structures (Tom et al., 2007; De Martino et al., 2010). Recent studies have shown that in aversive learning, discrimination and generalization patterns are influenced by properties of amygdalar networks (Armony et al., 1997; Laxmi et al., 2003; Shaban et al., 2006; Bergado-Acosta et al., 2008; Han et al., 2008; Chavez et al., 2009; Duvarci et al., 2009), and that the amygdala can modulate cognitive processes automatically (Dolan and Vuilleumier, 2003), suggesting it can underlie the stimulus-generalization independent of an aware decision by the subjects. More electrophysiological studies and understanding of how amygdala circuitry is involved in the coding of aversive reinforcers (Maren and Quirk, 2004; Paré et al., 2004; Salzman et al., 2007) can clarify the neuronal mechanisms that underlie the generalization curve.
Such neural mechanisms for generalization can suggest a working model for anxiety and mood disorders. In extreme situations, a fearful stimulus can elicit sustained and generalized fear (Hearst, 1960). In more common real-life scenarios, wider generalization curves can explain why aversive experiences elicit emotional responses in later seemingly nonrelevant situations, as with posttraumatic-stress-disorder (PTSD) (Grillon and Morgan, 1999; Dunsmoor et al., 2009; Lissek et al., 2010). We thus suggest that individual differences in the effect that emotional valence has on generalization could underlie susceptibility to long-term effects of emotional events.
Footnotes
This work was supported by the Minerva Foundation and Marie Curie International Reintegration Grant to R.P. We thank Roy Harpaz for help in the experiments and Yadin Dudai for discussions.
- Correspondence should be addressed to Dr. Rony Paz, Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel 76100. rony.paz{at}weizmann.ac.il
