Aversive Prediction Error Signals in the Amygdala

Prediction error signals are fundamental to learning. Here, in mice, we show that aversive prediction signals are found in the hemodynamic responses and theta oscillations recorded from the basolateral amygdala. During fear conditioning, amygdala responses evoked by footshock progressively decreased, whereas responses evoked by the auditory cue that predicted footshock concomitantly increased. Unexpected footshock evoked larger amygdala responses than expected footshock. The magnitude of the amygdala response to the footshock predicted behavioral responses the following day. The omission of expected footshock led to a decrease below baseline in the amygdala response suggesting a negative aversive prediction error signal. Thus, in mice, amygdala activity conforms to temporal difference models of aversive learning.


Introduction
Learning that certain stimuli predict negative outcomes is essential to survival. This can be studied experimentally by pairing an initially neutral cue (e.g., a tone) with an aversive unconditioned stimulus (US; e.g., electric shock). With sufficient pairings, the tone becomes a conditioned stimulus (CSϩ) that predicts the occurrence of the US and elicits emotional responses in the absence of the US. This type of Pavlovian fear conditioning requires the amygdala (Maren et al., 1996), but the mechanisms by which the CSϩ acquires the capacity to predict impending aversive events are not known.
In contrast, there is strong evidence that predictions about impending reward are encoded by midbrain dopaminergic (DA) signals. Initially, DA neurons are strongly activated by US delivery, but as the animal learns that the CSϩ is a reliable predictor of the US, US evoked firing decreases concomitantly with an increase in CSϩ evoked firing. When an expected US is withheld, DA neurons inhibit their firing (Schultz et al., 1997). Thus, DA neurons encode the disparity between expected and actual reward, i.e., the prediction error. Notably, however, DA neurons do not respond in this way when the US is unpleasant and therefore do not encode aversive prediction errors (Fiorillo, 2013).
So where are aversive prediction signals found in the brain? In rats, amygdala neurons reduce their firing-rate to a repeatedly encountered footshock, consistent with a role in predicting aversive events. However, unlike DA cells, the same amygdala neurons do not exhibit a concomitant increase in CSϩ (auditory cue) evoked firing nor do they reliably inhibit their firing below baseline when the expected aversive outcome is withheld . Thus, the extent to which aversive prediction signals are represented in amygdala single-unit activity is unclear.
Human neuroimaging studies have also failed to establish whether aversive prediction signals are present in the amygdala. Some studies report higher amygdala blood-oxygen level-dependent (BOLD) responses to an unexpected versus an expected or blocked US, consistent with a role in predicting aversive events (Dunsmoor et al., 2008;Eippert et al., 2012;Wood et al., 2012;Metereau and Dreher, 2013). However, several studies explicitly investigating aversive prediction signals do not report significant amygdala involvement at all (Seymour et al., 2004;Menon et al., 2007;Schiller et al., 2008). These discrepant findings may reflect methodological differences. Alternatively, regions outside of the amygdala may generate and process these signals (McNally et al., 2011;Kobayashi, 2012).
To investigate aversive prediction error signals, here we recorded hemodynamic responses and local field potentials (LFPs) from the basolateral amygdala in freely moving mice during discriminative Pavlovian fear conditioning. Hemodynamic responses were recorded in the form of tissue oxygen (T O2 ) signals, which are driven by the same physiological mechanisms as the BOLD signal, and therefore provide a close proxy measure (Offenhauser et al., 2005;Logothetis, 2007;Lowry et al., 2010;. Positive aversive prediction errors were investigated during fear learning and negative aversive prediction errors were investigated during fear extinction, when the expected footshock was omitted.

Subjects
This study used 24 male mice on a CBA ϫ C57bl/6 background that were 5-to 6-months-old at the time of surgery. Mice were maintained on a 12 h light/dark cycle (lights on 7:00 A.M.; off 7 pm) with ad libitum food and water. Mice were housed in groups of two to four before surgery and individually after surgery. The experiments were conducted in accordance with the United Kingdom Animals Scientific Procedures Act (1986) under project licenses 30/2561 and 30/3068 and were approved by local ethical review.

Surgery
Under isoflurane anesthesia, mice were implanted with a carbon paste electrode (CPE; gifted by Lilly UK, Surrey, UK) into the basolateral amygdala (BLA) of one hemisphere to measure tissue oxygen and a silver wire electrode into the BLA of the contralateral hemisphere to measure LFPs (approximately equal numbers received left CPE/right LFP and right CPE/left LFP). CPEs were constructed from Teflon-coated 200m-diameter silver wire (ϳ270 m coated diameter, Advent Research Materials) with the insulation pulled down the wire to produce a 2 mm cavity which was subsequently packed with carbon paste and smoothed (O'Neill et al., 1982). LFP electrodes were made from 125-m-diameter silver wire (ϳ177 m coated diameter, Advent). Coordinates for BLA implantations were Ϫ1.35 mm anterior/posterior, Ϯ3.10 mm medial/ lateral and Ϫ5.00 mm dorsal/ventral, relative to bregma. Auxiliary and reference electrodes (200-m-diameter silver wire) were implanted into parietal cortex. A ground electrode was wrapped around a skull screw. Each electrode was soldered to a gold pin (E363/0, Plastics One), which was inserted into a pedestal plug (MS363, Plastics One) and secured with skull screws and dental cement ("Simplex Rapid," Associated Dental Products). Mice were allowed to recover for at least 7 d after surgery.

Apparatus
T O2 was measured using constant potential amperometry, as described previously in detail (Bolger et al., 2011;Barkus et al., 2014). A constant potential (Ϫ650 mV relative to a reference electrode) was applied to the electrode using a low-noise potentiostat (Biostat, ACM Instruments). The applied potential produces the electrochemical reduction of dissolved O 2 on the surface of the electrode, inducing an electrical current, which is measured by the potentiostat. The availability of O 2 for this two-step reaction (O 2 ϩ 2H ϩ ϩ 2e Ϫ 3 H 2 O 2 ; H 2 O 2 ϩ 2H ϩ ϩ 2e Ϫ 3 2H 2 O) is determined by the local T O2 concentration. Thus, changes in O 2 concentration around the tip of the electrode produce directly proportional changes in the measured Faradaic current (Hitchman, 1978). The spatial resolution is estimated to be a sphere with diameter approximately twice the diameter or the electrode, i.e., 400 m ( Thompson et al., 2003;Li et al., 2011b).
Mice were connected to the potentiostat via a six-channel rotating commutator (SL6C, Plastics One) held on a counter-weighted arm (PHM-110P1, Med Associates) using screened cables (363-363 6TCM, Plastics One). A Powerlab 8/30 interface (AD Instruments) was used for analog/digital conversion and data were collected on a Windows PC running Chart v5 software (AD Instruments). LFPs were recorded using a differential amplifier (DP-301, Warner Instruments) or using the potentiostat with no potential applied, in which case the potentiostat acts as a differential amplifier. T O2 and LFPs were sampled continuously at 4 kHz.
Fear conditioning was conducted in one of three operant chambers (ENV-307A, Med Associates), each with distinct visual and olfactory cues (Fig. 1A). Stimulus delivery was controlled by a custom-written script in the MED-PC language. Timed transistor-transistor logic pulses to the AD converter ensured that stimulus delivery was synchronized with the electrophysiological recordings at 1 ms resolution.

Procedure
Twenty-four mice underwent fear conditioning. The procedure on each day was virtually identical. First, mice were connected to the recording equipment and placed in the "neutral context," i.e., a chamber in which the mouse had been previously habituated for at least 2 h and in which they never received shocks. The potential was then applied to the T O2 electrode for 10 min before the experiment began to ensure that T O2 signals were stable. Pre-exposure (day 1) was performed entirely in the neutral context: five tone (2900 Hz) and five white noise stimuli (both 30 s duration, 80 dB) were played in pseudorandom order with a mean ITI of 80 s (range 60 -100 s), with no shocks administered. There were no differences in behavioral, T O2 , or LFP responses to the auditory cues before training and these results are not reported further. On training days 1-3 (experimental days 2-4) mice were placed into the neutral context for 10 min and then transferred to one of the conditioning chambers (e.g., context A or B, counterbalanced). Mice then received five tone and five white-noise stimuli presentations, the same as pre-exposure, but one of the stimuli (counterbalanced across mice) was always paired with coterminating footshock (0.3 mA, 0.5 s).
After fear conditioning, in one subset of mice (n ϭ 8) we investigated amygdala responses to expected versus unexpected footshock (day 5). Mice were placed into a novel conditioning chamber (e.g., context B if trained in context A) and underwent a probe session containing expected shocks (i.e., CSϩ followed by footshock), unsignaled shocks (i.e., footshock occurring in the absence of any auditory cue), and presentations of the CSϪ without footshock. Three of each trial type were used and presentation order was pseudorandomized and counterbalanced across mice. Behavioral data and LFP recordings were obtained from eight mice and T O2 data were obtained from seven of these mice.
A separate subset of mice underwent either one (n ϭ 16) or two (n ϭ 10) fear extinction sessions (days 5 and 6). On each session, mice were placed first into the neutral context for 10 min and then placed into a novel conditioning chamber (e.g., context B if trained in context A) and the five tone and five white noise stimuli were presented with no shocks administered (Figs. 1C,D).

Histology
Mice were injected with sodium pentobarbitone; 200 mg/kg) and perfused transcardially with physiological saline (0.9% NaCl) followed by 10% formol saline (10% formalin in 0.9% NaCl). Their brains were removed and placed in 10% formol saline for 3 d, and then transferred to a 30% sucrose-formalin solution for 24 h and frozen. Coronal sections (40 m) were then cut on a freezing microtome and stained with cresyl violet to enable visualization of the electrodes. Only mice with electrodes in the basolateral amygdala were included in the analyses (Fig. 1B).

Data analysis
T O2 responses. Auditory cue-evoked T O2 responses were calculated by subtracting the mean T O2 signal during the 5 s before cue onset (i.e., baseline) from the T O2 signal during the 30 s cue presentation. The magnitude of US evoked responses was also calculated by subtracting the mean T O2 signal during the 5 s before shock onset, which in all cases except unsignaled shock would be the last 5 s of CS presentation. Then, the 30 s signal was divided into fifteen 2 s time bins (i.e., 0 -2 s, 2-4 s, 4 -6 s…28 -30 s), with each data point equal to the mean value during each 2 s time bin (Fig. 1E). For analysis, we averaged T O2 responses over the five (or three in the case of the probe day) CSϩ, CSϪ, US, or US omission periods of each session and determined the time bin with the highest value (max T O2 ; Fig. 1E).
On the extinction days, we were interested in whether the T O2 signal decreased when the expected footshock was omitted and so we did not use the maximum value for analysis. Instead we analyzed how the signal changed in the 30 s following CS offset by dividing the 30 s T O2 signal into fifteen 2 s time bins and directly comparing responses following CSϩ versus CSϪ offset (ANOVA model: day 2(extinction day 1, extinction day 2) ϫ stimulus type 2(CSϩ, CSϪ) ϫ timebin 15 , n ϭ 10 mice). A decrease below baseline could occur simply as a consequence of the offset of the auditory cue but if this was the case, then this decrease should be equivalent following CSϩ and CSϪ presentations. Note that the baselines for shock omission (and the equivalent period following CSϪ presentations) were based on the mean signal during the last 5 s of CSϩ or CSϪ presentation and so values below zero during the shock omission periods indicate signals lower than those at the end of CS presentation.
LFPs. Stimulus-evoked LFPs were bandpass filtered between 1 and 45 Hz. We calculated power spectra during the 10 s after CS onset, US onset, or US omission (i.e., the 10 s following CS offset). Note that the 0.5 s period during shock administration was not contained in these epochs. Spectra were averaged over the five CSϩ, five CSϪ, five US, or five US omission periods for each mouse for each day of training and extinction. Spectra were computed in MATLAB (Mathworks) using a fast Fourier Transform size of 2000 samples (with a Hamming window, 50% overlap) at a sampling rate of 1 kHz, giving a frequency resolution of ϳ0.5 Hz. For statistical analysis, the power spectral density in each frequency bin (⌽ i, ) was transformed into a proportion of the total power between 1 and 45 Hz: where P i ϭ proportional power, ⌽ i ϭ raw power density (mV 2 /Hz). From these proportional spectra, we determined the peak power in the theta frequency range (5-10 Hz) as shown in Figure 1F. Spectrograms (time ϫ frequency ϫ power) were generated in MATLAB, using a sliding time window of 1 s, with 500 ms overlap, a frequency resolution of ϳ0.25 Hz, and the "jet" color map (blue, lower-power; red, higher-power). To analyze US omission, we used proportional spectra (as described above) and normalized spectrograms, which were obtained by z-transforming the frequency dimension of each spectrogram as follows: where Z ij is the power at the i th frequency and j th time bin expressed in SD units, X ij is the raw power at the i th frequency and j th time bin, X i is the mean power at the i th frequency across all time bins, and SD(X i ) is the SD Circles denote mice that underwent extinction; squares denote mice that underwent the probe session containing expected and unexpected shocks. C, Schematic of the experimental sessions. Twenty-four mice underwent fear conditioning, eight of these also underwent the probe session. The remaining 16 mice underwent one extinction session, with 10 of these doing two sessions. D, Fear conditioning sessions consisted of a 10 min pre-session period, followed by a 23 min session with 10 auditory cues (5 ϫ tone, 5 ϫ white noise), presented in pseudorandom order. On training days, one of the cues (CSϩ) was always paired with coterminating footshock; the other cue (CSϪ) was never paired with footshock. No shocks were given during the pre-exposure or extinction sessions. During the probe session three CSϩ (with shock), three CSϪ (no shock), and three unsignaled shocks were given. E, Left, Example of raw T O2 data during a CSϩ presentation (CS onset at 0 s, offset at 30 s). E, Right, T O2 data transformed into ⌬T O2 signal by subtracting the baseline (mean T O2 signal in the 5 s before CS onset) and binning the raw T O2 signal into fifteen 2 s epochs. Arrow shows the peak T O2 signal. F, Left, Example power spectral density (PSD) for an LFP signal in the pre-CSϩ period (blue trace), and the CSϩ period (red trace). F, Right, Proportional PSD calculated by expressing the power in each frequency bin as a proportion of the total power between 1 and 45 Hz. Arrow shows peak theta power.
at the i th frequency across all time bins. For analysis, we calculated the mean z-transformed power in the 10 s before CSϩ onset and compared this to the mean z-transformed power in the 10 s period after CSϩ offset (i.e., in the 10 s after an expected footshock was not delivered). Because of noise in some LFP recordings on extinction day 2, we restricted LFP analysis of US omission to the first extinction day. In addition, on the probe day in Experiment 2, we excluded the first expected shock and the first unexpected shock trials due to noise issues in the LFP recordings.
Behavior: locomotor and freezing data. Locomotor activity was monitored via a video camera in the roof of the operant chamber and measured using a script in NIH Image (Schneider et al., 2012), which compared consecutive video frames for pixel changes (1 Hz sampling rate, therefore 1 s resolution). Freezing was determined on the frame-toframe percentage pixel change being below a set threshold, calibrated for an absence of movement except for breathing (Richmond et al., 1998). If the percentage pixel change was above threshold then a score of 0 was given, if it was below threshold then a score of 1 was given. These scores were converted into percentages by summing the values during CS presentation, dividing by the CS duration, and multiplying by 100. To reduce the impact of freezing evoked by the training context, a "difference score" was calculated such that percentage freezing during the 30 s before cue presentation was subtracted from the percentage freezing during the 30 s cue presentation (i.e., positive freezing scores indicate increased freezing to the cue and negative freezing scores indicate decreased freezing to the cue relative to the pre-cue period). For the regression analyses, we used a freezing discrimination score based on the absolute difference between CSϩ evoked freezing minus CSϪ evoked freezing. For example, if a mouse had a percentage pixel change below threshold in 18 of the 1 s time bins during a 30 s CSϩ presentation, this would yield a freezing score of 60% [i.e., (18/30) ϫ 100 ϭ 60]. If the equivalent freezing during CSϪ presentation was 40%, then the freezing discrimination score would be: 60 -40% ϭ 20%.
Statistical procedures. Data were analyzed using t tests, ANOVA, or multiple linear regression in SPSS. ANOVAs are described using a modified version of Keppel's notation (Keppel, 1982) in which the dependent variable is defined in the form: A 2 ϫ B 3 , where A is a factor with two levels and B is a factor with three levels. All graphs show the mean Ϯ 1 SEM. The familywise error was set at ␣ ϭ 0.05.

Behavioral responses during discriminative fear learning
Over the 3 d of training, the mice learned to discriminate between the auditory cue that predicted footshock (CSϩ) and the other auditory cue that did not predict footshock (CSϪ), such that freezing levels were markedly higher during CSϩ than CSϪ presentations (Fig. 2). During training, mice froze less during CSϪ presentations compared with the 30 s before CSϪ onset (hence negative scores in Fig. 2A) and froze more during CSϩ presenta-tions compared with the 30 s before CSϩ onset (hence positive scores). Analysis of CSϪ versus CSϩ evoked freezing responses during training (ANOVA: CS type 2(CSϪ, CSϩ) ϫ day 3 , n ϭ 24 mice) confirmed that mice froze more during CSϩ presentations (CS type ϫ day interaction: F (2,46) ϭ 4.8, p ϭ 0.01, CSϩ vs CSϪ on day 3, p ϭ 0.03). In short, the mice learned that the CSϩ, and not the CSϪ, predicted the aversive outcome.

Aversive prediction errors in amygdala hemodynamic responses
Amygdala T O2 responses changed markedly over the 3 d of training such that responses to the CSϩ increased, responses to the footshock (US) decreased, whereas responses to the CSϪ changed little (Fig. 2 B, C). We analyzed these data using ANOVA with within-subjects factors of CS type (CSϩ, CSϪ), CS phase (cue phase, shock phase), and day (days 1, 2, 3). Crucially, there was a three-way interaction between these factors (F (2,46) ϭ 4.7, p ϭ 0.01), allowing us to perform further analyses to investigate the changes in T O2 responses over training.
The first thing to note is that T O2 responses evoked by the CSϩ were significantly higher than those evoked by the CSϪ and T O2 responses following footshock were higher than those in the "no US" period following the CSϪ (CS type ϫ CS phase interaction: F (1,23) ϭ 4.6; p ϭ 0.04, CSϩ Ͼ CSϪ, p ϭ 0.04; US Ͼ no US, p Ͻ 0.001), consistent with the behavioral data showing higher freezing levels during CSϩ trials.
Second, and most importantly, amygdala T O2 responses evoked by the CSϩ and the US changed systematically over training. On training day 1, responses evoked by the US were higher than those evoked by the CSϩ but by training day 3, the US evoked response had significantly decreased, whereas the CSϩ evoked response had significantly increased (Fig. 2 B, C). Analysis (ANOVA: stimulus type 2(CSϩ, US) ϫ day 3 , n ϭ 24 mice) revealed an interaction between stimulus type and day (F (2,46) ϭ 6.4, p ϭ 0.003). US evoked responses were significantly higher during day 1 than day 3 ( p ϭ 0.008), whereas CSϩ evoked T O2 responses were significantly higher during day 3 than day 1 ( p ϭ 0.005). These patterns suggest that amygdala hemodynamic responses contain aversive prediction error signals.
Moreover, the systematic changes in CSϩ and US evoked T O2 responses were not seen for CSϪ evoked responses and the equivalent no US period during the 30 s after CSϪ offset (no stimulus type ϫ day interaction: F (2,46) ϭ 0.4, p ϭ 0.7; Fig. 2 B, C). Although CSϪ evoked signals were higher than those during the no US period, this did not reach significance (F (1,23) ϭ 4.0; p ϭ 0.06). Importantly, T O2 signals in the CSϪ and no US period did not change across days (all pairwise comparisons p Ͼ 0.5), arguing that the prediction error-like changes were specific for CSϩ and US evoked amygdala T O2 responses.

Aversive prediction error signals in theta oscillations
The onset of the CSϩ and the US evoked theta frequency oscillations (5-10 Hz), and theta power changed in a similar way to T O2 responses over training (Fig. 3). As with T O2 signals, analyses of theta oscillations revealed a three-way interaction between CS type, CS phase, and day (F (2,46) ϭ 13.1, p Ͻ 0.001). Three key findings emerged.
Second, like the amygdala T O2 responses, US onset evoked an increase in theta power on training day 1 which diminished over the 3 d of training. Conversely, theta oscillations evoked by the CSϩ increased in power over this period (Fig. 3A-C). Analysis (ANOVA: stimulus type 2(CSϩ, US) ϫ day 3 , n ϭ 24 mice) confirmed these observations (stimulus type ϫ day interaction, F (2,46) ϭ 10.9, p Ͻ 0.001). CSϩ evoked theta power was higher on day 3 compared with day 1 ( p ϭ 0.03), whereas US evoked theta power was higher on day 1 compared with day 3 ( p ϭ 0.045). Thus, positive aversive prediction errors are represented via a decrease in US evoked theta oscillations and a concomitant increase in CSϩ evoked theta oscillations as the mouse learns that the CSϩ predicts the US.
Third, analysis of CSϪ evoked responses and the no US epoch after CSϪ offset (i.e., the equivalent period to that following US  delivery on CSϩ trials) also yielded a stimulus type ϫ day interaction (F (2,46) ϭ 3.3, p ϭ 0.047). Theta power was higher during CSϪ presentations on day 1 compared with day 2 ( p ϭ 0.04) and day 3 (p ϭ 0.02), and therefore showed the opposite pattern to CSϩ evoked theta oscillations (Fig. 3D-F ). Moreover, theta oscillations during the no US period did not show any change over days (all p Ͼ 0.5). In short, unlike the CSϪ and no US evoked signals, theta oscillations changed systematically to the CSϩ and the US during training in a manner congruent with an aversive prediction error signal.
The changes we observed in amygdala T O2 responses and theta oscillations could not be explained by changes in locomotor activity (Fig. 4 A, B). Whereas CSϩ evoked T O2 responses and theta oscillations increased over training, locomotor activity during CSϩ presentations decreased over training (day 1 vs day 3: p ϭ 0.03). Moreover, whereas US evoked T O2 responses and theta oscillations significantly decreased over training, the burst in locomotor activity evoked by the US (i.e., the unconditioned response to the shock) remained constant over training (day 1 vs day 3: p ϭ 0.6; Fig. 4B).

Responses to expected versus unexpected shock
Fear learning resulted in decreased T O2 responses and theta oscillations to the footshock over training, consistent with an aversive prediction error signal. However, this decreased response could reflect habituation to the shock, rather than a prediction error per se. To test this, a subset of the fear-conditioned mice experienced a probe session in which they were given both expected shocks (i.e., shocks signaled by CSϩ presentations) and unexpected shocks (i.e., shocks unsignaled by any auditory cue). If these signals contained aversive prediction errors, then responses to unexpected shock should be greater than those to expected shock.
Unexpected shock evoked significantly higher amygdala T O2 responses than expected shock (t (6) ϭ 3.8, p ϭ 0.01; Fig. 5 A, B). Moreover, unexpected shock was associated with increased theta oscillations (8 -12 Hz) compared with expected shock (t (7) ϭ 2.4, p ϭ 0.05; Fig. 5C). This difference in amygdala responses was not due to locomotor activity differences because expected shock and unexpected shock evoked almost identical locomotor responses (mean movement during 30 s postshock: t (7) ϭ 0.2, p ϭ 0.8; maximum movement during 30 s postshock: t (7) ϭ 0.3, p ϭ 0.8; Figure 5D). In short, the decrease in US evoked amygdala responses with training reflect expectation of the shock and do not simply reflect habituation to the shock.

Responses during fear extinction learning
In the remaining fear conditioned mice, we investigated what happened when the expected footshock was omitted, to see whether the amygdala signals represent negative aversive predictions errors. Over the two extinction sessions, behavioral discrimination between the CSϩ and CSϪ remained relatively constant (Fig. 6A). Analysis (ANOVA: stimulus type 2(CSϪ, CSϩ) ϫ trial 10 , n ϭ 10 mice) confirmed there was significantly higher freezing during CSϩ than CSϪ presentations (main effect of stimulus type: F (1,9) ϭ 7.0, p ϭ 0.03), but there was no evidence of extinction learning over the 10 unreinforced CSϩ trials (no effect of trial, or stimulus type ϫ trial interaction: F Ͻ 1.6, p Ͼ 0.1).

Negative aversive prediction errors in amygdala hemodynamic responses
During extinction, the omission of expected footshock led to a negative T O2 response, i.e., a decrease below baseline (Fig. 6 B, C).
When the expected US was not delivered at the end of CSϩ presentations, T O2 signals decreased to a level that was significantly lower than the equivalent period following CSϪ offset (CS type ϫ time bin interaction, F (14,126) ϭ 2.0; p ϭ 0.03; CSϩ Ͻ CSϪ, 16 -20 s after expected US was omitted; note that the baselines for this analysis were based on the last 5 s of CSϩ or CSϪ presentation and so US omission resulted in a T O2 signal decrease below levels at the end of CSϩ presentation). Importantly, the decrease in T O2 signals following US omission was also below the pre-CSϩ baseline (t (9) ϭ 2.7, p ϭ 0.03; Fig. 6C). Thus, negative aversive prediction errors are also present in amygdala hemodynamic responses.
Omission of expected footshock was also associated with reduced theta power compared with the CSϩ period (and the pre-CSϩ period), and this reduction was focused within the same narrow frequency band that was elevated during CSϩ presentation (Fig. 6D-G). Analysis of power spectra (ANOVA: phase (before omission, after omission)2 ϫ freq 19 , n ϭ 16 mice) revealed a significant reduction in theta power in the 10 s after US omission compared with the 10 s before US omission (phase ϫ frequency interaction, F (18,270) ϭ 2.3, p ϭ 0.003, 6 -6.5 Hz, p Ͻ 0.05; Fig. 6D). Moreover, when analyzing the z-transformed power spectra (see Materials and Methods), theta power after US omission was lower than theta power in the period before CSϩ onset (based on theta power at 6 Hz: t (14) ϭ 2.5, p ϭ 0.03; Fig. 6E-G; note this was also the case for the averaged power between 6 and 7 Hz: t (14) ϭ 2.3, p ϭ 0.04). The decreased amygdala signals during US omission could not be explained by locomotor activity changes because there was increased movement following US omission (Fig. 4C). Thus, negative aversive prediction errors are associated with significantly reduced theta power below baseline.  Figure 5. A, T O2 signals were higher to an unexpected compared with an expected footshock (i.e., one that was preceded by the CSϩ). Arrow indicates onset of footshock. B, Peak T O2 data responses to expected and unexpected shock. Although the error bars overlap, the difference (unexpected minus expected) was significantly greater than zero (gray bar). C, Theta power (8 -12 Hz) was higher for unexpected than expected shock, again with the difference shown in the gray bar. D, Locomotor responses did not differ between the expected and unexpected shocks. A-D, Mean Ϯ SEM. A, B, n ϭ 7 mice; C, D, n ϭ 8 mice; *p Ͻ 0.05.

Amygdala responses to footshock predict subsequent fear-related behavior
Finally, we used multiple linear regression to see whether T O2 signals on day n could predict behavioral discrimination between the aversive (CSϩ) and nonaversive (CSϪ) conditioned cues on day n ϩ 1. We calculated a simple behavioral discrimination score for each mouse based on the absolute difference in percentage freezing during CSϩ and CSϪ presentations (i.e., CSϩ freezing minus CSϪ freezing, see Materials and Methods). Three independent variables were used in the regression model: the maximum T O2 signal during CSϩ presentations (T O2 _CSϩ), the maximum T O2 signal during CSϪ presentations (T O2 _CSϪ), and the maximum T O2 signal in the 30 s following footshock delivery (T O2 _US). In the regression model, T O2 _CSϩ, T O2 _CSϪ and T O2 _US on training days 1, 2, and 3 were used to predict behavioral discrimination on training days 2 and 3 and the extinction session, respectively. Stepwise linear regression was used to determine the maximum variance explained with the fewest independent variables.
The optimum regression model contained only T O2 _US and accounted for 18% of the variance in behavioral discrimination, which was significantly greater than zero (F (1,46) ϭ 11.5, p ϭ 0.001, adjusted R 2 ϭ 0.18). The correlation between T O2 _US and behavioral discrimination was positive, with the standardized ␤ coefficient ϭ 0.45 (t (46) ϭ 3.4, p ϭ 0.001). In other words, mice who exhibited a larger T O2 response to the footshock showed better behavioral discrimination the following day.
We repeated this regression analysis using the maximum theta power during CSϩ, CSϪ, and US presentations (_CSϩ, _CSϪ, _US, respectively). As above, the optimum model con-tained only _US, accounting for 7% of the variance which was significant (F (1,46) ϭ 4.4, p ϭ 0.04, adjusted R 2 ϭ 0.07). The correlation between _US and behavioral discrimination was positive, with the standardized ␤ coefficient ϭ 0.3 (t (46) ϭ 2.1, p ϭ 0.04). In short, both theta oscillations and T O2 responses to the footshock predicted subsequent behavioral discrimination. Unfortunately, because we observed little in the way of extinction, we could not use this regression approach to test whether the negative T O2 and theta responses following US omission predicted the amount of extinction learning.

Summary of results
Here we show that hemodynamic responses and theta oscillations recorded from the amygdala show activity patterns consistent with aversive prediction error signals. The significance of these findings is threefold. First, prediction errors are necessary for learning and the amygdala is necessary for fear learning; and yet the evidence for amygdala aversive prediction error signals has been equivocal. Our data demonstrate for the first time that aversive prediction error signals, analogous to appetitive prediction error signals found in DA neurons, can been seen in the amygdala, even though these may not be detectable at the singleunit level . Second, this is the first study to track theta oscillations evoked by both the CSϩ and the US over the course of fear learning. Our data show that the changes in CSϩ and US evoked theta oscillations are systematic and proceed in opposite directions as the mouse learns that the CSϩ predicts the US. Furthermore, omission of an expected US led to reduced theta oscillations. These patterns of theta oscillations provide a  Ext2). B, C, T O2 signals evoked by the absence of an expected footshock (US omission) were significantly lower than pre-US omission baseline signals and the pre-CSϩ baseline. This was not the case for the equivalent CSϪ period. D, Power spectra illustrating lower theta power in the 10 s following US omission (USo, black line) versus the 10 s before US omission (pre-USo, gray line). E, Z-transformed, group-averaged spectrogram illustrating increased theta power in a narrow frequency range (ϳ6 Hz) during CSϩ presentation and the marked decrease of activity in this range following US omission. F, As in E, but for the equivalent CSϪ period. G, Theta activity at 6 Hz (expressed in SD units) was significantly lower following US omission compared with the pre-CSϩ period. A-D, G, Mean Ϯ SEM. A-C, n ϭ 10 mice, D-G, n ϭ 16 mice; *p Ͻ 0.05.
novel neural correlate for fear learning. Third, we found that mice who exhibit strong T O2 and theta responses to the footshock (US) show better behavioral discrimination between the aversive (CSϩ) and the nonaversive (CSϪ) conditioned cues the following day. Thus, the magnitude of US evoked responses may reflect the processing or attention directed to the US, strengthening the association between the predicting cue and the aversive outcome.

Linking rodent T O2 signals with human neuroimaging
This is the first study to show that amygdala T O2 responses in mice represent aversive prediction errors. T O2 and BOLD signals are closely related, for example, both are hemodynamic responses dependent upon increased cerebral blood flow (Ogawa et al., 1990;Lowry et al., 1997), allowing for cross-species comparisons that are not usually possible. Consistent with our data, several human neuroimaging studies have reported reduced US evoked BOLD responses over training and greater BOLD responses to an unexpected or partially predicted US versus an expected, blocked, or fully predicted US (Dunsmoor et al., 2008;Eippert et al., 2012;Wood et al., 2012;Metereau and Dreher, 2013). However, other neuroimaging studies investigating aversive prediction errors do not report significant amygdala involvement (Seymour et al., 2004;Menon et al., 2007;Schiller et al., 2008). These failures could reflect methodological issues. First, it is unclear whether the scanning parameters were optimized to measure amygdala responses. The amygdala exhibits small BOLD responses (Norbury et al., 2010) and is vulnerable to susceptibility artifacts (Morawetz et al., 2008). Moreover, intersubject variability in human amygdala volume makes normalization to a standard atlas problematic (Robinson et al., 2004). Consequently, amygdala activations may be missed in whole-brain analyses. Second, the typical CSϩ duration (i.e., the interval between CSϩ onset and US onset) is Ͻ10 s, whereas the hemodynamic response evolves over Ͼ10 s. Therefore, it is difficult to disambiguate responses due to CSϩ versus US onset on reinforced trials. Consequently, many studies have used partial reinforcement between CS and US and only analyzed responses from nonreinforced CSϩ trials. However, partial reinforcement limits the attenuation of the US response and reduces CSϩ evoked responses over training (Dunsmoor et al., 2007(Dunsmoor et al., , 2008. To avoid these issues, in the present study we recorded hemodynamic responses directly from the amygdala, used a CS duration of 30 s, and used 100% reinforcement between the CSϩ and US during training. One recent study has argued that amygdala BOLD signals encode the associability between the CS and US rather than the prediction error (Li et al., 2011a). Our T O2 data are not consistent with this interpretation. Specifically, their model incorporates part of Pearce-Hall theory, which states that associability will decrease over trials as the organism learns that the CSϩ is a good predictor of the US (Pearce and Hall, 1980), and therefore predicts a decrease in CSϩ evoked amygdala signals over training, contrary to what we observed. This could reflect a species difference but there are also important methodological differences. First, Li et al. (2011a) used angry faces as the conditioned stimuli, which are known to evoke unconditioned amygdala responses even before being paired with an aversive outcome (i.e., pairing angry faces with shock may represent a US-US association, rather than CS-US association). Second, Li et al. (2011a) used partial reinforcement (33%) and a short CS duration of 4 s, which complicates the interpretation of US and US omission evoked responses, as discussed above. Third, Li et al. (2011a) did not analyze amygdala responses to the US. These methodological differences limit comparison with our study. Seymour et al. (2005) report that amygdala BOLD signals are consistent with appetitive but not aversive prediction errors. In their paradigm, participants experienced persistent pain and visual cues predicted either the exacerbation (aversive signal) or relief (appetitive signal) of pain. Their failure to see aversive prediction errors could be due to the participants experiencing persistent pain rather than distinct aversive versus nonaversive epochs.

Theta oscillations and the amygdala
Several studies have investigated cue (CSϩ) evoked theta oscillations during fear learning (Seidenbecher et al., 2003;Narayanan et al., 2007) but we are the first to show systematic (and opposite) changes in CSϩ and US evoked theta during fear learning. However, although our study recorded oscillations from electrodes in the basolateral amygdala, theta may be generated elsewhere. Most models of neuronal oscillation assume that there must be spatial separation of current sinks and sources (to produce a dipole), and that these must be in coherently aligned neurons for oscillations to be observed in the LFP. As a nonlaminar structure, it is unclear whether the cellular architecture of the basolateral amygdala allows for the generation of theta oscillations. Nevertheless, intracellular theta oscillations can be induced in amygdala neurons by near-threshold membrane depolarization (Pare et al., 1995;Pape et al., 1998) and lateral amygdala single-units are modulated by theta oscillations during discriminative fear conditioning (Pare and Collins, 2000). Thus, amygdala neurons are influenced by theta, even if it is not generated locally.

What is the function of theta oscillations?
There are several theories about the function of theta, and these are not mutually exclusive. First, theta may synchronize activity across different structures to facilitate reciprocal information transfer (Buzsáki, 2002;Seidenbecher et al., 2003;Sirota et al., 2008). Second, theta oscillations influence spike timing and so can influence synaptic plasticity (Song et al., 2000). Third, theta may set the "gain" (attention/arousal) for stimuli, boosting the signal without influencing its content, which could also facilitate learning (Sejnowski and Paulsen, 2006).

Relationship between amygdala signals and freezing behavior
How do CS and US evoked signals relate to the behavioral correlates of learning? We found that mice with larger US evoked T O2 and theta responses exhibited better behavioral discrimination between the aversive (CSϩ) and nonaversive (CSϪ) conditioned cues the following day. This result was surprising because, at a group level, US evoked responses diminished over the course of learning whereas behavioral discrimination improved, suggesting that lower US evoked signals might indicate greater US expectancy and hence greater likelihood of CSϩ/CSϪ discrimination. However, our results suggests that, at the level of individual mice, higher US evoked responses may reflect greater attention to the US, which in turn may facilitate the association between the CSϩ and the US. One important caveat in our findings is that we could not relate the amygdala negative prediction error signals to behavior because we did not see any significant loss of fear across the two extinction sessions. Future studies using prolonged extinction may be required to investigate this further.
What is the origin of the aversive prediction signal? Why do we see aversive prediction signals if they are not present in amygdala single-unit activity (Belova et al., 2007;? First, hemodynamic responses and LFPs sample from a larger spatial area and over a longer temporal window, pooling activity from many neurons. Second, hemodynamic responses and LFPs predominantly reflect the input to a region whereas unit activity reflects output (Logothetis et al., 2001). Thus, amygdala T O2 responses and LFPs may reflect aversive prediction signals generated elsewhere. There is good evidence implicating the periaqueductal gray in the generation of these signals, although it does not project directly to the amygdala (McNally and Cole, 2006;Johansen et al., 2010). Another candidate structure for generating aversive prediction error signals is the habenula. Cells in the primate lateral habenula increase their firing to a CS that predicts aversive events and fire more to unpredictable than predictable aversive events (Matsumoto and Hikosaka, 2009). However, lesions of the lateral habenula do not prevent fear learning (Heldt and Ressler, 2006) and the habenula does not project directly to the amygdala (Kim, 2009). An alternative theory is that serotonergic neurons might provide aversive prediction signals, although evidence for this theory is currently limited (Daw et al., 2002).

Conclusion
The amygdala is necessary for both the acquisition and extinction of conditioned fear. By demonstrating that positive and negative aversive prediction errors are represented in the amygdala, we provide new insights into the mechanisms of fear learning.