Dynamic Encoding of Action Selection by the Medial Striatum

Subjects.

Seventeen male Long–Evans rats (Harlan) were individually housed and kept on a 12 h light/dark cycle with lights on at 7:00 A.M. All procedures were approved by the Institutional Animal Care and Use Committee at the John B. Pierce Laboratory. To motivate behavior, rats had restricted access to water for 18 h before behavioral sessions. They were given ad libitum access to water at least 1 d every week. Food was always available ad libitum. Rats earned about half of their daily water during the behavioral sessions, and the remaining volume of the animal's average water consumption was provided in the home cage, 1–2 h after the behavioral sessions. Rats maintained 90–95% of their free water body weight throughout the training and testing period. Electrodes were implanted in all rats (see below). However, one rat did not survive the surgery and the implanted electrodes in two rats yielded poor recordings. All neuronal analyses are therefore based on the remaining 14 rats.

Operant chamber.

Initial training took place in a standard operant chamber (ENV-008, Med Associates). One wall had a central nosepoke hole with an infrared beam to identify nosepoke entry (ENV-114BM, Med Associates). The opposite wall had a metal spout connected to a water pump and a custom-built electronic circuit, which detected contact by measuring resistance between the spout and the floor. Above the spout was a speaker (ES1, TDT Technologies) for presentation of acoustic stimuli (RP2.1 Processor, TDT Technologies). Tones and noise (white noise generated via the TDT DSP System) were presented from the speaker above the waterspout at 65 dB. The speaker and the nosepoke each had a light above them (ENV-221M, Med Associates). A light stimulus consisted of turning on both lights simultaneously. Except for when lights were used as a brief stimulus, behavior occurred in the dark. Protocols in this chamber were controlled using Med Associates software (MedPC).

The recording chamber was similar to the standard operant box but modified for electrical recordings (Med Associates). All walls and floor bars were made of acrylic plastic; the long walls sloped diagonally outward as they went up. A copper shielded electrostatic speaker (ES1, TDT Technologies) was used to present the tones and noise. Light stimuli were made from LEDs covered with sand-blasted plastic discs that were 1 inch in diameter (Pierce Laboratory Instruments Shop). The waterspout was made of glass and licking was registered by break of an infrared beam in front of the spout (Pierce Laboratory Instruments Shop). A 32-channel commutator (Plexon) was mounted in the center of the ceiling of the chamber. This device connected cables from the implanted probes to the recording system. The chamber was placed on a steel plate within a sound attenuation chamber lined with sound foam, itself surrounded by a Faraday cage. Behavioral protocols in the recording chamber were controlled using a digital input/output card (PCI-DIO-96, National Instruments) run by the Matlab Data Acquisition Toolbox (Mathworks) and the freely available Psychophysics Toolbox (Brainard, 1997).

Initial training.

After handling and fluid restriction, rats were placed in the operant chamber with the nosepoke hole closed. They underwent 1 week of simple tone conditioning. Sessions lasted for 1 h each day. On the first day, an 8 kHz tone (6 s, 65 dB) was presented approximately every 30 s, and 120 μl of water was delivered at the spout (S+). Starting on the second day, water delivery occurred only if the rats licked the spout during the S+. On one-third of trials, no tone was presented and we measured the probability that rats would lick at the metal spout in the absence of the conditioned stimulus. Once rats licked consistently more on tone trials than silent probe trials over at least two sessions, they were advanced to nosepoke training.

Nosepoke training.

As shown in Figure 1, rats initiated presentations of stimuli by inserting their snout into the nosepoke hole, breaking the infrared beam inside it. Rats had to maintain this nosepoke position for a set time (foreperiod) to receive a stimulus. The foreperiod was increased from 25 to 500 ms within sessions across the first 3 d of nosepoke training. If rats withdrew from the nosepoke early, no stimuli were presented and responses were not rewarded. No time-out or other form of negative reinforcement was used to reduce the occurrence of premature responses and these trials were not included in behavioral or physiological analyses. Once a stimulus was presented, rats had to withdraw from the nosepoke within a set time [reaction time (RT); limit shortened to 1 s] and cross to the other side of the chamber (30 cm) to make contact at the spout within a set time (movement time; limit shortened to 5 s) to collect water (80 μl). We use the term “Go response” to describe nosepoke withdrawals within 1 s of stimulus onset that were followed by contact with the waterspout within 5 s. Stimuli remained on until a Go response at the waterspout, a new nosepoke, or the time window for a response had elapsed. Trials were self-paced, initiated by the rats' nosepokes, and sessions lasted ∼90 min.

Discrimination training.

After initial nosepoke training, rats were advanced to a simple Go/No-go discrimination task. On any trial, during the nosepoke, the rat could receive one of two types of stimuli. Only one stimulus was presented on each trial. If the rat received an S+ (8 kHz tone), it could make a Go response to collect a water reward. If the animal made a No-go response to S+, there was no punishment or indication that this was incorrect (although there is an implicit missed opportunity to collect a reward). If a rat received an S− (30 kHz tone), Go responses were incorrect and did not lead to rewards. Instead, a Go response forced a repetition of S− on the next trial. Once rats were trained, they typically reinitiated a new trial with a nosepoke after receiving S−. New stimuli were pseudorandomly selected with the constraint that no stimulus type could be selected more than three times in a row.

Stimulus blocks.

Rats were trained on blocks of familiar discrimination pairs for ∼4 months (∼75 sessions per rat). In any pair, one stimulus was rewarded and one stimulus was unrewarded. Throughout training, the reward status of two stimuli never changed (S+: 8 kHz tone was always rewarded, S−: 30 kHz tone was never rewarded). The reward statuses of the other stimuli (SW) were switched within sessions (SW+ when rewarded, SW− when unrewarded). In the sessions presented in this study, a noise burst was primarily used as the flexible, switching stimulus. Over training, rats also received practice with another switching stimulus (lights) as well as sessions that did not involve switching (i.e., stimuli–reward contingencies did not change within the session). Data with lights as the SW stimulus was qualitatively similar to that obtained with noise as the SW stimulus.

In each “switch session,” rats made discriminations between S+ (always rewarded 8 kHz tone) and SW− (unrewarded switching noise) for the first half of the session. Then, the stimulus–reward contingencies were changed. The switching stimulus was now SW+ (same noise, now rewarded) and S− was used as the unrewarded stimulus (never rewarded 30 kHz tone). Switches in reward contingencies were not signaled and occurred after 50 correct responses after achieving criterion on the initial discrimination. The criterion for accurate discrimination was 18 correct trials within a window of 20 stimuli (90% accuracy).

Electrode implantation.

Before surgery, a rat was initially anesthetized with 4% vaporized halothane. The rat was then injected intraperitoneally with ketamine (100 mg/kg) and xylazine (10 mg/kg). Smaller supplementary doses of ketamine were given as needed to suppress responses to toe pinches. The scalp was shaved and the rat was placed in a stereotaxic apparatus using blunt 45° ear bars to prevent eardrum rupture. Eyes were covered with ophthalmic antibiotic ointment to prevent desiccation. The scalp was disinfected with iodine, incised, and retracted to expose the skull surface. Lambda and bregma were leveled and bilateral craniotomies were made to implant multielectrode arrays.

Multielectrode arrays were composed of 50 μm stainless steel wires, coated with Teflon, and spaced in 2 × 8, 4 × 4, or 3 × 3 × 2 configurations with 250 μm spacing between wires (Neurolinc). The wires were connected to Omnetics connectors (0.025 or 0.050). All rats but one were implanted with 16 wire arrays in each hemisphere; the remaining rat received a 16-wire array in one hemisphere and an 8-wire array in the other hemisphere (3 × 3 × 2 configuration). Neuronal activity was monitored as electrodes were lowered into the medial striatum. Arrays were placed dorsally (centered at 0 mm AP, 2.2 mm ML, −4 mm DV) or ventrally (centered at 0.4 mm AP, 1.8 mm ML, −6 mm DV): four rats received bilateral medial implants, two rats received bilateral ventral implants, and eight rats received one medial and one ventral implant in opposite hemispheres.

After electrode implantation, the craniotomy was covered with cyanoacrylate (Slo-Zap) and cyanoacrylate accelerator (Zip-Kicker). The scalp was covered with methyl methacylrate (AM Systems). Wound margins were daubed with antibiotic ointment (Vetropolycin). Rats received one subcutaneous injection of buprenorphine (0.03 mg/kg) and oral enrofloxacin (34 mg/L) in their drinking water for 1 week. After recovery from surgery (one week), animals were fluid restricted to reinitiate behavioral sessions with neurophysiological recordings.

Recordings.

Rats were briefly anesthetized with 2% halothane to connect and tape headstage cables to the implanted connectors. Rats recovered spontaneous motor activity within a few minutes and were then placed in a separate acrylic chamber for at least half an hour before behavioral recording. During this time, spike sorting was performed and spontaneous recordings of neuronal activity were made, including recordings of wideband signals from each electrode (sampling at 20 kHz; analog filtering from 0.5 Hz to 5.9 kHz).

Signals from the implanted electrodes were fed into the recording system (Plexon Multichannel Acquisition Processor) and amplified 1000–20,000 times. Spike activity was isolated using a voltage threshold. Waveforms that crossed the threshold were sampled, timestamped, and stored at 40 kHz. Unique waveforms were identified online. Online root-mean-square values, while rats were resting, were typically 20 μV (calculated within the Plexon OnLine Sorter). Waveforms were then processed off-line (Plexon OffLine Sorter) to remove artifacts and sorted into different units using principal component analysis and template-based methods. After processing, units had to meet several criteria to be considered single units: (1) Mean peak-to-peak voltage had to be at least 100 μV. (2) Signal-to-noise ratio had to be at least 3:1. (3) Fewer than 2% of interspike intervals could be <2 ms. (4) The mode of the interspike interval histogram had to be >5 ms. (5) Baseline firing rates had to be <30 Hz. (6) The distribution of maximal waveform points had to be relatively normal (skewness <0.75). This latter measure ensured that waveforms from putative single-units were effectively isolated from the noise threshold.

We limited our analysis to 30 correct trials before and after the change in stimulus–reward contingencies. Inspection of behavioral records indicated that the animals performed the task in a consistent manner during this period. This also allowed us to compare activity around changes in stimulus–reward contingencies with activity in other, equal durations of the session. Furthermore, we required that a neuron's spike waveforms and interspike interval statistics were consistent throughout the behavioral session. We also evaluated neuronal activity during the drinking period, a time epoch that was unrelated to our analysis of action selection, to ensure that firing rates were generally stationary over trials. During the drinking period, neurons had to fire at least once on 10% of trials and have a z-score of <4 on a runs test (Siegel, 1956).

Histology.

At the conclusion of the recording sessions, rats were killed with an intraperitoneal dose of pentobarbital (>100 mg/kg). Rats were then perfused with chilled saline followed by 4% formaldehyde. Brains were extracted and placed into solutions of 25% sucrose for cryoprotection. Brains were cut on a frozen sliding microtome, stained with thionin, dehydrated in an increasing concentration of ethanol followed by xylenes, and mounted and coverslipped on gelatin subbed slides. Electrode holes were identified using light microscopy and plotted onto a rat brain atlas (Paxinos and Watson, 1998). Three-dimensional models of the rat striatum were constructed using freely available software written for Matlab by E. Y. Kimchi: http://spikelab.jbpierce.org/3DAnatomy.

Identifying neuronal modulations.

Data analysis was done using Matlab (Mathworks) and R (http://www.R-project.org). Timestamps from identified single units were aligned to the time of the stimulus to create perievent rasters and perievent histograms (bin size: 1 ms, time-window: −1 to + 1 s around the stimulus). We initially evaluated neuronal modulations at the time of the trigger stimulus by comparing firing rates in two windows surrounding the stimulus (100 ms before and after stimulus onset). Activity in these two windows was compared using the signrank test (p < 0.05). For initial analyses, we only analyzed data from correctly performed trials with reaction times >100 ms.

In addition, we evaluated neuronal modulations over the entire peri-stimulus epoch using a structural change test (Chow, 1960). The test was done as follows: (1) a cumulative sum histogram was calculated for data series (the average normalized firing rate), (2) a linear model was fit to full data window, (3) a series of linear models were fit to smaller data windows, 0.5 s in duration, that were moved over the data series in 1 ms increments, (4) the F statistic (Chow, 1960) was calculated as the sum of squared residuals between the coefficients for a given data window and the coefficients for the full data window (see supplemental materials for details, available at www.jneurosci.org), and (5) the F statistic was evaluated using standard values for the F distribution (Hansen, 1997) with a criterion of p < 0.05. This analysis was done using the strucchange library for R (Zeileis et al., 2002).

Decoding neuronal activity related to components of action selection.

Decoding methods were used to compare neuronal activity with various components of action selection. Specifically, we assessed whether neuronal activity contained information about changes in stimulus–reward contingencies, reward collection behavior, or variability in reaction times. Decoding methods were well suited for this study, compared with other methods such as ANOVA or ROC analysis, because decoding methods provide measures of the confidence of classification (i.e., posterior probabilities) on a trial-by-trial basis.

Sets of trials surrounding the switch in stimulus–reward contingencies were selected for decoding analysis. Neuronal activity on the last 30 SW− trials was compared with that on the first 30 SW+ trials after presentation of S− (unrewarded tone stimulus). The goal was to quantify when striatal neurons reflected this change in stimulus–reward contingencies. Such sensitivity determines what the appropriate behavioral response should be, regardless of what it actually will be. The stimulus–reward contingency for each trial was classified using firing rate, measured during a 0.6 s epoch starting at onset of the stimulus. Firing rates for two types of trials (e.g., SW− vs SW+) were compared using a probabilistic classifier (which is known as naive Bayes or the MAP classifier, where MAP stands for maximum a posteriori) (for review, see John and Langley, 1995; Domingos and Pazzani, 2004). We used the implementation of naive Bayes in the e1071 library for R. Additional analysis was done using alternative measures of neuronal activity, e.g., temporal changes in firing rate measured using wavelet methods (Laubach, 2004), and using other types of classifiers, including an unsupervised method for cluster analysis. The results reported here were consistent across methods, and so, for clarity, we report the simplest methods in the main body of the paper.

The naive Bayes classifier was based on nonparametric kernel density estimation. Training data were used to estimate the relative density of firing rates for each type of trial (e.g., firing rates on trials with SW− and SW+). Standard methods (the density function from the stats library for R) and Gaussian smoothing kernels were used for kernel density estimation. Testing data were evaluated by measuring the most likely trial type based on the neuron's firing rate and the density estimates for firing rate obtained from the training data. For each trial, we estimated the posterior probability that the trial occurred before or after the switch in stimulus–reward contingencies. Leave-one-out cross-validation was used. Based on simulations of random data, the minimum number of trials needed to obtain reliable estimates of successful classification (see the next paragraph) depended on having at least 30 trials of each type.

To control for nonspecific effects over the experimental sessions (Prokopenko et al., 2004), we compared neuronal activity from the first 30 SW+ trials to activity from the next 30 SW+ trials. This analysis was done for activity on trials with the same stimulus and reward contingencies. Since stimulus–reward contingencies were unchanged, any decoding would be due to nonspecific changes in neuronal firing, such as changes in motivation.

Due to issues of statistical power, we did not attempt to analyze differences between correct and error responses. The number of error trials is relatively low and to address this we would need to increase the number of trials that were analyzed before and after the switches in stimulus–reward contingencies. This would complicate the results by introducing longer-term variability into the analyses, which we were already explicitly measuring. As such, we have deferred error analysis for this data set.

Quantification of decoding results.

All decoding results were summarized using receiver operating characteristic (ROC) analysis (using the verification library for R) and information theory (Krippendorff, 1986). ROC analysis is based on plotting, for a given type of trial, the fraction of true positives (e.g., fraction of actual SW+ trials correctly predicted to be SW+) versus the fraction of false positives (e.g., fraction of actual SW− trials incorrectly predicted to be SW+). Significant levels of discrimination were then estimated by calculating the area under the ROC curve and the significance of the area under the curve was measured using a Wilcoxon rank-sum test. Essentially, this procedure measured the probability that the area under the curve was significantly greater than that expected for random data (Mason and Graham, 2002).

Information theory was used to calculate the mutual information between predictions of trial type (e.g., predicted SW− or SW+) made by the naive Bayes decoder and the actual trial type (e.g., actually SW− or SW+). Mutual information was assessed for the confusion matrix, a matrix with two columns for the actual class types and two rows for the predicted class types. The significance of the level of mutual information was assessed using a χ² test (Krippendorff, 1986). Detailed summaries of our use of these analyses are available (Laubach et al., 2000; Narayanan et al., 2005) and further details are provided in the supplemental material, available at www.jneurosci.org. For both the ROC and information theory measures, significant decoding was assessed using a criterion of p < 0.05.

Dynamics of changes in neuronal activity across trials.

Significant differences in Go responding and neuronal predictions of a rewarded Go response (switching stimulus now SW+) were estimated using the methods described above for measuring significant changes in firing rate (i.e., sctest) and for measuring the timing of such changes (i.e., breakpoints). Data were compared for blocks of 30 trials before and after a switch in stimulus–reward contingencies. Behavioral responding was measured for each trial as a 0 (No-go response) or 1 (Go response). Neuronal predictions of responding were measured either using the posterior probabilities from the naive Bayes decoder or using a thresholded version of the posterior probability, with 0 for predictions of No-go responding and 1 for predictions of Go responding. We obtained equivalent results for the estimation of the timing of neuronal changes using the raw posterior probabilities and the thresholded probabilities. As above, we obtained equivalent results using classic F-statistic (Chow, 1960), Bayesian (Barry and Hartigan, 1993), and other (Zeileis et al., 2003) methods for structural change analysis.

To ensure that the results were specific to changes in stimulus context, we also only analyzed data from 10 of that 14 rats that made Go responses to the switching stimulus after the first presentation of the unrewarded tone stimulus (S−) and that had neurons that fired during the period of Go responding. Two of the 14 rats switched spontaneously and were excluded, as their behavior was not comparable to the rest of animals. These animals may have discovered the switch in stimulus–reward contingencies by randomly checking for rewards at the time that the stimulus block changed and so may not have experienced a change in stimulus context. Two other animals did not have neurons that were sensitive to action selection, and behavioral data from these animals was excluded from further analysis.

Time course of changes in neuronal activity within a trial.

A “moving window” analysis was used to quantify the time course of changes in neuronal activity after a change in stimulus–reward contingencies. This analysis explored when, within a trial, neuronal sensitivity developed. Firing rates were measured using a 0.6 s time-window. The window was stepped in 0.05 s increments over the period from 0.6 s before to 0.4 s after the onset of the switching stimulus (SW−/SW+ noise). The first window in the series represented the firing rate from 0.1 s before entry into the nosepoke until the time of stimulus onset (0.6 s later, referenced as 0 s). The final window represented the firing rate from 0.4 to 1.0 s after stimulus onset, when rats moved to collect rewards or initiated new trials.

Firing rates in the series of time-windows were then evaluated using decoding methods. Naive Bayes classifiers were trained and tested with data from each time-window. Identical methods to those described above were used. Two measures of the success of classification (area under the ROC curve, mutual information measured from the confusion matrix) were noted for each time-window. The average values of mutual information were then plotted for the series of time-windows. Plots were made for all neurons in the medial and ventral striatum, and for all neurons that significantly decoded the change in stimulus value during the blocks of ± 30 trials around the switch in stimulus–reward contingencies. Changes in the data series were analyzed using change-point analysis, as described above. Two points in the data series were noted: the first time-window that had a significant level for the F (or Chow) statistic and the time-window that had the maximum level for the F statistic (the change-point).