Plasticity of Temporal Pattern Codes for Vocalization Stimuli in Primary Auditory Cortex

Electrophysiology in naive animals

We characterized responses from 142 single units from A1 of five adult ferrets. As a first step, we analyzed these units in the manner identical to that used by Wang and colleagues (Wang et al., 1995; Wang and Kadia, 2001). The mean spike rates of each unit in response to each natural (R_Nat) and time-reversed (R_Rev) twitter call were calculated over a 1200 ms period from stimulus onset, and the corresponding “selectivity index” (d) was calculated using the formula d = (R_Nat − R_Rev)/(R_Nat + R_Rev).

Figure 2 shows scatter plots of R_Nat against R_Rev for all of the cells in our study, with responses to the first, second, and third exemplar of the twitter stimuli. The spectograms of these stimuli are shown at the top left of Figure 1. If we assume that ferret A1 units are not selective for natural marmoset calls, then we expect that the data in these scatter plots should clusters more or less tightly around the main diagonal (R_Nat = R_Rev), and the distribution of the corresponding values of d should have a mean very close to 0. This expectation is very much borne out. The means ± SD of the values for d obtained for each of the three stimuli were 0.030 ± 0.101, −0.003 ± 0.093, and −0.020 ± 0.099, respectively. The overall median value for d (0.002) was not significantly different from 0 (sign test, p = 0.74). The selectivity index values we obtained in the ferret are comparable with those reported by Wang and Kadia (2001) for cat A1 (0.047 ± 0.265, 0.086 ± 0.238, and 0.068 ± 0.372, respectively) and are smaller by an order of magnitude or more than those reported for the marmoset (0.479 ± 0.361, 0.335 ± 0.302, and 0.385 ± 0.340). Preferences for natural over time-reversed marmoset calls in ferret A1 are therefore tiny or 0, and a coarse rate-code-based scheme would have difficulty distinguishing natural from time-reversed calls on the basis of ferret A1 responses.

Encoding of twitter calls in temporal spike patterns

When analyzed in terms of mean spike rate over the entire stimulus duration, units in naive ferret A1, like those in cat, typically lack “selectivity” for natural marmoset twitter calls. Like previous authors (Wang et al., 1995; Gehr et al., 2000; Wang and Kadia, 2001), we studied call selectivity in an anesthetized preparation, and, although we have at present no reason to assume that rate-based call selectivity in A1 is radically altered by anesthesia, future studies in awake preparations will be needed to confirm this experimentally. To human observers, the natural and time-reversed twitter calls sound quite different. We do not know whether ferrets also perceive these sounds differently, but, if they do, then any information that might underpin any perceptual differences between the stimuli would most likely be either carried in a temporal pattern code or processed in a neural pathway that bypasses A1. When we plotted the neural responses in a standard raster plot format, we noticed that ferret A1 units varied considerably in their ability to respond to the marmoset twitter calls with distinctive, reproducible discharge patterns. Considerable unit-to-unit variability in the properties of A1 responses to vocalization stimuli has been noted by other authors (Wallace et al., 2005) and is as such not unexpected. Figure 3 gives illustrative examples from two different units. The spike patterns shown in Figure 3A appear to be highly “informative,” in that the unit responds to the twitter calls with a fairly reproducible series of short bursts. The latency of these bursts and the number of spikes in each burst varies somewhat from trial to trial and these burst patterns are subjected to some degree of “noise” from spontaneous activity, but one nevertheless notices that the burst patterns evoked by the natural call exhibit systematic differences from those evoked by the time-reversed call. (Note also that the response pattern elicited by the time-reversed call does not appear to be a time-reversed copy of the response to the natural stimulus.) The spike patterns exhibited by the unit illustrated in Figure 3B, in comparison, appear much less informative with respect to stimulus identity than those seen in Figure 3A, in that the spike patterns are more variable from trial to trial and the patterns evoked by the normal and the reverse stimulus are less distinctive.

To be able to quantify these apparent unit-to-unit differences in the amount of information that appears to be carried in the temporal response patterns, we devised a simple pattern recognition algorithm that attempts to “guess” which stimulus evoked a particular response pattern. The first step in this algorithm is to represent each individual response as a poststimulus time histogram (PSTH). The choice of bin size used to construct these PSTHs is of some importance, which is why a range of different bin sizes were considered, as will be discussed further below. Figure 4, A and B, shows the data for the units whose response rasters were shown in Figure 3, when the responses to each of the 20 presentations of each stimulus are binned in 20-ms-wide bins. Once individual responses are represented in this way, each 1200-ms-long spike train becomes a list of 60 spike count values and can be thought of as a point, or vector, in a 60-dimensional space. When represented in this manner, the set of responses to a particular stimulus effectively form a “cloud” in an abstract, high-dimensional response space, and one can quantify how similar two responses are by calculating the Euclidian distance between the two responses in this space. If we now draw one of the response patterns from the dataset (a “test pattern”) and we calculate the Euclidean distance of this test pattern to each of the “cloud centers” (mean response patterns) formed by the remaining responses (the “training sets”), then we can try to guess which stimulus might have evoked this test response pattern simply by assigning the test pattern to the stimulus associated with the closest training set. If the response patterns are reproducibly similar for repeated presentations of the same stimulus and reproducibly different from patterns evoked by other stimuli, then the response patterns will form distinct clusters in the response space and most test patterns will be correctly assigned, but, if the responses lack reproducible and distinctive patterns, then the assignments will be essentially random. By picking each response in turn as the test pattern and noting the proportion of correct assignments, we can therefore quantify how informative individual response patterns are with respect to stimulus class.

In practice, we found that performing a principle component analysis (PCA) on the response pattern of each unit before running this assignment algorithm improves the performance of this decoding algorithm somewhat in a number of cases. PCA transforms the coordinates of the response pattern vectors in a manner that exploits the correlation structure of the dataset and allows us to reduce the dimensionality of the response space by disregarding dimensions that capture only minimal amounts of the variance of the data. The small amount of variance in the dataset attributed to these dimensions is most likely attributable to noise in the response pattern, e.g., from spontaneous activity, whereas dimensions that account for much of the variance in the data can be thought of as “features” of the data. By running our assignment algorithm in a “principal component space” in which we retained a sufficient number of principal components to capture 90% of the variance of the data (between 16 and 40 components, depending on the unit), we were able to increase the performance of the classifier in some cases, presumably because it allowed us to avoid problems of “overfitting,” but results obtained by running the classification on the raw spike count vectors are essentially similar.

Figure 4, C and D, shows “confusion matrices” that illustrate how well the classifier algorithm was able to “decode” individual spike trains. The grayscale indicates the proportion of test spike patterns from the set of responses to the stimulus given on the x-axis were “assigned” (i.e., judged to be “closest,” or “most similar to” the set of responses evoked by) the stimulus indicated on the y-axis. If the algorithm was to identify all spike trains correctly, then the confusion matrix would feature a black main diagonal on a white background. However, if the algorithm fails to find useful information in the spike patterns, then the assignments are essentially random. Qualitatively, it certainly appears that the responses of the unit shown in Figure 4A are often correctly identified by the classifier, whereas those of the unit shown in Figure 4B are misclassified so frequently that it is not immediately clear whether the decoding process performed significantly above chance.

Confusion matrices like those shown in Figure 4, C and D, allow us to estimate the “information content” of the response patterns, or, more accurately, the mutual information (MI) between response and stimulus class. The MI (in bits) is given by Shannon's formula: where x and y are the values taken by the random variables “presented stimulus class” and “assigned stimulus class” (x, yϵ{1, 1R, 2, 2R, 3, 3R}), and one adopts the convention that 0 · log(0) evaluates to 0. The a priori probability p(x) of any one stimulus having evoked any one particular response is because we used six different stimuli in the experiment and each stimulus was presented with the same frequency. The probability of a response being assigned to any one stimulus class p(y) and the joint probability of observing a particular combination of stimulus and response assignment p(x, y) are not known a priori but can be estimated from the observed frequency distributions in the confusion matrix. However, it is important to remember that MI estimates calculated in this manner can be subject to non-negligible positive sampling biases, because using the observed frequency distributions as a necessarily rough estimator for the true underlying probabilities can easily lead to somewhat inflated MI estimates (Rolls and Treves, 1998; Trappenberg, 2002; Nelken et al., 2005). Here we estimated the expected size of this bias by calculating MI values for “shuffled” data, in which the response patterns had been randomly reassigned to stimulus classes. The shuffling was repeated 10 times, and the mean MI estimate for the 10 shuffled datasets was used as estimator for the bias. All MI values reported below were “bias corrected,” i.e., the bias estimate obtained for each unit was subtracted from the original MI estimate. Bias estimates varied little from unit to unit, the median bias was 0.18 bits/response, and <5% of bias estimates exceeded 0.28 bits/response, so that bias corrected MI values >0.1 bits/response might, as a rough approximation, be deemed “statistically significant.”

When analyzed in this manner, we estimate that the responses of the unit shown in Figure 4A transmit on average 1.28 bits of information about which of the twitter stimuli were transmitted, whereas the MI estimate for the unit shown in Figure 4B is only 0.21 bits (a much more modest value but one that exceeds our rough significance criterion of 0.1 bits, and, indeed, if we treat the confusion matrix shown in Fig. 5D as a χ² contingency table, then we can reject the null hypothesis that the actual and assigned stimuli are statistically independent at p < 0.001, confirming that the firing patterns of this unit does transmit very modest but statistically significant amounts of information).

As noted above, it is to be expected that the performance of our pattern classifier algorithm, and hence the MI estimates obtained, will depend on the bin width chosen at the first step, when individual response patterns are expressed as PSTHs. One could attempt to devise a more sophisticated algorithm that might be able to estimate the information content of a response pattern without temporal binning (Victor, 2002). However, we felt that it would be of considerable interest to explore the dependency of the MI estimate on the initial bin size, or “temporal resolution,” of the decoding step by systematically varying the bin size. The result of reanalyzing the data from the two units from Figure 4 at various bin sizes from 2.5 to 640 ms is shown in Figure 5A. One observes that, for the responses of the first unit, the pattern classifier performs well, extracting in excess of 1 bit of information per response, as long as it operates at a temporal resolution of 40 ms or finer. For coarser temporal resolutions, the amount of information that can be extracted from the temporal response patterns drops dramatically. For the responses from the unit shown in Figure 4B, the amount of information that can be extracted remains low and barely exceeds the critical value for statistical significance of 0.1 bits/response regardless of the temporal resolution used in the decoding step.

Figures 3⇑–5A may give the impression that there may be two very different types of cells in auditory cortex, one transmitting large amounts of information about the identity of complex, natural stimuli through fairly precise temporal discharge patterns and the other conveying very little information about stimulus identity regardless of the temporal resolution at which its responses are analyzed. An obvious question is whether these units are representative examples of two more or less discrete classes of cells or rather two samples along a continuous spectrum of neural response properties. We address this question in Figure 5B, which shows a “waterfall” plot of estimated MI (on the x-axis) as a function of temporal resolution (on the z-axis) for all 142 A1 units recorded in the naive animals. Units are ordered by their maximal MI value along the y-axis of that plot. From this plot, it becomes apparent that units appear to form a continuum, with many cells transmitting practically no information about stimulus identity (although they responded vigorously to the stimuli), whereas others transmitted various amounts of information, up to ∼1.7 bits/response.

Figure 5B suggests that the units in our sample form a continuum rather than clearly distinct classes, but, to facilitate additional discussion, we shall nevertheless draw a somewhat arbitrary distinction between “highly informative” and “poorly informative” units simply on the basis whether they transmit >0.5 bits/response. For the 47 highly informative units in our dataset, it is clear that the amount of information extracted from the responses tends to decline when the temporal resolution of the decoding becomes much coarser than 40 ms. Figure 5C serves to visualize the dependency of MI on temporal resolution for these units more clearly. It shows the normalized MI estimates (relative to the units own maximal MI value) plotted against temporal resolution. Data from all 47 highly informative units are shown superimposed. The mean normalized MI values across these 47 units is also shown as a light gray line. Although there is some unit-to-unit variability, one nevertheless observes a clear trend for MI values to reach a maximum at resolutions between 10 and 20 ms. For most units, MI values start to decline sharply at resolutions >40 ms. MI estimates sometimes also decline somewhat if the temporal resolution used by the decoder becomes too fine, i.e., <5 ms.

An additional question one might ask is whether the entire response pattern is necessary to distinguish the various stimuli in this set. To investigate this question, we repeated the analysis of the response patterns at a 10 ms time resolution but fed only the first 100, 200, 300, … ms of the response patterns into the classifier algorithm. The results of this analysis with restricted time windows is shown in Figure 5D. It shows that the amount of information extracted by the classifier from the spike patterns increases monotonically with the length of the spike trains available for analysis, as one might expect, but, for most units, the extracted MI appears to asymptote so that there are much more rapid gains in MI with an increased analysis period during the first 600 ms compared with the second 600 ms of the response.

Comparing Figures 3, 4, and 5A, one might get the impression that perhaps those units that exhibit responses that are clearly strongly time locked to individual syllables or other transients in the stimulus envelope should be highly informative, whereas those that do not carry little information about stimulus identity. However, that is probably an oversimplification. In Figure 6, we show dot raster displays for the responses of every third unit in this sample to the first twitter call played forward and reversed. The dot rasters are arranged in order of increasing MI (i.e., the same rank order used in Fig. 5B) from the bottom up, and the waveforms of the stimuli are reproduced at the top of the figure to facilitate a visual comparison between stimulus envelope and timing of response bursts. Although there is a general trend for units with a clear tendency to time lock to individual syllables to occur closer to the top of the figure, that alone would not explain the ranking. Compare, for example, the responses plotted at the very top with the 12th unit from the top in this plot. These responses are marked by little arrows to the left of the raster plots. The rank order of these units in Figure 5B is 142 and 109, respectively, but, whereas the second appears to give more vigorously time-locked responses, the first one carries almost twice as much information in each response, presumably because its discharge pattern, although not easily attributable to particular peaks in the envelope of the stimulus, are nevertheless quite reproducible for representations of the same stimulus but distinct for different stimuli.

In summary, the responses recorded in naive ferrets are similar to those reported by Wang and Kadia (2001) for the cat and very different from those reported for the marmoset, in that overall mean spike counts do not discriminate natural from time-reversed twitter calls. Nevertheless, our analysis clearly demonstrates that many ferret A1 neurons carry significant amounts of information about the marmoset twitter stimuli in their temporal pattern, information that could be used to distinguish natural from time-reversed calls. Furthermore, although the decline in MI at resolutions <5 ms seen for some units in Figure 5C is clearly a limitation of our decoder (a more sophisticated decoder might be able to combine bins appropriately to recover the information available at coarser resolution), we believe that the decline at resolutions coarser than 40 ms reflects physiological properties of the representation of this particular stimulus set at the level of A1. Of course, it would be of considerable interest to know whether this representation changes and whether it perhaps becomes more “marmoset-like,” if ferrets are familiarized and trained to recognize marmoset twitter stimuli. To explore this question, two ferrets were trained to associate marmoset twitters with water rewards in a Go/No-go paradigm.

Behavioral training

Using the paradigm described in Materials and Methods, two ferrets were trained to respond to marmoset twitter stimuli by pushing a reward spout to receive a water reward (a Go response). Sound stimuli other than marmoset twitters were to be ignored, and the next trial was to be initiated by pushing a start spout (a No-go response). Figure 7A shows a typical example of the performance of the animals during a behavioral testing session after ∼2 months of training. The animal has clearly learned an association between marmoset twitter stimuli and water reward, making Go responses to almost every presentation of the twitter stimuli but only very rarely making Go responses to sounds other than the marmoset twitters. Figure 7B shows the “reaction times,” i.e., times from the onset of the sound stimuli to the triggering of the reward spout (Go response) or the start spout (No-go response), respectively, for the same training session. Bearing in mind that the sound stimuli are typically just >1 s in duration (compare with Fig. 1) and that the behavioral training program was set up not to register any responses made <0.9 s after stimulus onset to encourage the animals to listen carefully before making their choice, it is apparent that the trained animals typically make their choices very rapidly after the end of the stimuli. Figure 7C shows the “learning curves” for the two trained animals. The percentage of correct responses in each training session is plotted against time (in days) elapsed since the first training session. In an initial training period, lasting 9 d for the first animal and 16 d for the second, during which the animal familiarized itself with the setup and the mechanics of the task, only Go stimuli were used, and, because this early period of procedural training involved no acoustic discrimination, no performance data are shown for this early stage of training. Although there are slight differences in the shapes of the learning curves for these two animals, both animals typically perform at 80% correct or better after 25 d of training and routinely exceed 90% correct after ∼2 months of training. As mentioned in Materials and Methods, intense training periods during which an animal would frequently run two sessions per day were interspersed with “training holidays,” which could last anywhere from 2 d to >3 months. The timing of these training holidays was not governed by scientific considerations but was dictated partly by animal welfare considerations (prolonged uninterrupted training periods with their accompanying restrictions of access to drinking water could otherwise lead to weight loss and adverse health effects) and partly by other constraints on the experimenters' time. Figure 7C illustrates that the performance in the first session immediately after a long training holiday can be noticeably reduced, particularly during early stages of the training, but the animals typically return to >90% performance after just a single “refresher” session.

Electrophysiological responses in the trained animals

After the training periods charted in Figure 7C, the animals were prepared for physiological recording in a manner identical to that used for the naive animals. In total, we obtained responses to the marmoset twitter calls from 501 units in the two trained animals.

The data from these animals were analyzed in a manner identical to that used for the naive animals. Figure 8 plots the response rate evoked by natural twitter calls against rates evoked by their time-reversed counterparts for the trained animals. The layout of the figure is identical to that of Figure 2. As in the untrained ferrets, the data cluster tightly around the main diagonal (R_Nat ≈ R_Rev), and, for all three twitter stimulus exemplars, the mean values of the selectivity index d were close to 0 (means ± SD of −0.030 ± 0.106, −0.0036 ± 0.0978, and −0.028 ± 0.1069, respectively). The overall median selectivity index was, in fact, slightly but statistically significantly negative (−0.021; p < 10⁻⁵, sign test), indicating that the responses to the natural twitter calls, which had been part of the training set, were if anything ever so slightly weaker on average than those to the time-reversed sounds that the animals had never heard before the electrophysiological recording. However, this effect is so small that, statistical significance notwithstanding, one would be hard pushed to accord it much “physiological significance.” In any case, the fact that the trained animals have learned to recognize and associate a reward with the natural marmoset twitter vocalizations has clearly not induced the sort of preference or selectivity for these natural marmoset twitters that has been reported for neurons in marmoset A1 (Wang et al., 1995).

In Figure 9A, we show the estimates of the mutual information between response pattern and stimulus for the trained data obtained with the decoding algorithm described above and plotted in a waterfall format identical to that used in Figure 5A. Like in the untrained data, we notice that the amount of information extracted by the pattern classification algorithm varies from unit to unit and depends on the temporal resolution, with the highest MI values typically seen at resolutions finer than 40 ms. When comparing Figures 9A and 5A, one also notices that the maximum MI values are in general higher in the trained than in the untrained animals. In Figure 5A, only approximately one-third of units reached MI values exceeding 0.5 bits/response. In the trained animals, this proportion has grown to approximately two-thirds, and the most informative cells reach MI values in excess of 2 bits/response, close to the theoretical maximum given by the entropy of the stimulus set of log₂(6) = 2.58 bits/response. Figure 9B shows the differences between the MI distributions in the trained and naive animals. It effectively plots the difference between the waterfall plots shown in Figures 9A and 5B, respectively, but, given that the number of units in the trained and untrained samples was not the same, the “MI surfaces” shown in the waterfall plots first had to be resampled along the unit rank dimension (x-axis) using the resample function of the Matlab signal processing toolbox to give interpolated MI surfaces with 100 “unit centile rank” steps for each dataset. The difference between the resampled surfaces is shown in Figure 9B as a grayscale map. It shows that, although the amount of information recoverable from the spike trains at coarse temporal resolutions >80 ms hardly changed as a consequence of training, for shorter temporal resolutions, one sees fairly substantial increases in information throughout a large proportion of the sample. A Wilcoxon's rank-sum test comparing the maximum MI values across units for the trained and untrained animals was used to confirm that these marked increases in transmitted information are statistically highly significant (p < 10⁻⁸).