Incorporating Naturalistic Correlation Structure Improves Spectrogram Reconstruction from Neuronal Activity in the Songbird Auditory Midbrain

Least informative prior: uncorrelated Gaussian distribution. A, An example spectrogram with power spectral density indicated by color. B, Normalized histogram of power spectral density values across all songs and spectrogram bins (blue dots). The mean and variance of these power values is used to construct a Gaussian prior (black line) that confines estimated values of power spectral density to regions found in actual song spectrograms. C, To visualize the information provided by the prior, a sample spectrogram drawn from this prior is plotted. This prior does not provide information on spectrotemporal correlations in spectrograms, as demonstrated by this sample.

Spectrotemporaly correlated Gaussian prior. A, The spectrotemporal covariance matrix is modeled as separable in frequency and time. The frequency component is the spectral covariance matrix (top). The temporal component is fully described by the temporal autocorrelation function in song spectrogram power density (bottom, red line). The prior uses an approximation to this function using an autoregressive model (blue line). B, The full spectrotemporal covariance matrix is a concatenation of several spectral covariance matrices, like those shown in the top, each corresponding to the covariance at a different temporal lag. The bottom, Approximate Covariance, plots the separable covariance matrix, and the middle, True Covariance, plots the nonseparable covariance matrix. C, Top, An example spectrogram used in determining song statistics for constructing the Gaussian prior. Bottom, Sample spectrogram drawn from the Correlated Gaussian prior. D, Two-dimensional power spectra, also called the modulation power spectra (MPS), for song spectrograms (top) and for the prior (bottom); the prior does a good job of capturing information about spectrotemporal modulations except at joint regions of high spectral modulations and temporal modulations near zero.

Most informative prior: hierarchical model with a two-state hidden variable that infers whether the spectrogram is in a vocalization or silent period. These periods have different statistical properties not captured by a single Gaussian prior. The state variable determines which spectral covariance matrix and mean the prior uses to inform reconstructions. A, Example spectrogram overlaid with vocalization and silent states (black line). B, Top left, Spectral covariance matrix used during vocal periods. Top right, Spectral covariance matrix used for silent periods. Bottom, Prior information of transition rates between silent and vocal periods determined from song spectrograms. C, Sample spectrogram drawn from this prior; the sharp transitions in song statistics during vocal and silent periods better match song spectrograms.

Conspecific song discrimination based on the likelihood of spike-trains from multiple neurons. A, Spike trains from multiple neurons in response to presentation of song segment 1. Under a two-alternative forced choice (2AFC) test, song discrimination is performed by choosing the song which leads to a greater likelihood of observing the given spikes. Spikes from a given neuron are plotted at the BF at which that neuron's receptive field reaches maximal value. Neurons with the same BF are plotted on the same row. B, 2AFC results as a function of response duration and the number of neurons used for discrimination. The 2AFC test was performed multiple times for each possible pairing of the 20 songs in the data set. Each panel shows the frequency of correct trials across all possible song pairings. Above each panel, the average of the histogram is reported. On average, neurons performed at chance level when stimulus segments were only 3 ms in duration. Near-perfect song discrimination can be achieved using 189 responses and response durations of at least ∼30 ms, or 104 neurons and durations of ∼100 ms.

Single-cell decoding of song spectrogram. A, Top, Spectrogram of birdsong that elicited the two spikes shown immediately below. Spikes are plotted at the frequency at which this neuron's receptive field reaches maximal value. B, Top left, The most probable spectrogram from the posterior distribution (MAP estimate) given the two spikes shown in A and using an uncorrelated prior. When a single spike occurs, the MAP is determined by the neuron's STRF (right). In the absence of spikes, the MAP is determined by the prior mean. Bottom left, MAP estimate using the correlated Gaussian prior; when a spike occurs the MAP is determined by the neuron's STRF multiplied by the prior covariance matrix (right). Immediately after a spike, the MAP infers spectrogram values using prior knowledge of stimulus correlations.

Population decoding of song spectrogram with varying degrees of prior information of song statistics. A, Top, Spectrogram of birdsong played to 189 different neurons leading to the spike responses shown immediately below. Spikes from a given neuron are plotted at the BF at which that neuron's receptive field reaches its maximal value. Neurons with the same BF are plotted on the same row. A–E, MAP estimate given the responses in A using an uncorrelated prior (B), a prior with temporal correlations and no spectral correlations (C), a prior with spectral correlations and no temporal correlations (D), and a prior with spectral and temporal correlations (E). Combining the spike train with spectral information is more important for reconstructing the original spectrogram than combining the spike train with temporal information. However, combining spikes with joint spectrotemporal information leads to the best reconstructions.

Decoding performance given different amounts of prior information and numbers of neurons. A, Spectrogram reconstructions (top) for an example song (Fig. 7A) using a Gaussian prior with spectrotemporal correlations and using varying numbers of neuronal responses (bottom). Above each reconstruction is the SNR used to measure similarity between the reconstructed song and the song presented to the bird. B, Solid lines show the SNR averaged across all decoded songs, whereas dashed lines show 1 SE. The prior used for decoding is denoted by color. Spectral prior information leads to faster growth in the SNR than temporal information. For reference, the magenta line shows the growth in SNR for the commonly used the OLE. The OLE has access to both spectral and temporal correlations. C, Coherence between spectrograms and reconstructions under the four different priors. The horizontal axis reports temporal modulations and the vertical axis reports spectral modulations. All plots display the highest coherence at low spectral and temporal modulations. The primary effect of adding spectrotemporal prior information is to improve reconstructions at lower spectral and temporal modulations.

Single-neuron and population decoding using a hierarchical prior. A, Song spectrogram along with a single cell's response to this song (left) and the response of this cell plus 49 other cells with nearby characteristic frequencies (right). B, MAP estimates using a single, correlated Gaussian prior (top) are compared with estimates using the posterior mean and the hierarchical prior (bottom); in both the single-neuron and population decoding cases, the estimate using a hierarchical prior looks similar to the MAP with a Gaussian prior. C, The expected value for vocalization state given responses; single-cell responses do not yield enough information to accurately infer the spectrogram's vocalization state; however, as the number of neurons used for inference increases, the vocalization state becomes more pronounced.