The Magnitude, But Not the Sign, of MT Single-Trial Spike-Time Correlations Predicts Motion Detection Performance

Motion detection task and behavior

Two monkeys were trained to perform the motion detection task illustrated in Figure 1C. At the start of each trial, the animals viewed two static RDPs that began moving with 0% coherent motion at time 0 (trial start). The two RDPs were generated with unique random-number seeds and were thus uncorrelated. We will hereafter refer to the first 500 ms of the 0% coherent motion as the “preamble” (Fig. 1C), during which no motion pulses occurred. The preamble was followed by a variable motion period from 500 to 10,000 ms in which a 50 ms coherent motion pulse could appear in one or both RFs with a flat hazard rate.

The task of the animal was to release a lever within a 200–800 ms RT window after the motion pulse. A lever release within the RT window is referred to as a “hit” outcome, while failure to detect the motion pulse is referred to as a “miss” outcome. A lever release before the motion pulse was scored as a “false alarm.” Because we were interested in the 0% coherent motion period leading up to the motion pulses, we combined trials from the one- or two-pulse conditions in our analyses. Before a motion pulse occurred, all trials were the same (0% coherent motion). Whether one or two motion pulses were presented was randomly interleaved, and the animals did not know in advance where a motion pulse would occur. While the monkeys performed this task, we simultaneously recorded from two recording sites (MU 1 and MU 2) in area MT.

During the preamble, the monkeys knew there was zero probability of a motion pulse. At 500 ms, the probability of a motion pulse was no longer zero and followed a flat hazard rate. This ensured that the animals could not predict when the motion pulse would occur after the initial 500 ms preamble. As such, we observed that after an initial ramp up, the proportion of false alarm and hit behavioral outcomes was relatively constant over time (Fig. 1D). Note that the false alarm rate (Fig. 1D, orange) is shifted leftward by the median RT. These behavioral results suggest that the animals anticipated the end of the preamble and the beginning of the nonzero hazard portion of the trial.

For each experiment, we recorded MT neural activity from nonoverlapping RFs located in the same visual hemifield using two microelectrodes separated by 1–2 mm (Fig. 1B; see Materials and Methods). The location and size of each RDP was matched to those of the RFs, while the direction and speed of the motion pulse were matched to that preferred by the neurons under observation. This increased the probability that our electrodes recorded neural activity that would be correlated with the motion detection task (Bosking and Maunsell, 2011). From previous analysis, the link between our MT single-unit firing rate and behavior on a trial-to-trial basis was relatively strong (Smith et al., 2011).

In 47 sessions, we extracted MU neural population activity from each electrode. The reason we used MU activity was to reduce the detrimental effect that a low number of spikes has on estimating neural correlations, and to ensure that all sessions had approximately the same firing rates. Thus, we set our spike-sorting parameters to produce ∼200 spikes/s just before the motion pulse occurred (see Materials and Methods) for each of our 94 individual MU recordings. Note that single-unit results are also included below and were similar to our MU results (see Fig. 6).

We first wanted to establish that our MU spikes responded to our stimulus as expected from single-unit MT recordings. Figure 1E (left) shows that the average MU firing rates transiently increased in response to the start of the 0% coherent motion, and then experienced a small decay as the trial progressed, consistent with what is typically observed in MT visual cortical neurons (Müller et al, 2001). MU activity showed a robust response when aligned to the motion pulse (Fig. 1E, right), as expected from our single-unit observations (Smith et al., 2011). Also in agreement with our single-unit observations, the MU responses to the motion pulse were greater for hit trials (solid blue) compared with the miss trials (solid red). Importantly, there was no appreciable MU response when the motion pulse occurred in the other RF (dashed lines), which supports that our RFs were largely nonoverlapping. The difference in firing rate for hit and miss trials after the motion pulse occurred was previously accounted for by a model with two independent sensory channels (Smith et al., 2011). This model, however, did not include nonsensory effects, such as shifts in attention after the preamble, or the effects of neural correlations. Thus, we next computed three measures of single-trial spike-time correlation between our two MU responses, and linked these measures to behavioral outcome.

The trial-by-trial link between spike-time correlations and behavioral outcome

We hypothesized that single-trial spike-time correlations between our two MU responses would predict behavioral outcome. To examine this hypothesis, we focused on the following two trial periods where the stimulus was 0% coherent: the preamble and just before the motion pulse. We did not include the period after the motion pulse to avoid potential confounds when estimating neuronal correlations during the strong transient increase in spike rate. Thus, our task provided the following two comparison periods: the preamble aligned to the start of the trial, when the animals knew the motion pulse would not occur (probability of a motion pulse = 0); versus the period just before the motion pulse began (probability of a motion pulse > 0).

Figure 2A illustrates how we computed the spike-time correlations between our two MU responses for an example trial. First, MU spikes were smoothed by convolving with a Gaussian kernel (1.5 ms SD), centered on each spike. The width of the Gaussian kernel provides a measure of the timescales (or frequencies) that contribute to the spike-time correlation. As the kernel width increases, the contribution of high frequencies is removed. For example, a 1.5 ms Gaussian kernel has a low-pass −3 dB cutoff of ∼88 Hz. We will address the effect of the kernel width below.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Measures of spike-time correlation and DP for a single session. A, Single-trial example for two analysis periods: after the trial began (preamble, left) and before the motion pulse (right). MU spikes were first smoothed by convolving with a 1.5 ms Gaussian kernel. Within a sliding 300 ms analysis window (gray), the smoothed MU responses were used to compute three measures of spike-time correlation: R values (black), |R| values (red), and MI values (blue). The average firing rate for both MU responses was also computed using the same analysis window (green). B, Distributions of single-trial spike-time correlations and firing rates for an example session when the analysis window aligned to the start of the motion pulse. Distributions are shown separately for hit (top) and miss (bottom) behavioral outcomes. The corresponding ROC-metric DP is shown above each pair of distributions. DP values that depart from 0.5 suggest a link with behavioral outcome. C, Time course of the correlational and firing rate DPs for the same example session. The time corresponds to the leading edge of the 300 ms analysis window for either the preamble period (left) or before the motion pulse (right).

Next, spike-time correlations were computed from the smoothed neuronal response within a sliding 300 ms window (Fig. 2A, gray box). A 300 ms window was chosen to provide enough data to reliably estimate correlations while still enabling us to observe their time course. Estimated correlations were aligned to the leading edge of the window (arrows). We stopped the analysis window at 50 ms after the motion pulse because this is when MT typically began responding to the coherent motion.

Using the smoothed MU response, we computed the following three metrics of single-trial spike-time correlation: R, |R|, and MI. R is simply the zero-lag correlation between the two smoothed MU responses and is a standard measure of similarity (Cohen and Kohn, 2011; Smith et al., 2011), but has limitations that our other two measures were better equipped to handle. For example, we used the absolute value of the Pearson's correlation because it has been suggested that negative correlations may be as relevant as positive ones (Chelaru and Dragoi, 2016). Mutual information was also used as a measure of single-trial spike-time correlation because it captures nonlinear dependencies and is insensitive to the sign of the correlations. While the single-trial Pearson's correlation is simple to compute, for mutual information we used a common method adapted from the study by Moon et al. (1995) to compute our MI values from density estimates. Note that although |R| and MI values are positively biased measures of correlation, we were interested only in how they differed between hit and miss behavioral outcomes.

Next we linked the spike-time correlation metrics (R, |R|, and MI) to behavioral outcome using the ROC-based metric DP (Cook and Maunsell, 2002). DP is similar to choice probability (Britten et al., 1996; Shadlen et al., 1996; Parker and Newsome, 1998; Price and Born, 2010), and is commonly used for expressing the covariation between neural responses and two behavioral outcomes (hit vs miss) on a trial-by-trial basis. A DP of 0.5 indicates that our measure of correlation did not vary with the behavioral performance of the animal. A DP of >0.5 suggests that spike-time correlations were larger on hit trials versus miss trials, whereas a DP of <0.5 indicates the opposite. We estimated the DP at 5 ms time points using the hit and miss distributions of each of our three measures of spike-time correlations and firing rate.

The time course of our three measures of spike-time correlation for the example trial are shown in Figure 2A (R, black; |R|, red; MI, blue). The average firing rate across both MU responses within the same 300 ms sliding window is also shown (Fig. 2A, green). Figure 2B illustrates the distributions and corresponding DP values for this example session when the leading edge of the 300 ms window was aligned to the start of the motion pulse. Distributions for hits (top) and misses (bottom) are shown separately for our three measures of spike-time correlation and firing rate.

For the example session shown in Figure 2C, the time course of the DP values was generally >0.5 during the preamble period where the probability of a motion pulse was zero. Thus, spike-time correlations and firing rates were slightly higher for hit trials versus miss trials during this period. By comparison, before the motion pulse the DP values for our three measures of spike-time correlation generally were <0.5, especially DP_|R| and DP_MI. Thus, trial-by-trial spike-time correlations using the absolute value of the Pearson correlation (Fig. 2C, red) and mutual information (Fig. 2C, blue) were smaller for hit trials versus miss trials just before the motion pulse (t = 0), but not for the Pearson's correlation (Fig. 2C, black). As illustrated here, there was usually a significant amount of variability in single-session DPs. In addition, we also computed the DP using the average firing rates for the two MU recordings over the same 300 ms sliding window (Fig. 2C, green). For this example session, the trial-by-trial firing rates were generally higher for hits compared with misses, especially as the trial approached the motion pulse.

Spike-time correlations predicted behavioral outcome just before the motion pulse

The population DP time course of our three measures of spike-time correlation and firing rate are shown in Figure 3A. All three DP values based on correlation (R, |R|, and MI) were near 0.5 during the preamble period at the start of the trial. At ∼100 ms before motion pulse onset, the population averages of both DP_MI and DP_|R| (Fig. 3A, blue and red, respectively) began to exhibit the same dynamics with a downward trend that peaked just as the motion pulse began (t = 0). The dynamics of DP_R (Fig. 3A, black) by comparison was relatively flat. At the start of the motion pulse, the population means of DP_MI and DP_|R| were significantly <0.5 (p = 0.0002 and p < 0.0001, respectively, one-sided bootstrap; N = 47), but not that of DP_R (p = 0.09). Note that DP values for individual sessions are shown below (Fig. 4). Thus, just as the motion pulse began, the two measures of spike-time correlation that were independent of the sign (|R| and MI) were reliably smaller on hit trials versus miss trials. Toward the end of the 50 ms motion pulse, DP_MI and DP_|R| values began to rise. The significance of these late dynamics is difficult to interpret, as MT responses begin to be dominated by the coherent motion pulse.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Population DP and spike-time correlation time course. A, For the three measures of spike-time correlation and spike rate, DPs are aligned to either the start of the preamble period (left) or the start of the motion pulse (right). Firing rate (green) and spike-time correlations computed with |R| (red), MI (blue), but not R (black), were predictive of behavioral outcome just as the motion pulse occurred (time = 0). B, Population averages of the three measures of spike-time correlation aligned to the start of the preamble period (left) and the motion pulse (right). Note the change of the ordinate scale between the preamble and motion-pulse alignments. Within each session, trials were grouped according to behavioral outcome: hit (blue) and miss (red). All DPs and averages were computed using the 300 ms sliding window shown in Figure 2A and an MU spike Gaussian smoothing kernel of 1.5 ms. Shading is the SEM.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Comparison of individual DP values for each session. A, DP_|R| vs DP_MI. B, DP_|R| vs DP_R. C, DP_Rate vs DP_|R|. D, DP_|R| vs DP_|R| minus the shuffle subtraction control (see Materials and Methods). Shuffling trials eliminated the ability of |R| to predict behavioral outcome and produced a DP_|R|shuffled value of ∼0.5. For all plots, DPs were computed from the 300 ms analysis window aligned to the start of the motion pulse, and MU spikes were smoothed using a 1.5 ms Gaussian. Triangles correspond to mean values, ρ is the Pearson's correlation of the points shown, and dashed lines are the unity slope. The p values are for the marginal distributions computed using a one-sided bootstrap (N = 47).

Firing rates, by comparison, had the opposite dynamics for predicting behavioral outcome. The DP_Rate (green; Fig. 3A) value was slightly <0.5 during the early preamble, but began an upward trend at the end of the preamble that greatly accelerated during the motion pulse. Interestingly, the magnitude of DP_Rate value at the start of the motion pulse (p = 0.01, one-sided bootstrap; N = 47) was about the same as that of DP_MI and DP_|R|, but in the opposite direction.

This suggests that firing rate and spike-time correlation had similar links to behavioral outcome (equal magnitude, but opposite sign), and is in agreement with other studies that have used across-trial noise correlation measures (Cohen and Newsome, 2008; Cohen and Maunsell, 2011).

Although the number of paired recordings was small (N = 47), our experimental design had several beneficial features. First, the stimulus parameters of the motion pulses were matched to that preferred by the neural populations recorded, which has been shown to increase DP values based on firing rate (Bosking and Maunsell, 2011). Second, we collected a large number of trials for each session (range, 156–1389), which helped to reduce the variability in our DP estimates. Third, our long trials and brief 50 ms stimulus placed a premium upon the animals directing their attention to the RDPs at the time that the motion pulses occurred. Once the motion pulse was over, all stimulus information was lost. Finally, during the preamble period at the start of the trial the animals presumably knew that the motion pulses would not occur. Thus, the neural activity during the preamble provides a control period for interpreting DP values. Note that the average time between the end of the preamble and the start of the motion pulse was similar for hit trials and miss trials (781 and 764 ms, respectively).

Our DP values reflected both the differences in the mean of our correlational metrics and their variability. Figure 3B shows the mean values of our three measures of spike-time correlation during the preamble and before the motion pulse. All three measures show a downward trend during the preamble. This decrease in correlation at the start of a trial has been observed in MT using across-trial measures of correlation (deOliveira et al., 1997; Churchland et al., 2011; Oram, 2011). With the exception of the R values, both |R| and MI values showed an increased separation between hit trials (Fig. 3B, red) and miss trials (Fig. 3B, blue) just before the pulse onset (note the expanded vertical scale for the right column).

The DP values at the start of the motion pulse for individual sessions are shown in Figure 4 (filled triangles are means). The marginal histograms show that for most sessions, DP_MI and DP_|R| values were <0.5. We observed a high degree of correlation (Pearson's ρ = 0.95) between DP_|R| and DP_MI values across sessions (Fig. 4A). Thus, |R| and MI were almost equally predictive of trial outcomes. This is in contrast with the lack of correlation observed between DP_|R| and DP_R values (Fig. 4B). Furthermore, others have reported a relationship between spike rate and noise correlations (Cohen and Kohn, 2011; de la Rocha et al., 2007). However, we observed no relationship between DP_|R| and DP_Rate across sessions (Fig. 4C). Finally, we examined whether the slow decreases in firing rates just before the motion pulse (Fig. 1E) contributed to our DP values. We found that DP_|R| was unaffected when the effects of slow changes in firing rate were removed from our spike-time correlations using a shuffle-subtraction control (Fig. 4D; see Materials and Methods). Thus, when trials were shuffled, the link between |R| and behavioral outcome was eliminated (DP_|R|shuffled = 0.5). The same shuffle result was observed for our two other measures of correlation (data not shown).

The DP results so far suggest that just before the motion pulse has reached area MT, two single-trial measures of spike-time correlation that ignore the sign of the correlation (|R| and MI) were as good at predicting behavioral outcome as firing rate. Thus, both strong positive or negative correlations between the two MU spike trains were more likely to occur when the animals missed the motion pulse. This result raised two questions that we next addressed: what was the timescale of the spike-time correlations that best predicted behavioral outcome? And why was R a poor predictor of behavioral outcome?

The timescale of the spike-time correlations that predict behavioral outcome

We next examined the temporal resolution by which our correlational measures predicted behavioral outcome. For this, we used the analysis window aligned to the start of the motion pulse (Fig. 5, top). First, the MU spike trains were convolved with a Gaussian kernel of variable width, with the SD (σ) of our kernels varying from 0 (no smoothing) to 15 ms (Fig. 5A). The Gaussian kernel provides a notion of the timescale (or frequency range) over which the spike-time correlations were computed. For a kernel width of 0, DP_MI and DP_|R| values were ∼0.5. Gaussian widths between 1.5 and 5 ms generally yielded the strongest DP_MI and DP_|R| values (note the equivalent low-pass cutoff frequency at the top of Fig. 5A for different kernel widths). At a higher σ value, DP values gradually approached chance levels of 0.5. One possibility for why DP_|R| and DP_MI values were equal to 0.5 for a zero-width kernel (no smoothing) is that the unfiltered MU spike trains contained high frequencies that masked the low-frequency contributions. As shown, DP_R was generally a poor predictor of behavioral outcome for all Gaussian kernel widths.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Temporal resolution of the spike-time correlation DPs. A, DPs as a function of the SD (σ) of the MU spike smoothing Gaussian. DP_|R| and DP_MI are the strongest for a σ from 1.5 to 5 ms. The corresponding low-pass cutoff frequency for each is shown on top. B, Correlational DPs as a function of spike jitter. Each MU spike was randomly moved within a window with or without jitter. Increasing jitter steadily reduced correlational DPs toward 0.5. A jitter = 0 corresponds to the same data as in Figure 4. A smoothing Gaussian of 1.5 ms was used during the jitter analysis. C, Population means for the three measures of spike-time correlations as a function of jitter for hit (blue) and miss (red) trials. All plots used the same analysis window aligned to the start of the motion pulse. Shaded area is the SEM.

Jitter methods have been shown to remove stimulus-locked and slow correlations due to fluctuations in rate (Smith and Sommer, 2013). Given that our DPs were strongest for relatively small Gaussian kernel widths, we expected that small amounts of spike jitter would impair the ability of spike-time correlations to predict behavioral outcomes. Thus, DPs were next computed with spikes jittered within a window spanning from 0 to ±30 ms (using a fixed kernel width of 1.5 ms). Increasing the size of the jitter window beyond a few milliseconds had a marked reduction on DP_MI and DP_|R| values (Fig. 5B).

To further illustrate the effect of spike jitter, we plot the mean values of our correlational measures as a function of jitter window size. Spike-time correlations (R, |R|, and MI) rapidly decreased as a function of the size of the jitter window (Fig. 5C). For |R| and MI, the separation between hit (blue) and miss (red) was highest when there was no jitter. As expected, the link between spike rate and behavioral outcome (DP_Rate) was not affected by varying the Gaussian kernel width or jittering spike times (data not shown). Overall, these results suggest that correlations on relatively short timescales were linked to behavioral outcome.

The relationship between DP_R, DP_|R|, and DP_MI

Why did DP_R fail to reliably predict behavior? And why were DP_MI and DP_|R| values similar across sessions? These questions can be addressed by plotting the individual R values versus MI values for all trials where the analysis window was aligned to the start of the motion pulse (Fig. 6A). Although high mutual information does not necessarily imply high correlation values, it is well known that for linearly dependent bivariate Gaussian signals, mutual information is a function of the squared Pearson's correlation coefficient (Pinsker, 1964). As suggested by our data in Figure 6A, our single-trial mutual information values followed the Pearson's spike-time correlation with an exponent of 2.2 (Fig. 6A, red line is fit). This suggests that the trial-by-trial dependencies between our two MU recordings were mostly linear, and that accounting for the negative Pearson's correlation is necessary for predicting a trial outcome.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

The relationship between DP_R, DP_|R|, and DP_MI is dependent upon the sign of the spike-time correlation. All panels show results for the analysis window aligned to the start of the motion pulse (top). A, Single-trial MI and spike-time R values computed from the smoothed MU responses (σ = 1.5 ms) for the entire dataset. Note that only every other data point was plotted for clarity. Single-trial values of MI followed R values with an exponent of 2.2 (red line is fit to all data points). B, Model for how an R value might affect the ROC-based DP metric. If single-trial values of R are both positively and negatively stronger on missed trials, then for all trials DP_R = 0.5 (left). In comparison, trials with R > 0 would produce DP_R < 0.5 (middle), while trials with R < 0 would produce DP_R > 0.5 (right). C, Summary of population DP values using all trials (left bar plots; same data from Fig. 4), trials with R > 0 (middle bar plots), and trials with R < 0 (right bar plots). When trials were grouped by the sign of R, DP_R (black bars) values became larger (i.e., departed from 0.5) and reversed sign, as predicted by the model. There was an approximately equal number of trials with positive and negative R values. D, Individual DP_R and DP_MI values from all sessions for trials with R > 0 (open) and R < 0 (black). Triangles are the means. The p values are one-sided tests for mean DP_R > 0.5 (R < 0) and mean DP_R < 0.5 (R > 0), bootstrap (N = 47). E, DP_R as a function of the SD (σ) of the MU spike smoothing Gaussian for trials with R > 0 (dashed line) and R < 0 (solid line). F, Behavioral performance (proportion hits) for trials with negative spike-time R values vs positive spike-time correlations. The dashed line is unity slope. G, Summary of DPs using single-unit spikes, grouped by the sign of the spike-time correlations. Note that the single-unit spikes were convolved with a 4 ms Gaussian, and a weighted average based on the number of trials was used to compute the mean DPs (see Results). Only trials with at least two single-unit spikes per electrode within the analysis window were included. H, Single-unit DPs as a function of the SD (σ) of the spike-smoothing Gaussian for trials with R > 0 (dashed line) and R < 0 (solid line). Error bars and shading are SEM, and p values are from paired one-sided bootstraps that DP_R (R < 0) is greater than DP_R (R > 0; N = 47).

Figure 6B illustrates the potential confound with using Pearson's spike-time correlation values for computing our ROC metric DP_R. If we assume that missed trials were associated with both stronger positive and negative Pearson's correlations, then this would cause DP values to be pulled toward 0.5 (all trials, left). If this were the case, then examining trials with only positive correlations (middle) or negative correlations (right) would yield DP_R values that depart from 0.5 in opposite directions. Thus, for trials with a positive Pearson's correlation this hypothesis would predict DP_R values <0.5; whereas for trials with negative spike-time correlations, DP_R values would be >0.5.

To test this model, we recomputed DP values grouped on the sign of the Pearson's spike-time correlation (Fig. 6C, analysis window aligned to the start of the motion pulse). There was about an equal number of trials with positive and negative correlations on each session (average ± SD proportion of trials with R > 0, 51.7 ± 6.7%, N = 47; Fig. 7B). Although the mean population DP_MI (Fig. 6C, blue) and DP_Rate (Fig. 6C, green) values remained relatively the same for trials with only positive or negative correlations, DP_R (Fig. 6C, black) values became stronger and deviated significantly from 0.5 in opposite directions depending on the sign of the correlation (DP_R < 0.5 for R > 0 and DP_R > 0.5 for R < 0). The p value in Figure 6C reports the pairwise difference in DP_R between positive and negative correlations (one-sided bootstrap, N = 47). This result confirms that DP_R was a poor ROC-based predictor of trial outcome due to the combined effect of both stronger positive and negative correlations associated with missed trials.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Spike-time R values and FRs modulate behavioral performance. A, Example single-trial R vs FR for hit (blue) and miss (red) trials from an example session (ρ is the Pearson's correlation of all data points). B, Bottom, Distribution of the Pearson's correlation between R and FR (ρ). Top, Distribution of the proportion of trials with R > 0. Histograms include all sessions (N = 47), and triangles are the means. C, Raw and normalized behavioral performance binned on R and FR for an example session. Behavioral performance was normalized by the mean (dashed line). D, Average of the normalized behavioral performance (proportion hits) binned separately for R and FR. For each session, we first normalized the behavioral performance by dividing by the mean. R and FR bin centers are the average of the centers across all sessions. Bin centers were chosen separately for each session so that approximately the same number of trials contributed to each bin. The p values are for the paired comparisons shown, one-sided bootstrap (N = 47). E, Behavioral performance jointly binned on both R and FR values for the example session in A. Data were binned using a 5 × 5 grid (25 bins total), and bin edges were selected so that each bin had approximately the same number of trials. F, Population average of the normalized behavioral performance jointly binned on R and FR values. Bin centers reflect the average across all sessions, and each session was normalized by the mean proportion of hits. Hits were more likely on trials with both higher firing rates and spike-time correlations near 0. G, Average MU firing rates for trials binned by R. Colors correspond to the same bins in D. Note that all five traces significantly overlap and that only trials where the motion pulse occurred in the RF of the MU were included. Representative error bars are shown for clarity. H, Average normalized MU firing rate as a function of R. Normalization was accomplished by dividing single-trial firing rates by the mean firing rate for that session. Firing rates were computed either before the pulse (filled circles and solid line) or right after the pulse (open square and dashed line). Black bars in G indicate the time window used to compute firing rates. Error bars are the SEM and, for some, plots are smaller than the symbols.

Individual session DP_R values are shown in Figure 6D for each group (R > 0, open circles; R < 0, black circles), and they followed DP_MI as suggested by the approximately linear dependence between R and MI highlighted in Figure 6A. The p values in Figure 6D correspond to DP_R values either >0.5 or <0.5 (one-sided bootstrap, N = 47). With the exception of the magnitude relative to 0.5, the effect of the smoothing kernel width (σ) on DP_R was similar for trials with negative and positive R values (Fig. 6E). Additionally, we found that behavioral outcomes were, on average, the same regardless of the presence of either negative or positive neural correlations (Fig. 6F). Thus, the link between neural correlations and behavior appears to be the same regardless of the sign of the spike-time correlation. These observations account for the differences and similarities among DP_R, DP_|R|, and DP_MI values.

Finally, we sought to evaluate whether the results obtained using MU responses were in agreement with our single-unit recordings. It has been suggested that spike sorting can increase or decrease across-trial measures of correlation (Cohen and Kohn, 2011). Estimating single-unit spike-time correlations was difficult because some MT neurons had relatively low firing rates. Thus, an estimate of the single-trial spike-time correlation required a minimum of at least two spikes from each single unit to fall within the 300 ms analysis window. Trials with only a single spike or no spike in the analysis window were excluded. We also used a slightly wider Gaussian kernel width of 4 ms to smooth the spikes to reduce the effects of the low firing rates. To compensate for the reduced number of trials for some single units, the population DP values shown in Figure 6G are a weighted average, where the weighting was based on the total number of trials used to compute the DP. Although the single-unit DP values were weaker than the MU data for our three correlational measures and firing rate, they reproduced the same trends when grouped by positive or negative correlations. Notably, single-unit DP_R (Fig. 6G, black bars) showed a significant pairwise flip from <0.5 to >0.5 when separated by the sign of the correlations (one-sided bootstrap, N = 47). Across different smoothing kernel widths, the single-unit DP_R had a similar shape as that for the MU data (compare Fig. 6E,H).

Detection performance versus single-trial spike-time correlations (R)

We have shown that when the sign was accounted for, the single-trial Pearson's correlation of the two Gaussian smoothed MU responses before the motion pulse predicted behavior in our detection task. What is not revealed by the DP metrics used above, however, is the magnitude of this effect or a potential mechanism. For example, why would both positive and negative correlations before the stimulus lead to reduced detection performance? And how might both positive and negative spike-time correlations arise?

One important aspect of our data was the large variability in the single-trial spike-time R value within each recording session. The scatter plot in Figure 7A illustrates this variability for a single session for hit trials and miss trials (note the analysis window was aligned to the onset of the motion pulse). It has been reported that measures of correlation increase with firing rate (de la Rocha et al., 2007; Cohen and Kohn, 2011). As illustrated by the example session in Figure 7A, we did not observe this relationship between R and FR (Pearson's correlation ρ = 0.002 using all trials). Across sessions, the average ± SD Pearson's correlation between R and FR was 0.02 ± 0.09 (N = 47), as shown by the histogram in Figure 7B (bottom). As mentioned above, we also typically observed an almost equal number of positive and negative single-trial values of R within a session (Fig. 7B, top).

To illustrate the magnitude of how detection performance changed as a function of single-trial spike-time R values and FRs, we binned trials on either R or FR, and computed the proportion of hits (i.e., correct outcomes) within each bin (Fig. 7C, example session). Note that bin edges were chosen to produce an approximately equal number of trials per bin and that the middle correlational bin was centered on r = 0. In the example session in Figure 7C, the highest proportion of hits was associated with values of R near zero and high FRs. For each session, we normalized how behavior was modulated by R and FR by dividing by the mean. For the example session shown, hits were modulated by approximately ±6% about the mean (12% total modulation) as a function of R. FR by comparison was associated with a ±10% modulation of hits about the mean.

The average modulation of the proportion of hits is shown in Figure 7D as a function of R and FR. Both the strongest positive (Fig. 7D, green) and negative (Fig. 7D, blue) correlations significantly reduced detection performance by ∼9% when compared with trials with little or no correlation (Fig. 7D, yellow). The p values in Figure 7D are paired comparisons (one-sided bootstrap, N = 47). As shown on the right, there was a similar modulation in detection performance between the smallest and largest FRs.

To further illustrate how spike-time R and FR values jointly contributed to behavioral outcome, we binned trials according to both their firing rate and correlation values (25 bins), and examined the proportion of hits within each bin. Because of the large number of bins, the joint contribution of R and FR on detection performance was relatively noisy for a single session (Fig. 7E). However, the population average across sessions revealed that high FRs and low correlations were associated with more hits (Fig. 7F). In comparison, the fewest hits occurred in bins associated with low firing rates and high positive and negative correlations.

Both positive and negative spike-time correlations are associated with increased variance.

To examine potential mechanisms for the modulation of hits as a function of R, we first confirmed that the average firing rate for each correlation bin had the same dynamics (Fig. 7G, colors correspond to the same points in D). As shown, there was no effect on the average MU time course, before or after the motion pulse, when trials were binned by the spike-time R value (computed in the 300 ms before the motion pulse). Note that only trials where a motion pulse occurred in the RF of the MU were included. Figure 7H summarizes this result by plotting the average normalized firing rate as a function of R using the two analysis windows shown in Figure 7G (before and after pulse, black bars). Thus, the mean MU response does not seem to explain the modulation in behavioral performance associated with spike-time correlations.

The variance of sensory information is commonly thought of as a major factor during decision-making (Churchland et al., 2011; Zylberberg et al., 2016). If downstream brain areas are integrating MT population activity to detect the weak motion pulses, then the amount of variance in the integrated response before the pulse becomes important. For example, we simulated the integration of our MU activity by convolving spikes with an exponential kernel (τ = 50 ms; Fig. 8A). The integrated MU response after the motion pulse would likely be more salient when preceded by activity with less variance (Fig. 8A, low variance, left). Likewise, a higher variance of the integrated response before the motion pulse could make the response after the pulse harder to detect (Fig. 8A, high variance, right).

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Positive and negative single-trial spike-time R values are associated with increased variance in the neural response. A, Two example trials where the MU spikes were integrated with an exponential filter (time constant, 50 ms). Low variance of the integrated response (σ²) before the motion pulse could make the response to the motion pulse more salient (left). In contrast, the high variance of the integrated response before the motion pulse could have the opposite effect (right). The effect of variance before the motion pulse can be captured by defining an SNR as the ratio of the mean integrated response after the motion pulse (μ) to the variance of the integrated response before the motion response (σ²). B, The average modulation in the variance of the integrated MU response before the motion pulse as a function of R for the example session in Figure 7. Shown is the variance of one electrode. The y-axis shows both the raw variance (left) and the variance normalized to the mean (right). The analysis window was aligned to the start of the motion pulse, and the MU smoothing kernel was σ = 1.5 ms. For all plots in this figure, R was binned in the same manner as shown in Figure 7. C, Example of the average modulation in the mean integrated response after the motion pulse as a function of R. Note that only trials where the motion pulse occurred in the RF of the MU were included. D, Example of the average modulation of SNR as a function of R. SNR was computed for each trial as shown in A, and then averaged across trials for each session. E, Population average modulation of the normalized variance of the integrated MU response as a function of R. Shown is the variance computed before and after the motion pulse. F, Population average modulation in the normalized mean integrated response as a function of R. Shown is the mean response computed before and after the motion pulse. G, Population average modulation of the normalized SNR as a function of R computed for a range of integrator time constants (5–100 ms). Only error bars for a τ = 50 are shown for clarity. For E–G, each of the 47 sessions contributed two MU recordings. R is the Pearson's correlation of the MU response computed in the 300 ms window before the motion pulse (same data as in Fig. 5). H, Functional model of the two MT sensory pools. Simulated MU activity was generated with a Gaussian distributed random variable where the mean increased in response to the motion pulse (for details, see Materials and Methods). The simulated MU was then integrated with a 50 ms exponential. An additive common noise input (also modeled as a Gaussian random variable) was applied to the output of each sensory pool. To produce both positive and negative spike-time correlations of the simulated MU response, a variable delay changed the relative phase of the common noise arriving at one pool on every trial between 1 and 25 ms. I, The variance of the model and the SNR of the integrated response as a function of R of the simulated MU activity (average of 300,000 trials). Note that the R value of the simulated MU of the model was computed in the same manner as the recorded MU activity.

Thus, we examined the variance of our integrated MU responses as a function of the spike-time R value before the motion pulse (note that the R value was binned the same as in Fig. 7).

As illustrated by the example session, we commonly observed that the variance of the integrated MU responses before the motion pulse (Fig. 8A, σ²) was less for values of R near zero (Fig. 8B). Note that this is the same example session shown in Figure 7 with the same binning on R. The population-normalized variance of the integrated response as a function of R (Fig. 8E, filled circles) showed a similar shape. In comparison, the variance of the integrated MU response after the motion pulse was relatively flat as a function of R (Fig. 8E, open squares).

As shown by the example session, the mean integrated MU response after the motion pulse (Fig. 8A, μ), was only weakly modulated by spike-time correlations (Fig. 8C), especially for the population average (Fig. 8F, open squares). The mean population response before the motion pulse was also a relatively flat function of R (Fig. 8F, filled circles).

The variance of the integrated response before the motion pulse suggests a way to link spike-time R values to the modulation in behavioral hits shown in Figure 7D. We quantified the ability of the integrated MU response to signal the occurrence of motion using an SNR measure that was a function of the mean integrated response after the pulse (Fig. 8A, μ) and the variance of the integrated response before the pulse (Fig. 8A, σ²). Figure 8D shows that the average SNR for the example session is modulated by R, such that the highest SNR corresponds to the smallest values of R. Across our population, the average normalized SNR of our integrated MU responses (Fig. 8G, black) was modulated by R in a similar manner as the proportion of hits (compare Fig. 7D). Note that only trials where the motion pulse occurred in the RF of the MU were used to compute SNR.

For every session, the SNR was computed for each trial individually and then averaged across trials for each R bin (Fig. 8D). One potential problem with this approach was that trials with very low variances could produce exceedingly high SNR outliers that bias the average session SNR. We also computed the median SNR for each session and observed the same SNR shape as shown in Figure 8G (data not shown). Thus, our SNR values were not biased by extreme outliers.

As we do not know how downstream areas integrate MT responses, we computed SNRs for a range of integration time constants (Fig. 8G). In all cases, the highest population SNR corresponded to spike-time R values near 0; however, the magnitude of the SNR modulation decreased for smaller integration time constants.

It has been suggested that neural correlations may stem from global fluctuations of neural activity (Ecker et al., 2014). Thus, a common input that modulated our two MT neural pools could lead to both increased spike-time correlations and variance. In this scheme, strong spike-time correlations (either positive or negative) would be associated with an increase in the variance of the integrated MT activity, making it more difficult for downstream areas to detect the brief motion-pulse transient.

To examine this hypothesis, we constructed a simple computer model based on random Gaussian distributed signals to simulate our two MT MU responses (Fig. 8H; see Materials and Methods). A common additive noise input created spike-time correlations between the two simulated MU responses. This common noise alone, however, was unable to produce the same range of negative spike-time correlations observed in our data. To mimic both negative and positive correlations observed in our data required adding a small amount of temporal jitter (D in Fig. 8H, 1–25 ms) to the common input arriving at one simulated MT pool. As shown in Figure 8I, across 300,000 simulated trials, this produced variance and SNR plots of the integrated response of the model as a function of R that were similar in shape to that of our data.

Our model suggests that variability in the relative phase of the common noise arriving at the two MT pools is sufficient to produce negative correlations and add variance to the output of each MT sensory pool. For example, variability in the relative phase of the two noise inputs could arise if the animals were dynamically moving their attentional focus between the two stimuli. However, we can only speculate as to whether the variable delay in the model is a physiologically realistic mechanism.

Other contributions to the link between spike-time correlations and behavior

Last, we examined the following three potential contributions to the trial-by-trial spike-time correlations in our MU responses and their corresponding link to behavioral outcome: (1) fluctuations in our random dot stimulus; (2) small eye movements; and (3) the variance of our electrode signals. Just before the motion pulse occurred, the two RDPs contained independent 0% coherent motion. Although the 0% coherent stimulus was designed to have no net motion, small stochastic fluctuations were present in the proportion of dots moving in either the preferred or null directions. We first asked whether these small fluctuations contributed to the link between MU spike-time correlations and behavioral outcome.

In 36 of 47 sessions, we had saved the dot pattern for every stimulus frame. From these sessions, we were able to estimate the frame-by-frame motion strength by computing the net number of dots that moved in either the preferred or the null direction (similar to Cook and Maunsell, 2004). Figure 9A shows a single-trial example of the frame-by-frame net motion aligned to the pulse onset (t = 0), where values >0 indicate a net preferred-direction motion and values <0 represent a net null-direction motion. For each RDP, we normalized the motion during the 0% coherence period to have an SD of 1 across all trials (Fig. 9A, note scale bar).

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

The contributions of the stimulus, small eye movements, and electrode variance to the link between single-trial spike-time correlations values and behavioral outcomes. A, In 36 sessions, we were able to extract the dot pattern of the RDP stimulus. Example random fluctuations in the motion for each RDP. Motion was described as the net number of dots moving in the preferred or null direction (see Materials and Methods) and normalized by the session SD. B, Example MU STAs of the motion from a single session. Orange is the STA using the motion in the other RDP. C, Population average MU STAs. D, The correlation of the motion in the two RDPs (R for motion) vs the MU spike-time R value. E, DP_|R| was also computed using the RDP motion instead of the MU spikes. Across sessions, the average motion-based DP_|R| value was near 0.5 and not strongly correlated with DP_|R| values computed from the MU spikes (ρ). F, Example trial with three putative microsaccades (*) within the analysis window (gray) just before the motion pulse. G, Population microsaccade-triggered average of MU activity computed during the 0% coherent motion (orange is the shuffled control of microsaccade times). Note that the microsaccade onset time was shifted −40 ms to account for the 20 Hz low-pass filtering of the recorded eye signals (see Materials and Methods). H, The absolute value of the MU spike-time correlations (i.e., |R|) vs the number of microsaccades for hit trials and miss trials. A total of 25% of trials had putative microsaccades within the analysis window just before the motion pulse. I, Normalized microsaccade rate vs MU spike-time R values. J, DP_|R| computed with the MU spikes with and without microsaccades. Removing the 25% of trials containing microsaccades had little effect on DP_|R|. K, Electrode waveforms for an example trial. Waveforms were normalized by their session SD. L, MU |R| as a function of the binned average electrode variance for hit trials and miss trials. Bin edges were set on each session so that an equal number of trials contributed to each bin. M, Normalized electrode variance vs MU spike-time correlations. N, DP_|R| computed with the MU spikes with and without the highest electrode-variance trials. Removing the 33% of trials with the highest variance had little effect on DP_|R| values. Note that in all plots, data are from an analysis window aligned to the start of the motion pulse, except for the motion description (A), which was shifted by −50 ms to accommodate for the latency of MT. Also note that the MU DP_|R| values for all trials are the same data as in Figure 5. Error bars and shaded areas are the SEM. The p values in E, J, and N are for the mean DP values <0.5; one-sided bootstrap and small triangles are the mean DP values.

To assess our characterization of the RDP motion, we computed the MU spike-triggered average (STA) of the motion during the 0% coherence. As shown by the example STAs from a single session (Fig. 9B), MU spikes tended to follow motion in the preferred direction with a latency of ∼50 ms (Fig. 9B, black). In addition, we found that STAs were flat (Fig. 9B, orange) when computed using the motion in the other RDP located outside the RF. Our population average STA (Fig. 9C) suggests that our MU spikes were correlated with the frame-by-frame motion description and that there was little contribution from the motion in the RDP located outside the RF.

We next examined whether correlations in the frame-by-frame motion stimulus (R motion) were associated with MU spike-time correlations (R). Note that the R value is from the same data presented above and that the analysis window was aligned to the start of the motion pulse. The window for the motion stimulus was shifted 50 ms to account for the MT latency observed in our STAs (Fig. 9A, gray). As shown in Figure 9D, correlations in the motion did not seem to be related to spike-time correlations. Furthermore, to determine whether the correlations in the frame-by-frame motion predicted behavioral outcome, we computed the DP_|R| value using the motion stimulus and compared it with DP_|R| value computed from the MU responses (Fig. 9E). Although our subpopulation of 36 sessions still showed most MU DP_|R| values <0.5 (Fig. 9E, triangles are the mean), the mean DP_|R| value computed from the frame-by-frame motion stimulus was not significantly different from 0.5 (p = 0.33, by bootstrap; N = 36). Note that there was no significant correlation between the MU and motion DP values (Pearson's ρ = 0.21, p = 0.17). Although the STAs show that our MU responses were linked to the stochastic fluctuation in the 0% coherent motion stimulus, these stimulus-driven fluctuations did not appear to account for the link between MU spike-time correlations and behavioral outcome.

Care must be taken when interpreting this result, however, as we cannot rule out all aspects of the RDP motion as a potential contribution. For example, local dot interactions within the RF that would not be captured by our frame-by-frame global motion description could potentially contribute to spike-time correlations. Another possibility is that chance-correlated movement between the dots on the nearest edges of the two RDPs could simultaneously contribute to both MT pools. Although not used here, one way to better measure the stimulus contributions to spike-time correlation would be to include repeated stimulus dot patterns and examine whether correlations vary under identical 0% motion (Wimmer et al., 2015).

Next, we examined the effects of small eye movements on our MU responses. During a trial, small eye movements such as microsaccades can modulate neural activity in area MT (Herrington et al., 2009) and areas that drive MT such as V1 (Snodderly et al., 2001; McFarland et al., 2016), which could introduce spike-time correlations between our two MT sensory pools. We extracted the occurrence of putative microsaccades as shown by the example trial in Figure 9F (asterisks indicate that three saccades in the 300 ms analysis window aligned the motion pulse; see Materials and Methods). Note that due to the limitations of our eye-tracking system, eye signals were low-pass filtered at 20 Hz, which limited our ability to detect high-frequency saccades (see Materials and Methods). We assessed the relationship between our putative microsaccades and neural activity by computing a saccade-triggered average of the MU spikes (Fig. 9G). Consistent with previous results (Martinez-Conde et al., 2000), we observed a significant modulation in firing rates around the time of these small eye movements (Fig. 9G, orange is the control MU firing rate aligned to randomly shuffled microsaccades).

Given that microsaccades modulated the response of our MU recordings, we also assessed the relationship between the number of microsaccades and the absolute value of the MU spike-time correlation (|R|) in the 300 ms analysis window aligned to the motion pulse (Fig. 9H). We detected putative microsaccades within this window on 25% of trials, and these trials tended to have higher values of MU |R| for both hit (Fig. 9H, blue) and miss (Fig. 9H, red) outcomes. Interestingly, the rate of microsaccades was associated with increased positive spike-time R values, but not negative R values (Fig. 9I).

Figure 9, H and I, raised the possibility that microsaccades contributed to the spike-time correlations in our MU recordings. However, when the 25% of trials containing putative microsaccades were removed from the analysis, DP_|R| values remained relatively unaffected (Fig. 9J).

Furthermore, eye-velocity variance had the same effect on both MU spike-time correlations and DP_|R| values as our putative microsaccades (data not shown). It is possible that our microsaccade detection may have missed fast eye movements. Thus, we cannot fully rule out the potential contribution of small eye movements to the link between spike-time correlations and behavioral outcome.

Finally, we sought to address the possibility that large recording anomalies that occurred simultaneously in both electrodes contributed to the relationship between spike-time correlations and trial outcome. We hypothesized that if nonphysiological sources of noise produced large fluctuations in the variance of both electrode waveforms (Fig. 9K, example electrode recordings), then removing these high-variance trials would result in a weaker DP_|R|. Using the 300 ms analysis window aligned to the motion pulse (Fig. 9K, gray) and binning the average electrode variance with an equal number of trials per bin, we observed a weak but positive relationship between the average electrode variance and MU |R| for both hit (Fig. 9L, blue) and miss (Fig. 9L, red) trials. The difference between hit and miss MU |R| values, however, became smaller as electrode variance increased. Unsurprisingly, we also observed that electrode variance increased for both positive and negative spike-time correlations (Fig. 9M). However, DP_|R| values were relatively unchanged when the 33% highest-variance trials were removed (Fig. 9N). This suggests that trials with large electrode anomalies were not primary contributors to the link between MU spike-time correlations and behavioral outcome.