Abstract
Responses of individual task-relevant sensory neurons can predict monkeys' trial-by-trial choices in perceptual decision-making tasks. Choice-correlated activity has been interpreted as evidence that the responses of these neurons are causally linked to perceptual judgments. To further test this hypothesis, we studied responses of orientation-selective neurons in V1 and V2 while two macaque monkeys performed a fine orientation discrimination task. Although both animals exhibited a high level of neuronal and behavioral sensitivity, only one exhibited choice-correlated activity. Surprisingly, this correlation was negative: when a neuron fired more vigorously, the animal was less likely to choose the orientation preferred by that neuron. Moreover, choice-correlated activity emerged late in the trial, earlier in V2 than in V1, and was correlated with anticipatory signals. Together, these results suggest that choice-correlated activity in task-relevant sensory neurons can reflect postdecision modulatory signals.
SIGNIFICANCE STATEMENT When observers perform a difficult sensory discrimination, repeated presentations of the same stimulus can elicit different perceptual judgments. This behavioral variability often correlates with variability in the activity of sensory neurons driven by the stimulus. Traditionally, this correlation has been interpreted as suggesting a causal link between the activity of sensory neurons and perceptual judgments. More recently, it has been argued that the correlation instead may originate in recurrent input from other brain areas involved in the interpretation of sensory signals. Here, we call both hypotheses into question. We show that choice-related activity in sensory neurons can be highly variable across observers and can reflect modulatory processes that are dissociated from perceptual decision-making.
Introduction
When well-trained macaque monkeys make perceptual decisions, the activity of suitably tuned sensory neurons often predicts the animal's impending choice (Britten et al., 1996; Nienborg et al., 2012). This correlation between neuronal and behavioral responses has long intrigued neuroscientists, as it appears to provide a direct demonstration of the ways in which neuronal activity shapes perceptual experience. Specifically, choice-correlated activity has been interpreted as evidence for models in which perceptual decisions arise directly from simple decoding of stochastic sensory responses (Shadlen et al., 1996; Haefner et al., 2013; Kanitscheider et al., 2015; Pitkow et al., 2015). In these models, sensory neurons undergo collective noise fluctuations that influence the outcome of the decision process and hence create a correlation between neuronal and behavioral responses. More complex interpretations are possible because correlations need not imply feedforward causality (Nienborg and Cumming, 2009). Indeed, choice-correlated activity in task-relevant neurons can also be explained by an alternative class of models in which decisions arise from an inference process that involves both feedforward and feedback computations (Wimmer et al., 2015; Haefner et al., 2016). In these models, choice-correlated activity in sensory neurons in part reflects belief states fed back from other brain areas involved in decision making. In both classes of models, choice-correlated activity is inherent to decision making.
We investigated the relationship between the activity of orientation-selective cells in V1 and V2 and simultaneously measured psychophysical judgments of orientation in two macaque monkeys. Both animals performed the task at a level that approximated human performance, and both had V1 and V2 neurons with high orientation sensitivity. But neuronal and behavioral responses were significantly correlated in only one animal. Surprisingly, the sign of this relationship was negative: when an orientation-selective neuron fired more spikes, the animal was less likely to make a decision in favor of the orientation preferred by that neuron. Moreover, this correlation emerged relatively late in the trial, earlier in V2 than in V1, and more strongly when neurons exhibited stronger anticipatory activity before stimulus onset. These results demonstrate that choice signals are not an inherent feature of neuronal activity in visual cortex during perceptual decision making. Moreover, our results call into question the hypothesis that choice-correlated signals play a causal role in the formation of perceptual decisions. Rather, our findings suggest the existence of a mechanism that can shape choice-correlated activity in sensory neurons after decision formation is completed.
Materials and Methods
Observers.
Three male humans (R.L.T.G., C.M.Z., G.M.S.) and two male macaque monkeys (1 Macaca mulatta, 1 M. nemestrina) were trained to perform an orientation discrimination task. All observers had normal or corrected-to-normal vision. Experimental procedures for humans were approved by the Institutional Review Board at New York University. Experimental procedures for monkeys conformed to the National Institute of Health Guide for the care and use of laboratory animals and were approved by the New York University Animal Welfare Committee. Under general anesthesia, both animals were implanted with a titanium head post and recording chamber (Adams et al., 2007, 2011). Eye position was recorded with a high-speed, high-precision eye tracking system (EyeLink 1000).
Unit recording.
Extracellular recordings were made with dura-penetrating glass-coated tungsten microelectrodes (Alpha Omega), advanced mechanically into the brain. We distinguished V1 from V2 on the basis of depth from the cortical surface and changes in the receptive field location of the recorded units. We made recordings from every single unit with a spike waveform that rose sufficiently above noise to be isolated. We first presented suitably vignetted sinusoidal gratings to map each isolated unit's receptive field in a fixation task. Thereafter, we determined preferred stimulus size, spatial frequency, and drift rate and measured neuronal selectivity for orientation. We inspected the orientation tuning curve by eye and selected the orientation that coincided with its steepest part as discrimination boundary for the orientation discrimination task. After data collection was completed, we measured strength of neuronal tuning in the orientation discrimination task by computing the correlation between (relative) stimulus orientation and response mean. For orientation-selective neurons with a suitably chosen discrimination boundary, the absolute value of this correlation is close to one. Small values occur when orientation selectivity is very weak, or when the discrimination boundary is misplaced. Data are reported from every unit for which the absolute value exceeded 0.5 (Monkey 1: 121 of 125 experiments, Monkey 2: 114 of 116 experiments).
Behavioral task.
Subjects were seated in a dimly lit room in front of a gamma-corrected CRT monitor (iiyama HM204DTA) with their heads stabilized (by means of a surgically implanted head post for monkeys, and a chin and forehead support for humans). We presented visual stimuli at a viewing distance of 57 cm, a spatial resolution of 1280 × 960 pixels, and a refresh rate of 120 Hz. Stimuli were presented using Expo software (http://corevision.cns.nyu.edu/expo/) on an Apple Macintosh computer. Each trial in the orientation discrimination task began when subjects fixated a small white point at the center of the screen (0.2° diameter). After 250 ms, two choice targets appeared, one on each side of the fixation point (on the horizontal meridian, at 3.5° eccentricity). The choice targets were white lines (0.3° wide, 2.0° long) rotated −22.5° (choice target on the left) and 22.5° (choice target on the right) away from the discrimination boundary. After a 500 ms delay, a drifting grating appeared. In the animal experiments, the stimulus was positioned within the neuron's receptive field. Subjects judged the orientation of the stimulus relative to the discrimination boundary. The stimulus remained on for 500 ms. When the stimulus disappeared, the fixation point turned black and subjects reported their decision with a saccadic eye movement to the choice target whose orientation was closest to the stimulus orientation. If monkeys made a saccade to the correct choice target, they received a liquid reward. We varied stimulus orientation over a 30° range centered on the discrimination boundary. Stimuli were presented in random order. Zero signal stimuli, stimuli whose orientation matched the discrimination boundary, were rewarded randomly. Trials in which the subject did not maintain fixation within 0.6° (0.75° for Monkey 2) of the fixation point were aborted. Data are reported from every experiment for which at least 100 trials were completed.
Analysis of behavioral response.
We measured observers' behavioral capability to discriminate stimulus orientation by fitting the relationship between stimulus orientation and probability of “clockwise” choice with a psychometric function consisting of a lapse rate and a cumulative Gaussian function. Model parameters were optimized by maximizing the likelihood over the observed data, assuming responses arise from a Bernoulli process. We defined decision criterion and estimation noise as the mean and SD of the cumulative Gaussian. In the vast majority of experiments, lapses were rare, indicating that observers' responses were almost always informed by the stimulus. Experiments in which the lapse rate exceeded 15% were excluded from the population analysis (Monkey 1: 7 of 125 experiments; Monkey 2: 0 of 116 experiments).
Analysis of neuronal response.
We computed each neuron's stimulus response by counting spikes in a 500 ms window following response onset. For each cell, we chose a latency by maximizing the stimulus-associated response variance (Smith et al., 2005). We measured neuronal capability to discriminate stimulus orientation by fitting the relationship between stimulus orientation and probability of “clockwise” choice for an ideal observer with a cumulative Gaussian function. The ideal observer's choices were obtained by applying a deterministic decision criterion to the responses of each neuron. To minimize bias in the ideal observer's choices, the criterion was set to the median response to the zero signal stimulus. We defined neuronal estimation noise as the SD of the cumulative Gaussian. For each cell, we computed an expectancy index (EI) by taking the difference between baseline response and prestimulus onset response and dividing this term by their sum. We defined baseline response as the spike count between −500 and −250 ms, and prestimulus onset response as the spike count between −250 and 0 ms, whereby time is expressed relative to stimulus onset. For each cell, we computed choice probability (CP) for the zero-signal stimulus and the neighboring stimulus conditions. As described previously (Britten et al., 1996), CP is calculated by performing a receiver operating characteristic analysis on the choice-conditioned neuronal responses to repeated presentations of a single stimulus. We classified a CP estimate as statistically significant if it fell outside of the central 95% of the expected null distribution, computed from 1000 randomly permuted datasets (Britten et al., 1996).
Results
Two macaque monkeys performed a fine orientation discrimination task for stimuli in the near periphery while we recorded the activity of individual orientation-selective neurons in area V1 and V2. The animals judged the orientation of drifting sinusoidal gratings relative to a specified discrimination boundary (Fig. 1A). Stimulus orientation varied randomly across trials in a narrow range centered on the discrimination boundary (Fig. 1B). To compare neuronal and behavioral responses in a meaningful way, we tailored the stimulus and task to the tuning properties of the neuron under study (see Materials and Methods). Grating position, size, spatial frequency, and speed matched the neuron's preference, and the discrimination boundary was chosen to coincide with the steepest part of the neuron's orientation tuning curve, thus maximizing the task-related information encoded by the neuron's activity.
Measurement of orientation discrimination in the near periphery. A, Sequence of events in the orientation discrimination task. The subject fixated a central spot. After 250 ms, two choice targets (oriented lines) appeared. Subjects were trained to estimate an orientation discrimination boundary lying midway between the choice target orientations. 500 ms later, a grating appeared in the neuron's receptive field (indicated by the black circle). The subject judged whether the orientation of this grating was rotated clockwise or counterclockwise from the estimated discrimination boundary. 500 ms later, the stimulus disappeared, a go-cue appeared (the fixation mark turned black), and the subject communicated its decision with a saccade toward one of the two choice targets. B, Example stimuli, spanning a range of 60 degrees. For ease of presentation, orientation is expressed relative to the discrimination boundary. In our experiments, the range was 30 degrees. C, An example dataset, consisting of 261 trials. Top, Black symbols represent measured behavioral judgments. Full line represents the fit of a signal detection theory (SDT) model of choice behavior. Bottom, The inferred decision criterion (black dashed line) and the inferred distribution of orientation estimates for a stimulus of −10 degrees. The fraction of orientation estimates that exceeds the criterion determines the proportion of clockwise choices. D, Distribution of lapse rate (top), decision criterion (middle), and noise SD (bottom) for two monkeys and a group of human observers across a large number of experiments.
Both animals performed the task well. When the stimulus orientation was far from the discrimination boundary, they made few errors (Fig. 1C). When the stimulus orientation was close to the boundary, their judgments were less reliable. To distinguish perceptual sensitivity from bias, we fit the subjects' behavioral reports with a model in which choices arise from applying a deterministic decision criterion to a noisy orientation estimate (Fig. 1C; see Materials and Methods). Low noise variance indicates high perceptual sensitivity; large differences between decision criterion and discrimination boundary indicate strong bias. The model revealed that both animals typically had high sensitivity and little bias (Fig. 1D). Median noise SD was 3.4 degrees for Monkey 1, and 2.8 degrees for Monkey 2. This difference, although small, was significant (p < 0.001, Wilcoxon rank sum test) and likely arose from an interanimal difference in receptive field eccentricity (median eccentricity was 5.0 degrees for Monkey 1, and 3.8 degrees for Monkey 2, p < 0.001), as perceptual sensitivity for orientation decreases sharply with eccentricity (Mäkelä et al., 1993).
Sensitivity varied across experiments (Fig. 1D). The stimulus and task were tailored to each neuron's preferences. The variation in sensitivity was in part caused by across-experiment variation in stimulus properties, such as grating size (p = 0.016, F(1,237) = 5.93, analysis of covariance) and spatial frequency (p = 0.026, F(1,237) = 5.02).
We wondered how the animals' perceptual abilities compared with those of human observers. We measured orientation discrimination performance in a group of human observers using stimulus and task conditions that were representative of the ones used in the animal experiments (see Materials and Methods). Compared with the monkeys, humans had similar levels of lapse rate and bias, slightly higher sensitivity (median noise SD: 1.9 degrees), and similar levels of dispersion of sensitivity across experiments (Fig. 1D). We conclude that the behavior of both animals is comparable with humans performing orientation discrimination near psychophysical threshold.
Perceptual decisions depend not only on sensory stimuli and noise, but also on recent experience. To determine the role of such history effects, we fit the observers' behavioral reports with a probabilistic choice model that included a reward-history term in addition to stimulus orientation and overall choice bias. Consistent with many other studies, we found that both the animals and the human observers occasionally exhibited a history bias (Lau and Glimcher, 2005; Busse et al., 2011; Rabinowitz et al., 2015; Abrahamyan et al., 2016). Because this bias varied across experiments and did not show any systematic relationship to neural activity, we opted to use the sensitivity estimates obtained under the simpler choice model shown in Figure 1C for the remaining analyses.
Stimulus sensitivity of sensory neurons
In both animals, small stimulus changes produced substantial changes in average neuronal response. This can be seen in the responses of individual cells (Fig. 2A), as well as in the population average (Fig. 2B). Rotating the stimulus 15 degrees away from the discrimination boundary changed the mean response by ∼20 spikes per second in both V1 and V2. The mean response depended approximately linearly on orientation over the entire stimulus range (Fig. 2A,B). V1 neurons tended to fire more spikes than V2 neurons (V1: median of mean response to center stimulus = 48 ips; V2: median = 35 ips; p = 0.015, Wilcoxon rank sum test). Both animals had nearly identical average patterns of neuronal activity in V1 and V2 (Fig. 2B).
Comparison of behavioral and neuronal orientation sensitivity. A, Responses of an example V1 (left, recorded from monkey 1) and V2 (right, recorded from monkey 2) neuron, stimulated with gratings with different orientations. Responses were computed by counting spikes in a 500 ms window following response onset. The right-hand example data were obtained simultaneously with the behavioral measurements shown in Figure 1C. Ips is impulses per second. B, Mean response of a population of V1 and V2 neurons recorded from Monkey 1 (left) and Monkey 2 (right). Before averaging, negatively signed tuning curves were reflected over the center orientation. C, Distribution of neuronal responses (red, black, and orange histograms) to three stimulus orientations (−15, 0, and 15 degrees) for the example V2 neuron. The median response to the center orientation was used as decision criterion (dotted line) in the ideal observer analysis. D, Neurometric function of the example V2 neuron (dashed, gray line) obtained through an ideal observer analysis, together with the simultaneously measured psychometric function, reproduced from Figure 1C (full, black line). E, Distribution of the ratio of neuronal and behavioral noise SD for a population of V1 (top) and V2 (bottom) neurons recorded from Monkey 1 (left) and Monkey 2 (right).
Neuronal responses are probabilistic: repeated presentations of the same stimulus yield different spike trains (Tolhurst et al., 1983; Goris et al., 2014). This variability limits neuronal sensitivity. To estimate neuronal sensitivity, we performed an ideal observer (maximum likelihood) analysis by applying a deterministic decision criterion to the responses of each neuron (Fig. 2C). When plotted as a function of stimulus orientation, the average choice behavior of the ideal observer traces out a neurometric function (Fig. 2D). The steepness of this function is determined by the neuron's sensitivity. In many of our experiments, the neurometric function closely resembled the psychometric function that summarized the animal's choice behavior (Fig. 2D). Overall, the animals' sensitivity was approximately two-thirds better than that of single V1 or V2 neurons (Fig. 2E). There was no systematic difference between V1 and V2 neurons (median ratio of neuronal and behavioral noise SD: V1 = 175% Monkey 1 = 140%; Monkey 2 = 180%; V2 = 160% Monkey 1 = 168%; Monkey 2 = 149%, p = 0.81, Wilcoxon rank sum test).
Interanimal differences in anticipatory signals
Not all features of neural activity were similar across animals. We found a neuronal correlate of anticipation in one animal, but not in the other. The sequence of events within each trial of the orientation-discrimination task was stereotyped (Fig. 1A). The time of stimulus onset was thus predictable. In anticipation, some neurons exhibited a gradual increase of activity (Fig. 3A), which was not triggered by stimulation of the receptive field, and was unrelated to the most recent change to the visual display. Its origin must therefore be internal. Anticipatory signals were stronger in V2 than in V1. We quantified anticipatory signals with an expectancy index (EI) computed as the difference between prestimulus onset response and baseline response, divided by their sum (see Materials and Methods). This index captures the average temporal evolution of neural activity before stimulus onset in Monkey 1 (Fig. 3A). Its value ranges between −1 and 1, with positive values indicating an increase in activity in anticipation of stimulus onset.
Analysis of anticipatory signals in V1 and V2. A, Top, Response dynamics in anticipation of stimulus onset for two example neurons, recorded from Monkey 1 (left, V2) and Monkey 2 (right, V1). Peristimulus time histograms were created by averaging spike trains convolved with an exponential filter (tau = 10 ms). Bottom, Mean response dynamics in anticipation of stimulus onset for a population of V1 (green) and V2 (blue) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). Before averaging, responses of each neuron were normalized. Filled bars represent the intervals used to compute the EI. B, Distribution of the EI for a population of V1 (green) and V2 neurons (blue), recorded from Monkey 1 (left) and Monkey 2 (right). C, Mean response computed from trials with high EI and low EI for a population of V1 and V2 neurons recorded from Monkey 1. D, Estimation noise computed from trials with high EI plotted against estimation noise computed from trials with low EI for a population of V1 and V2 neurons. All experiments in which both groups of trials consisted of at least 50 observations are included (85 of 228 experiments).
For Monkey 1, the EI was positive on average, and stronger in V2 than in V1 (Fig. 3B; V1: median EI = 0.35; V2: median EI = 0.69; p = 0.001, Wilcoxon rank sum test). Monkey 2 did not show any systematic anticipatory signals (V1: median EI = 0.02; V2: median EI = 0.0; p = 0.57). Approximately 100 ms after onset of the choice targets, spontaneous activity was abruptly reduced and then returned to baseline over the next 400 ms (Fig. 3A). These dynamics are consistent with the suppressive effects of stimuli placed far outside a neuron's classical receptive field (Bair et al., 2003) and may result from the appearance of the choice targets.
We wondered whether the anticipatory signal carried any information about upcoming stimulus response or task performance. For each experiment, we calculated the EI for every trial and, when possible, divided the trials in two groups (median-split analysis, see Materials and Methods). We found that stimulus-evoked activity did not differ significantly between the low- and high-expectancy groups of trials (Fig. 3C). As can be seen in Figure 3D, behavioral sensitivity also did not depend on the strength of the anticipatory signal (V1: median difference in noise SD = 0.13 degrees, p = 0.90, Wilcoxon signed rank test; V2: median difference = 0.09 degrees, p = 0.61). Likewise, behavioral bias did not depend on expectancy (V1: median difference in decision criterion = −0.21 deg, p = 0.24; V2: median difference = 0.40 degrees, p = 0.38). These results suggest that the anticipatory signal reflects a nonsensory component of cortical activity that is present in some animals, and that is not related to stimulus coding or choice behavior.
Interanimal differences in choice-correlated activity
The two monkeys also differed in the pattern of choice-correlated activity in V1 and V2 neurons. We measured the association between neuronal and behavioral responses under constant stimulus conditions with the conventional choice probability (CP) statistic (Britten et al., 1996), defined as the probability that a neuronal response associated with a choice in favor of the orientation preferred by the neuron is larger than a response associated with a “nonpreferred” choice. A value of 0.5 represents chance performance, and a value of 1 represents a perfect association between neuronal and behavioral responses. Choice probabilities vary widely across neurons, so we focus on the population mean. For Monkey 1, neuronal and behavioral responses were negatively correlated: neural responses were typically reduced for trials in which the animal chose the orientation preferred by that neuron (Fig. 4A,B). CP was on average <0.5 in both V1 and V2 (Fig. 4C; V1: mean = 0.46, p = 0.02; V2: mean = 0.44, p < 0.001). For Monkey 2, there was no systematic choice-correlated activity evident in the responses of V1 or V2 neurons to zero-signal stimuli (i.e., stimuli whose orientation matched the discrimination boundary; Fig. 4C; V1: CP mean = 0.50, p = 0.90, t test; V2: mean = 0.51, p = 0.48).
Analysis of choice signals in V1 and V2. A, Mean response computed from “preferred” and “nonpreferred” trials for an example V1 neuron recorded from Monkey 1 (left), and the distribution of responses to the zero signal stimulus conditioned on behavioral choice (right). B, Mean response computed from “preferred” and “nonpreferred” trials for the subset of V1 and V2 neurons with significant negative CP recorded from Monkey 1. C, Distribution of CP for the zero-signal stimulus condition for a population of V1 (top) and V2 (bottom) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). Filled bars represent cases that were significantly different from 0.5 by the permutation test introduced by Britten et al. (1996). D, Mean CP for the weakest nonzero signal conditions plotted against CP for the zero-signal condition for a population of V1 (top) and V2 (bottom) neurons, recorded from Monkey 1 (left) and Monkey 2 (right).
The negative correlation between Monkey 1's neuronal and behavioral responses was not limited to the ambiguous stimulus condition (Fig. 4A,B). We calculated the mean CP for the neighboring stimulus conditions, provided that they elicited sufficient behavioral variability to yield meaningful estimates (i.e., each response alternative was chosen at least 5 times). For Monkey 1, CP was well correlated across stimulus conditions in V1 and V2 (Fig. 4D; V1: r = 0.61, p < 0.001; V2: r = 0.56, p < 0.001), indicating that this statistic captures a stable association between neuronal and behavioral responses. For Monkey 2, on the other hand, CP was not significantly correlated across conditions (Fig. 4D; V1: r = 0.15, p = 0.17; V2: r = 0.18, p = 0.40).
We wondered whether the negative choice signals found in Monkey 1 could have arisen as a byproduct of a confounding factor. For example, suppose that leftward and rightward choice-saccades were typically preceded by different patterns of small eye movements. Because small eye movements can modify neural activity in the visual cortex (Martinez-Conde et al., 2000), such difference could generate a correlation between neuronal and behavioral responses. To control for such artifacts, we investigated whether some cells were more strongly associated with the decision than others. We first asked whether CP depended on the sign of the tuning curve. If the animal judged the stimulus to be rotated clockwise relative to the discrimination boundary, it always had to make a rightward saccade. Therefore, if choice signals were artifactual, we would expect their sign to depend on the sign of the tuning curve. In Monkey 1, neurons with positively and negatively signed tuning curves both had average choice probabilities that tended to be <0.5 (across all negatively signed tuning curves, mean = 0.47, p = 0.13, n = 43; across all positively signed tuning curves, mean = 0.44, p < 0.001, n = 71). In Monkey 2, neither positively nor negatively signed tuning curves had choice probabilities that differed significantly from 0.5 (across all negatively signed tuning curves, mean = 0.50, p = 0.84, n = 48; across all positively signed tuning curves, mean = 0.50, p = 0.89, n = 66).
We also asked whether more sensitive neurons were more strongly associated with the animal's choice. Such a relationship has been observed in some, but not all, studies on CP. In Monkey 1, CP correlated weakly with neural noise (r = 0.21, p = 0.028, n = 114, Spearman rank order correlation). Thus, the more sensitive the neuron, the more negative the CP. This was true of neurons with either positively or negatively signed tuning curves (Fig. 5A). In Monkey 2, there was no evidence for a relationship between neuronal sensitivity and CP (r = −0.01, p = 0.92, n = 114), and this was consistent across neurons with either positively or negatively signed tuning curves (Fig. 5B). In summary, the systematic nature of Monkey 1's negative choice signals in task-relevant neurons suggests that an artifactual origin is unlikely.
Analysis of the relation between neural sensitivity and choice signals. A, CP for the zero-signal stimulus condition plotted against signed neural sensitivity (1/noise SD) for a population of V1 (green) and V2 (blue) neurons, recorded from Monkey 1. Red lines indicate the best fitting linear relation, calculated independently for the negatively (left half) and positively (right half) signed tuning curves. B, Same for Monkey 2.
Origins of choice-correlated activity
These results seem to present a paradox. Both monkeys perform a challenging perceptual task well. Yet the negatively signed choice signals of Monkey 1 appear to suggest that he disobeys the advice of his most informative sensory neurons. How can this be? The temporal dynamics of neuronal activity provide an important clue. In both monkeys, selectivity for stimulus orientation peaked at response onset and remained high throughout the entire trial (Fig. 6A), consistent with feedforward models of orientation selectivity (Hubel and Wiesel, 1962; Priebe and Ferster, 2008; Goris et al., 2015). In contrast, choice-correlated activity was initially absent from Monkey 1's neuronal responses. It appeared ∼250 ms after stimulus onset and gradually increased throughout the remainder of the trial (Fig. 6B). Presentations of the ambiguous stimulus associated with “preferred” choices generated a neural response that was ∼7 spikes per second less than “nonpreferred” choices by the end of the trial (Fig. 6C).
Analysis of response dynamics of stimulus and choice signals in V1 and V2. A, Temporal evolution of the mean response to three stimulus conditions (strong negative signal, zero-signal, strong positive signal) for a population of V1 (green, top) and V2 (blue, bottom) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). B, Temporal evolution of mean CP for the zero-signal stimulus condition for a population of V1 (green) and V2 (blue) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). CP was computed within a 100-ms-wide sliding window. C, Temporal evolution of the relative response to the zero-signal stimulus conditioned on the animal's choice (“preferred” vs “nonpreferred”) for a population of V1 (top) and V2 (bottom) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). The relative response was computed by subtracting the mean response within both groups of trials from the mean response calculated across all trials.
The late emergence of choice-correlated activity suggests two things. First, that these choice signals are likely not feedforward in origin. And second, that they might reflect the outcome of the decision-making process rather than the encoded evidence supporting the decision. Armed with these insights, we suggest the following interpretation: Both monkeys primarily relied on the early part of the stimulus epoch to judge stimulus orientation. After having committed to a choice (perhaps 250 ms after stimulus onset), the animals adopted different strategies for the remainder of the epoch. Monkey 2 waited patiently until the stimulus disappeared, after which he communicated his choice. Monkey 1 modulated the activity of the sensory neurons involved in measuring stimulus orientation, reducing the activity of the neurons that “won the vote.” This suggests a circuit that can “explain away” predictable observations, a common property of predictive coding theories (Mumford, 1992; Rao and Ballard, 1999; Spratling, 2010). Specifically, these theories conceive of sensory neurons as representing deviations from predictable image structure. In this view, the perceived orientation does not need to be represented by lower-level sensory neurons. Alternatively, one can imagine that the animal shifted attention to the nonchosen orientation, boosting responses of neurons with corresponding selectivity (Cohen and Newsome, 2008).
Several features of the data support the hypothesis (schematized in Fig. 7) that the origin of negative choice-correlated activity in Monkey 1 is top-down. First, as can be seen in the choice-probability traces, choice-correlated activity exhibited a reversed temporal hierarchy: in Monkey 1, it emerged in V2 before appearing in V1 (Fig. 6B). This is consistent with a top-down origin. In Monkey 2, CP hovered ∼0.5 throughout the entire trial.
Model schematic. A, A visual stimulus is encoded by a population of noisy sensory neurons whose responses are transformed into a perceptual decision by means of a deterministic mapping function. Responses of the sensory neurons can be modulated by nonsensory factors, some of which depend on the outcome of the decision-making process. B, Temporal evolution of sensory and nonsensory factors modulating responses of V1 and V2 neurons in Monkey 1.
Second, CP in Monkey 1 is related to anticipatory signals. Many V1 and V2 neurons in Monkey 1 showed an increase in activity in anticipation of stimulus onset that was not induced by visual stimulation (Fig. 3A,B). Presumably, the strength of this effect indicates the strength of a neuron's modulatory inputs. If choice-correlated activity originates from the same inputs, its magnitude might correlate with the strength of the anticipatory signal. We divided Monkey 1's neurons into four groups, based on the value of the EI, and computed the mean response to the ambiguous stimulus, conditioned by choice. As can be seen in Figure 8A, the firing of neurons without anticipatory signals on average did not depend on choice. In contrast, neurons with anticipatory signals did show choice-dependent responses. The stronger the anticipatory activity, the stronger the effect of choice on response magnitude (Fig. 8A). This relationship was also evident at the level of single neurons. In Monkey 1, the EI correlated weakly with CP (Fig. 8B). Neurons with stronger anticipatory signals had a more negative CP (across all neurons: r = −0.20, p = 0.033; V1: r = −0.14, p = 0.26; V2: r = −0.29, p = 0.048). In Monkey 2, no such correlation existed (across all neurons: r = 0.01, p = 0.90; V1: r = −0.02, p = 0.86; V2: r = 0.08, p = 0.71); notably, this monkey had few neurons with large values of EI.
Analysis of the relation between anticipatory and choice signals. A, Temporal evolution of the response to the zero-signal stimulus conditioned on the animal's choice (“preferred” vs “nonpreferred”) for four groups of neurons (groups are based on the EI, which increases from left to right). All neurons were recorded from Monkey 1. B, CP for the zero-signal stimulus condition plotted against EI for a population of V1 (green) and V2 (blue) neurons, recorded from Monkey 1 (left) and Monkey 2 (right). C, CP computed from trials with high EI plotted against CP computed from trials with low EI for a population of V1 (green) and V2 (blue) neurons, recorded from Monkey 1 (left) and Monkey 2 (right).
Modulatory inputs vary in strength across neurons and across trials. If choice-correlated activity originates from the same inputs as anticipatory signals, its magnitude might differ across trials with weak and strong anticipatory signals. For each experiment, we split the trials according to the EI (median-split analysis), and computed CP for both groups of trials, provided there was sufficient behavioral variability to yield meaningful estimates (84 of 228 neurons). In Monkey 1, high expectancy trials were associated with lower CP (Fig. 8C; across all neurons: p = 0.003, t test, n = 33; V1: p = 0.016, n = 15; V2: p = 0.085, n = 18). Thus, on trials in which top-down input was more prominent in anticipation of the stimulus, negatively signed choice-correlated activity was also more prominent. In Monkey 2, no relationship was evident (Fig. 8C; across all neurons: p = 0.83, n = 51; V1: p = 0.70, n = 41; V2: p = 0.54, n = 10).
Discussion
We have explored the relationship between the activity of orientation-selective cells in V1 and V2 and perceptual judgments of orientation. Surprisingly, we found that responses of optimally tuned neurons can be negatively correlated with perceptual decisions when monkeys perform at psychophysical threshold. Current theories consider positively signed choice signals in well-tuned neurons to be a manifestation of the decision-making process, arising either from feedforward propagation of noise (Kanitscheider et al., 2015; Pitkow et al., 2015), or from fed-back decision signals (Wimmer et al., 2015; Haefner et al., 2016). Consistent with this, choice signals have been observed in multiple sensory systems, brain areas, and psychophysical tasks (Vallbo and Johannson, 1976; Dubner et al., 1989; Nienborg et al., 2012). However, not all studies have found choice signals in well-tuned neurons. Disparity-selective V1 neurons, heading-selective otolith afferents, and vibration-selective neurons in the somatosensory thalamus all have good discriminative capabilities, yet their activity is uncorrelated with perceptual judgments (Nienborg and Cumming, 2006; Camarillo et al., 2012; Yu et al., 2015). Even when choice signals are present, they vary greatly in magnitude across animals and tasks (Britten et al., 1996; Dodd et al., 2001). Our data show a more significant discrepancy: one animal's neural responses were uncorrelated with choice, whereas those of another were anticorrelated.
Further comparison between our animals revealed a consistent relationship between perceptual and neuronal orientation discrimination capabilities but inconsistent patterns of anticipatory and choice-correlated activity. In the animal with systematic choice signals, choice-correlated activity emerged late in the trial, exhibited a reverse temporal hierarchy, and was correlated with anticipatory signals. We suggest that both monkeys primarily relied on the early part of the stimulus epoch to judge stimulus orientation. But before stimulus onset and after choice commitment, they used different strategies that differentially affected neuronal activity. For one animal, anticipation increased activity in the visual cortex (compare Sirotin and Das, 2009). Anticipatory signals were stronger in V2 than in V1, were correlated with choice signals, and were not related to stimulus coding or choice behavior. For that same animal, the firing of neurons that preferred the chosen stimulus was lower late in the trial, creating the choice probabilities below chance, and perhaps reflecting a feedback signal that seeks to “explain away” persistent activity. Neither of these signals was evident in the other animal.
If choice signals are not an inherent feature of neuronal activity in visual cortex during perceptual decision making, what other origin might they have? The activity of sensory neurons is modulated by a variety of contextual factors, such as adaptation, attention, and wakefulness (Luck et al., 1997; Kato et al., 2012; Benucci et al., 2013). Likewise, perceptual decisions are influenced by such factors as recent stimulus, choice, and feedback history (Lau and Glimcher, 2005; Fischer and Whitney, 2014; Abrahamyan et al., 2016). These phenomena might conspire to create correlations between neuronal and behavioral responses. Previous trials might not only bias animals' choice behavior but could also affect the allocation of their attention; for example, monkeys might attend more to the stimulus features corresponding to a favored response alternative, which could fluctuate from trial to trial. Because attention selectively changes the gain of sensory neurons, this could create the commonly observed positive correlation between neuronal and behavioral responses. But a different behavioral strategy, in which the animal changes its attentional state after reaching a decision, could allow these same signals to create negative correlations, as seen in one of the animals we tested. Direct evidence for this idea could come in the form of a systematic relation between choice history effects and CP. Our own data, however, do not reveal such a relationship.
We seek to understand the transformation of sensory signals into perceptual experience. Many previous studies have shown that neurons that are optimally tuned for a perceptual task can carry almost as much information about the stimulus as the animals' behavioral reports (Britten et al., 1992; Parker and Newsome, 1998). Our data (for both animals, and in both visual areas) are consistent with this. Many studies have also found that the responses of animals' most sensitive neurons are correlated with choices, and this finding was initially thought to be a straightforward signature of sensory decoding (Britten et al., 1996; Shadlen et al., 1996). More recent work has called this simple framework into question (Nienborg and Cumming, 2009; Nienborg et al., 2012) while still holding out hope that choice correlations arise from processes tied to sensory decision-making (Wimmer et al., 2015; Haefner et al., 2016). Our data are not consistent with either of these interpretations and suggest that many factors previously thought to be extraneous can contribute to choice correlations. While our findings indicate that choice correlations are not invariably associated with perceptual decision-making, the close relationship of the sensitivity of behaving animals and the responses of their neurons survives as a stable feature linking neural activity to perception.
Footnotes
This work was supported by National Institutes of Health Grants EY04440 and EY022428, Howard Hughes Medical Institute, National Science Foundation Graduate Research Fellowship to C.M.Z., and postdoctoral fellowships from the Fund for Scientific Research of Flanders and the Belgian American Educational Foundation to R.L.T.G.
The authors declare no competing financial interests.
- Correspondence should be addressed to Dr. J. Anthony Movshon, Center for Neural Science, New York University, 4 Washington Place, Room 809, New York, NY 10003. movshon{at}nyu.edu
