Abstract
Feature selectivity of visual cortical responses measured during passive fixation provides only a partial view of selectivity because it does not account for the influence of cognitive factors. Here we focus on primate area V4 and ask how neuronal tuning is modulated by task engagement. We investigated whether responses to colored shapes during active shape discrimination are simple, stimulus-agnostic, scaled versions of responses during passive fixation, akin to results from attentional studies. Alternatively, responses could be subject to stimulus-specific scaling, that is, responses to different stimuli are modulated differently, resulting in changes in underlying shape/color selectivity. Among 83 well-isolated V4 neurons in two male macaques, only a minority (16 of 83), which were weakly tuned to both shape and color, displayed responses during fixation and discrimination tasks that could be related by stimulus-agnostic scaling. The majority (67 of 83), which were strongly tuned to shape, color, or both, displayed stimulus-dependent response changes during discrimination. For some of these neurons (39 of 83), the shape or color of the stimulus dictated the magnitude of the change, and for others (28 of 83) it was the combination of stimulus shape and color. Importantly, for neurons with one strong and one weak tuning dimension, stimulus-dependent response changes during discrimination were associated with a relative increase in selectivity along the stronger tuning dimension, without changes in tuning peak. These results reveal that more strongly tuned V4 neurons may also be more flexible in their selectivity, and imbalances in selectivity are amplified during active task contexts.
SIGNIFICANCE STATEMENT Tuning for stimulus features is typically characterized by recording responses during passive fixation, but cognitive factors, including attention, influence responses in visual cortex. To determine how behavioral engagement influences neuronal responses in area V4, we compared responses to colored shapes during passive fixation and active behavior. For a large fraction of neurons, differences in responses between passive fixation and active behavior depended on the identity of the visual stimulus. For a subgroup of strongly feature-selective neurons, this response modulation was associated with enhanced selectivity for one feature at the expense of selectivity for the other. Such flexibility in tuning strength could improve performance in tasks requiring active judgment of stimuli.
Introduction
Tuning for visual object properties is typically investigated by having an animal passively fixate a central location on the screen, and presenting a stimulus in the receptive field (RF) of a neuron. Given the constraints of recording from an awake, behaving animal, and the need for repeated measurements because of response variability, fixation tasks are an efficient way to assess tuning in this manner. Such characterizations have formed the basis of much of our knowledge about the encoding of visual stimuli throughout the ventral pathway, a collection of brain areas that underlie the processing of visual objects. However, it is largely unknown how feature selectivity in such a passive context is related to selectivity in a more active context, where the animal is explicitly required to make a behavioral judgment about some aspect of the stimulus. Factors related to the behavioral task are evident as early as primary visual cortex (V1) (Fang et al., 2008; Chen and Seidemann, 2012; Nurminen et al., 2018) and are even more prevalent beyond V1 (Treue, 2001; Buffalo et al., 2010). Such modulation is thought to enable scene segmentation and performance on visually guided tasks (Poort et al., 2012), but how task-related factors influence the encoding of objects in ventral stream areas is still unknown. Specifically, we do not know whether stimulus selectivity, both in terms of which stimuli are preferred over others and the strength of this preference, is the same in active and passive task contexts.
Primate visual area V4 provides an opportunity to answer this question directly. Responses of V4 neurons reflect many aspects of visual objects, including form, color, brightness, and texture (Pasupathy et al., 2020); area V4 also receives many feedback projections from higher-order areas, and is known to be modulated by cognitive factors (Ogawa and Komatsu, 2006; Ungerleider et al., 2008; Bichot et al., 2019). Indeed, many studies have investigated the top-down influence of attention on V4 responses (e.g., Haenny et al., 1988; Maunsell et al., 1991; Motter, 1993; McAdams and Maunsell, 1999, 2000; Cohen and Maunsell, 2009; Anderson et al., 2013). Commonly, these experiments cue the animal to attend to a spatial location and/or an object feature (e.g., attend to “the blue object” or “the dots moving upward”), and have demonstrated that responses in the “attended” condition can be explained by applying a single multiplicative factor to the responses during the “unattended” condition (McAdams and Maunsell, 1999; Yantis and Serences, 2003; Reynolds and Chelazzi, 2004). Visual search tasks have produced similar results (Ogawa and Komatsu, 2004). Importantly, in these paradigms, the typical comparison is between responses in the attended and unattended conditions within the same active behavioral context; seldom between passive viewing and active behavior. As a result, while it is evident that cognitive factors can influence response magnitude in area V4, it remains unclear whether object representations and stimulus preferences are similar during passive fixation and active behavior. Here, we ask whether task context modulates the selectivity of V4 responses. To address this question, we studied the responses of 83 well-isolated V4 neurons while the animal performed two tasks: a passive task (fixation) and an active task (shape discrimination). We collected responses to the same objects presented in these two tasks, and compared neuronal selectivity for object shape and color. We interpreted our observations in the context of two alternative hypotheses: that responses during different tasks are modulated in a stimulus-agnostic manner, and feature selectivity is maintained; or that responses are modulated in a stimulus-dependent manner, alongside changes in feature selectivity.
Materials and Methods
Surgical methods
We implanted 2 adult male macaque monkeys (Macaca mulatta) with custom headposts and chambers positioned over dorsal area V4 (left hemisphere). The placement of the chamber over the prelunate gyrus and subsequent craniotomy were guided by structural MRI (for details, see Bushnell et al., 2011a). All animal procedures conformed to the National Institutes of Health guidelines and were approved by the Institutional Animal Care and Use Committee at the University of Washington.
Visual stimulus presentation
Stimuli were presented against a uniform gray background (luminance 5.4 cd/m2) using a spectrally calibrated (PR650, PhotoResearch) CRT monitor positioned 45 or 56 cm away (M1 and M2, respectively). The animal fixated a small white dot in the center of the screen within a window of radius 0.75°-1°. Eye position was monitored using an infrared eye tracking system (EyeLink 1000; SR Research) and coordinated with stimulus presentation using custom software based on Pype (Mazer, 2013).
Task design
Fixation task
Each trial of the fixation task (see schematic in Fig. 1A) began with the presentation of a fixation spot. Once the animal acquired fixation, a sequence of 3-5 stimuli were presented with a 200 ms blank interval preceding each stimulus. For the first 12 neurons we recorded, each stimulus was presented for 300 ms; for the rest, this presentation time was shortened to 250 ms to better match the timing in the discrimination task (see below). A total of 9 stimuli (3 shapes × 3 colors) customized to the shape and color preferences of the neuron under study (see Stimulus selection) were used to probe responses during the fixation task. Stimuli were presented in random order, and each was shown 20 times within a block of fixation task trials. The animal was rewarded with drops of juice for successful fixation for the entire duration of a trial (1500-2500 ms).
Discrimination task
Animals were trained on a sequential shape discrimination task (see Fig. 1B). Each trial began with the presentation of a fixation spot. Once the animal acquired fixation, a “reference” stimulus was presented at central fixation for 600 ms. After an interstimulus interval of 200 ms, a “test” stimulus appeared within the RF of the V4 neuron. Simultaneously, two small target dots appeared to the left and right of the fixation spot, each 6° of visual angle away from fixation. The animal then had up to 1500 ms to report whether the shape of the test stimulus was the same or different from the reference stimulus, regardless of the color of either stimulus. Animals responded via a saccadic eye-movement to the right or left target dot for “same” and “different,” respectively. The test stimulus disappeared as soon as the animal's eye left the fixation window; test stimulus duration varied from trial to trial, dictated by the reaction time of the animal. Mean reaction times, measured as the time at which the eye exited the fixation window, were 214 ± 56 ms (mean ± SD) across both animals (M1: 227 ± 45 ms; M2: 206 ± 60 ms). The same 9 stimuli used during the fixation task served as the reference and test stimuli in the discrimination task. Each of the 9 stimuli appeared randomly as a test stimulus for a total of 15-30 repeated presentations in each block of discrimination trials. The animal was rewarded with drops of juice for a saccade to the correct target location; the number of trials requiring a leftward or rightward saccade were balanced and chance performance was 50%; mean animal performance was 83 ± 5% (M1: 85 ± 7%; M2: 81 ± 3%). In our analyses below, we include only those trials terminating in a correct saccade (animals made few errors each day on the discrimination task and the neuronal responses on error trials were typically weak relative to both correct discrimination trials and fixation trials).
Data collection
Each day, we lowered a single tungsten microelectrode (FHC) using a stepper motor drive (Gray Matter Research). We amplified and filtered the signals from the electrode and sorted waveforms (Plexon Systems) to identify single units. Here, we include responses from 83 well-isolated neurons (31 from M1; 52 from M2) with RFs in the lower right visual field with eccentricities from 1.6° to 7.3°.
Stimulus selection
For each neuron in our dataset, we first assessed the spatial extent of the RF manually using a variety of shapes and colors. We then characterized each neuron's color/luminance preference either manually (6 neurons) or via an automated protocol that measured neuronal responses to a single shape presented in 25 colors that uniformly sampled the CIE space, and at 3 or 4 luminance levels (see Bushnell et al., 2011b). To assess shape selectivity, we presented 25 2D shapes (a subset of the shapes used by Pasupathy and Connor, 2001) at 8 rotations, for a total of 200 shape stimuli.
Given evidence that shape selectivity in V4 is independent of stimulus color (Bushnell and Pasupathy, 2012), we designed the stimuli to consider the influence of task on tuning separately for shape and for color. Based on the color/luminance and shape preferences of each neuron, we chose three shapes and three colors that represented the preferred, intermediate, and nonpreferred values along each of the two feature dimensions for each neuron. We then created a set of 9 stimuli by rendering each of the selected shapes in each of the selected colors (for an example set from one session, see Fig. 1C). When responses of more than one neuron were recorded simultaneously (14 sessions in Animal 2), stimuli were tailored according to the preferences of only one of the neurons.
The position of the test stimuli during the discrimination task and all stimuli during fixation were jittered by up to 5 pixels (0.14° for M1; 0.12° for M2) about the center of the neuron's RF. Stimuli were scaled such that all parts of the stimuli were within 80% of the estimated RF diameter based on data in Gattass et al. (1988); reference stimuli in the discrimination task were size-matched to the RF-based scaling of the test stimuli.
For each neuron, fixation and discrimination tasks were conducted in alternating blocks starting with the fixation task. Each fixation block was 45-60 trials long, and each discrimination block was 144-288 trials long; as described above, 3-5 stimuli were presented per fixation trial, while one stimulus was presented in the RF per discrimination trial. The characteristics of the fixation spot identified the task: a white cross indicated a fixation task while a blue square indicated a discrimination task. Here we include responses from neurons where we successfully captured at least 2 blocks of both tasks (median: three blocks for each task).
Analyzing neuronal responses
Stimulus onset and offset times were detected with a photodiode. To construct peristimulus time histograms (PSTHs) for each neuron, responses were aligned to stimulus onset and 1 ms bin responses were smoothed with a Gaussian kernel (σ = 10 ms). To obtain average responses for each stimulus, we computed the mean spiking rate on each trial in the time window of 50-200 ms from stimulus onset. This time window was chosen to account for V4 response latency (Zamarashkina et al., 2020) and to mitigate the influence of reward anticipation or saccade preparation (in the discrimination task, average time to saccade initiation was 214 ± 56 ms). Additionally, we replicated the analyses using a shorter window of 50-125 ms from stimulus onset to determine whether our results are distinct from the observations of Ipata et al. (2012), who reported that effects of feature attention during visual search affect V4 responses ∼50 ms after response onset (Ipata et al., 2012). Since median response latency of V4 neurons is ∼75 ms (Zamarashkina et al., 2020), this earlier windows ends before the time point at which attentional effects might be expected to begin based on Ipata et al. (2012). For the fixation task trials, multiple stimuli were shown in a sequence with a short interstimulus interval (200 ms), raising the possibility that the response might adapt over time. We compared stimulus responses based on order in the trial sequence, and found that responses to the first stimulus in a sequence were slightly higher than responses to the fourth stimulus (mean difference ± SEM: 1.23 ± 0.37 spk/s). However, this small difference was not statistically significant (p = 0.71; two-sample t test).
Regression models
To compare neuronal responses to the 9 stimuli during fixation and discrimination, we used individual trial responses to fit six regression models. Since the focus of the present study is the difference in responses between the fixation and discrimination task conditions, we treat responses in fixation trials as the independent variable in each model.
Stimulus-agnostic models
First, we considered the possibility that neuronal responses during the discrimination task scaled linearly relative to the responses during the fixation task, as reported in attentional studies; such a response change could be described using the linear-gain model as follows:
To perform the regression, we paired trials from fixation and discrimination blocks (Rfix and Rbeh). Since there were fewer discrimination trials, we downsampled fixation responses (with replacement) for each of the 9 stimuli (Rfix(s,c)) to match the number of trials during the discrimination task for that stimulus (Rbeh(s,c)). We then randomly paired fixation and discrimination responses to perform the regression.
We next considered a model where the relationship between responses in fixation and discrimination tasks is nonlinear (polynomial-gain model):
Stimulus-dependent models
As an alternative to the stimulus-agnostic models, we considered a case where the difference between neuronal responses during the discrimination and fixation tasks is related to the features of the stimulus in the RF. We used two simple stimulus-dependent models that depend on either the shape or color of the stimulus, as follows:
Where k(s) and k(c) represent separate coefficients for each of three stimulus shapes or colors, respectively, and k0 is the regression constant.
We also considered more complex stimulus-dependent models. The third model included both shape and color feature gains, for a total of 7 parameters:
The final model included separate coefficients for each of the 9 stimuli, for a total of 10 parameters:
Model performance and comparison
To mitigate overfitting, we calculated root-mean-square error (RMSE) for each model using sixfold cross-validation. Briefly, the data were split into 6 equal parts, with 5 parts used as the training set and 1 part used as the testing set. The fitting procedure was repeated with a different part serving as the testing set. We report testing RMSE averaged over the 6 folds. Given that the models had unequal numbers of parameters, we also computed the average Bayesian Information Criterion (BIC). For each neuron, we randomly paired all fixation and discrimination trials 50 times, for a total of 50 model fits using all the data. We considered the regression model with the lowest average BIC to be the best-fitting model for each neuron.
To obtain predicted responses during discrimination using each regression model, we averaged each model's coefficients over the 50 model fits. We then used the averaged coefficients together with the average responses to 9 unique stimuli in the fixation condition to generate predicted responses during the discrimination task from Equations 1–3. NB: the repeated random pairing and subsequent averaging of coefficients ultimately results in a deviation from an intuitive “perfect” fit even for models with many parameters.
Feature selectivity analysis
For each neuron, we estimated the strength of feature selectivity for shape and color by calculating the partitioned variance associated with each feature from a two-way ANOVA model with shape, color, and interaction terms. The selectivity metric, η2, was calculated as the partitioned sum of squares for each factor (shape, for η2shape; or color, for η2color) divided by the total sum of squares. The interaction term was small for all neurons; on average, it accounted for 4% of the relative variance. To qualitatively describe the strength of selectivity, we use the term “strong” to refer to η2 values ≥ 0.26, “moderate” to refer to η2 values ≥ 0.13, and “weak” to refer to η2 values < 0.13.
To compare the shape and color selectivity of individual neurons, we computed a ratio of η2shape to η2color. This relative feature selectivity metric is >1 for neurons that are more shape- than color-selective, and <1 for neurons that are more color- than shape-selective.
Contribution of other variables to responses in the discrimination task
We considered two other factors that can influence responses for a discrimination task. The first factor was the influence of the reference stimulus; that is, if a reference stimulus matches the preference of a neuron, the response to a test stimulus may be modulated (Treue and Martínez-Trujillo, 1999). To determine whether the identity of the reference stimulus contributes to the responses during the active discrimination task, we first separated neuronal responses to the test stimulus into two sets according to whether the reference stimulus matched or did not match the neuronal preference for shape and color. One set of data contained responses to test stimuli in trials where the reference stimulus was of the preferred shape in any color, or preferred color in any shape (reference-preferred). The other set contained responses to test stimuli in trials where the reference stimulus was any of the other remaining shape–color combinations (reference-other).
We then examined the magnitude of the reference stimulus-dependent effect, measured as the difference between responses on reference-preferred and reference-other trials, relative to the average response change during behavior. This metric corresponds to the following ratio:
The second factor we considered was whether the specific motor plan being prepared by the animal (a saccadic eye movement to the left or right target, only in the discrimination task) contributed to the difference between responses in fixation and discrimination tasks. We performed an equivalent analysis by separating responses during the discrimination task into two sets according to the saccade outcome of the task, and comparing the magnitude of the saccade-related effect using an equivalent ratio to Equation 4a as follows:
Statistical analysis
For all statistical tests presented here, we rejected the null hypothesis at the 5% significance level (α = 0.05) unless otherwise noted in the text. For all correlation values reported, we computed the Pearson correlation coefficient; associated p values test for the hypothesis that variables are not correlated. We report any additional details in Results and corresponding figures.
Results
To examine whether feature selectivity in area V4 changes depending on the task being performed, we studied the responses of 83 well-isolated V4 neurons (31 from M1, 52 from M2) to colored 2-D shape stimuli presented in two task conditions: a passive fixation task and an active shape discrimination task. During the fixation task (Fig. 1A), a sequence of stimuli was presented within the RF of the V4 neuron while the animal fixated on a small white cross in the center of the screen. During the discrimination task (Fig. 1B), the animal judged whether two sequentially presented stimuli had the same or different shape, and reported the decision with a rightward or leftward saccadic eye movement, respectively. Animals performed these tasks in repeated blocks of trials with distinct fixation targets (see Materials and Methods; Fig. 1A), alternating between fixation and discrimination conditions.
Task conditions and stimulus design. A, Fixation task. The fixation target was a white cross. Animals were required to fixate within a 0.75°-1° window as a sequence of stimuli were presented within the V4 neuron's RF (dotted circle, shown here for illustration only), separated by a 200 ms interstimulus interval. The animal was rewarded with drops of juice for continuously fixating a sequence containing 3-5 stimuli. B, Sequential shape discrimination task. Each trial began with a blue fixation square, followed by the presentation of a reference stimulus at the center of the screen (600 ms), a blank interstimulus interval (200 ms), a test stimulus within the V4 neuron's RF, and two saccade choice target dots. Correct saccades, rightward for match or leftward for nonmatch, were rewarded with drops of juice. In the example trial shown here, the two stimuli had the same shape so the animal would be rewarded for a saccade to the rightward target. C, Example stimuli from one session. We chose 3 shapes that spanned the dynamic range for each neuron based on preliminary screening; that is, we chose a shape that elicited a strong response, a weak response, and an intermediate level response. We used the same process to choose 3 colors (differing in chromaticity and luminance), and created a set of 9 stimuli with unique shape–color pairings that were then presented in the task contexts schematized in A and B.
For each neuron, we selected 9 stimuli that elicited responses spanning the dynamic range of the neuron's firing rate, as determined by preliminary characterizations. The stimuli were constructed from all possible combinations of three 2-D shapes and three different colors (for details, see Materials and Methods; sample stimuli from one session are shown in Fig. 1C).
We asked whether stimulus tuning was maintained in different task conditions by comparing each neuron's responses to the same stimuli during the fixation and discrimination tasks. We considered two specific alternatives for how neuronal responses were different between fixation and discrimination tasks: that the difference in responses between the two tasks could be described in terms of response scaling (i.e., agnostic of stimulus identity), or that the response differences were dependent on stimulus identity (e.g., shape or color of the stimulus).
These two alternatives are illustrated using simulated responses in Figure 2. When responses during fixation and discrimination trials can be related by simple scaling (Fig. 2A,B), we would expect the rank-ordered stimulus preference to be maintained across tasks (Fig. 2A). We would also expect feature selectivity, measured in terms of the fraction of explained variance in an ANOVA, to be maintained. By extension, the relative balance of such a neuron's selectivity for shape and color will be maintained as well. Figure 2B shows a relative feature selectivity metric computed as the ratio of shape selectivity to color selectivity based on responses during fixation (x axis) and discrimination tasks (y axis). Neurons that exhibit simple response scaling during discrimination (compared with fixation) fall along the diagonal in this space. This is true regardless of whether neuronal responses are more shape- than color-selective (relative feature selectivity >1; for details, see Materials and Methods), or more color-selective (relative feature selectivity < 1, as in the example in Fig. 2B).
Simulated comparison of alternatives relating responses during fixation trials to responses during discrimination trials. A, C, E, Simulated responses are shown on the ordinate. Black represents responses during fixation. Red represents responses during discrimination. Unlike stimuli varying along a single dimension (e.g., contrast or orientation), the shape–color combination stimuli in this study do not lie on a naturally ordered axis; thus, we arrange stimuli along the abscissa according to rank-ordered preference based on the magnitude of responses during fixation. Responses during discrimination are related to fixation by a stimulus-agnostic scale factor in A in contrast to stimulus-dependent modulation in C and E. B, D, F, Relative strength of shape and color feature selectivity during fixation (abscissa) and discrimination (ordinate) for the two alternatives. Blue shaded region represents stronger relative shape selectivity during discrimination. Yellow-shaded region represents stronger relative color selectivity during discrimination. The simulated stimulus-agnostic neuron (B) maintains the ratio of feature selectivity across task contexts, while the simulated stimulus-dependent neurons (D,F) have stronger relative shape selectivity during discrimination.
The second alternative is that responses during discrimination may be modulated by different amounts for different stimuli, more for some and less for others, compared with passive fixation. Figure 2C shows a simulated neuron example for the simple case where the modulation is dependent on stimulus shape. The tadpole-shaped stimulus is associated with stronger response modulation compared with the two other shapes. As a result, stimuli that evoke similar responses during fixation evoke disparate responses during the discrimination task and the rank-ordered preference changes. For such neurons, evoked responses during the two tasks cannot be related by a simple stimulus-agnostic scale factor and will instead require one or more stimulus-dependent modulation factors. Importantly, such stimulus-dependent response modulation will produce a change in feature selectivity. For the example in Figure 2C, this modulation produces responses with increased shape selectivity, and decreased color selectivity. As a result, these simulated responses show a difference in relative feature selectivity during discrimination. Neurons that become more shape-selective during discrimination will occupy the light blue region of the relative selectivity space (as for the example in Fig. 2D), while neurons that become more color-selective during discrimination will occupy the light yellow portion of the plot. A more complex stimulus-dependent case might have modulation that is dependent on both stimulus shape and stimulus color (Fig. 2E,F). Accordingly, such modulations will also produce a change in feature selectivity. However, in this complex case, it is possible for the relative feature selectivity to remain similar if the changes in shape and color selectivity are in concert (both increasing or decreasing) and proportional. These simulated examples provide the intuition for our analysis of neuronal selectivity and task-related modulations in the recorded population.
Below, we present neuronal responses during fixation and discrimination tasks, examine whether they are consistent with stimulus-agnostic or stimulus-dependent modulation, and assess the absolute and relative feature selectivity to understand how stimulus representations may change during different tasks.
Task-driven changes in stimulus preference
Many V4 neurons in our dataset exhibited differential responses during the fixation and discrimination tasks; three examples are shown below (Figs. 3–5).
Responses of example Neuron 1 during fixation and discrimination tasks. A, PSTH for responses to 9 stimuli while the animal performed a fixation task (black traces) or discrimination task (red traces). Insets, the corresponding stimulus for each panel. Panels are arranged such that stimuli sharing the same shape are in rows, and the same color are in columns. Responses were more color- than shape-selective during the fixation task (qualitatively, compare responses to stimuli in the top row or down the left column; for quantification, see Table 2). During the discrimination task, response magnitude decreased, but responses remained shape- and color-selective. Gray boxes represent the time window for response averaging (see below). B, Stimulus tuning for Neuron 1. Each point represents the average response to the stimulus depicted along the abscissa (50-200 ms time window, see gray window in A). Abscissa is ordered according to stimulus preference, that is, response magnitude during the fixation task (black trace). Upper and lower bounds are SEM. Note the similar shape of the tuning curve for the two task conditions.
The first example neuron displayed weaker responses during the discrimination task, compared with the fixation task, but the rank-ordered stimulus preference remained the same (Neuron 1, Fig. 3). During the fixation task (black traces), responses of this neuron were less shape- than color-selective, that is, modulated by stimulus shape (compare PSTH for stimuli in left column) less strongly than by stimulus color (compare PSTHs for stimuli in top row). During the shape discrimination task (red traces), firing rate was suppressed relative to responses during fixation (compare red PSTH traces to black PSTH traces). However, neuronal responses retained similar shape and color preferences: for example, the stimulus in Figure 3A (top left) elicited the strongest responses from this neuron during both tasks.
We examined the responses of this neuron further by constructing a rank-ordered tuning curve as in the simulations in Figure 2 above. The abscissa of the tuning curves in Figure 3B orders stimuli from most to least preferred based on the magnitude of responses during the fixation task (Fig. 3A, gray boxes in 50-200 ms from stimulus onset). Responses to stimuli during the discrimination task were lower than during the fixation task (NB: response suppression was common in our population; we found that 31 of 83 neurons (37%) showed suppressed responses during the discrimination task relative to the fixation task; for more examples, see Maunsell et al., 1991). Despite the change in response magnitude, the rank-ordered tuning was similar. This pattern of responses suggests that this neuron could be explained on the basis of stimulus-agnostic response scaling; we will assess this in detail later.
In other neurons, we observed that rank-ordered stimulus preference during the fixation and discrimination tasks was different. One such example (Neuron 2) is illustrated in Figure 4A. This neuron's responses during the fixation task were more shape- (compare PSTHs for stimuli in left column) than color-selective (compare PSTHs for stimuli in the top row). During the discrimination task, responses of Neuron 2 increased (red traces), but the changes in firing rate were not uniform across stimuli. That is, responses to some stimuli (e.g., top right) increased more than others (e.g., top left). This is evident from the rank-ordered stimulus tuning curve (Fig. 4B), which demonstrates a nonuniform modulation in responses during the discrimination task. Specifically, the “tadpole”-shaped preferred stimulus (Fig. 4A, top row) in brown and cyan evoked stronger responses during discrimination than would be expected from a multiplicative scaling of the responses during fixation (vertical lines highlight these responses in Fig. 4B). This pattern suggests that the responses of this neuron could be explained on the basis of a stimulus-dependent factor (i.e., shape identity) that drives the modulation of responses during discrimination.
Responses of example Neuron 2 during fixation and discrimination tasks. A, PSTH for responses while the animal performed fixation and discrimination tasks; layout and color scheme as in Figure 3A. Responses of Neuron 2 were more shape- than color-selective during the fixation task (qualitatively, compare responses to stimuli in left column and stimuli in top row, respectively; for quantification, see Table 2). During the discrimination task, responses to all stimuli with the same shape (in rows) became more similar in magnitude across different colors. Traces are averaged across all blocks of the same task condition; upper and lower bounds represent SEM. B, Stimulus tuning for Neuron 2, computed as in Figure 3B. Gray vertical lines indicate responses to stimuli in the top row of A. C, Shape- and color-specific PSTHs for responses during individual blocks of fixation and discrimination trials. Response differences between the two tasks are robust across repeated blocks, and consistent with responses averaged across blocks as shown in A.
Since responses in fixation and discrimination trials were collected in blocks, it is possible that some of the observed differences are because of drift in neuronal response over time. To mitigate this factor, we conducted at least 2 blocks of each task type. Here, we compare the modulations across blocks for Neuron 2 as a typical example of changes over time. Figure 4C shows the shape- and color-selective responses of Neuron 2 during individual blocks of trials, alternating between fixation and discrimination tasks (arranged chronologically, left to right). Responses to shapes (averaged across colors, left half of panel) increased consistently during discrimination task blocks, with responses for the tadpole-shaped stimulus increasing more than responses to the other two stimulus shapes. Responses to colors (averaged across shapes, right half of panel) became more similar during each block of discrimination task trials. This pattern of responses suggests that there was a change in feature selectivity during discrimination: the responses become relatively more shape-selective than color-selective.
Figure 5 presents the responses of Neuron 3, another example where rank-ordered stimulus preference was not maintained across the two tasks. This neuron's responses during fixation trials (Fig. 5A, black traces) were modulated less strongly by stimulus shape than by stimulus color (compare PSTHs for stimuli in left column and top row, respectively). The stimulus shown in the top left panel elicited the strongest response from this neuron in both task conditions. While firing rate increased for some stimuli during the discrimination task, it did not for others (Fig. 5B). During discrimination trials, red-colored stimuli did not elicit the same increase in firing rate as the other stimuli, an increase that would have been expected given the ranked stimulus preference during fixation trials. The responses of Neuron 3 are more consistent with the second alternative: that a stimulus-specific modulation drives responses during behavior.
Responses of example Neuron 3 during fixation and discrimination tasks. A, PSTH for responses of Neuron 3 while the animal performed fixation and discrimination tasks; layout and color scheme as in Figure 3A. Responses of Neuron 3 were more color- than shape-selective during the fixation task (qualitatively, compare responses to stimuli in top row and stimuli in left column, respectively; for quantification, see Table 2). B, Stimulus tuning for Neuron 3, computed as in Figure 3B. Gray vertical lines indicate responses to stimuli in the middle column of A.
Modeling response differences between fixation and discrimination tasks
To quantify the differential modulation of responses to the same stimuli during fixation and discrimination tasks, we compared the 6 candidate regression models (Eqs. 1–3) to assess which factors could account for the change in responses observed in individual neurons. We considered two models based on the first alternative that the magnitude of responses during fixation and discrimination trials can be related by stimulus-independent scaling factors. These stimulus-agnostic models included a linear model (Eq. 1) and a second degree polynomial model (Eq. 2). Here, the magnitude of responses may change as a function of task, but rank-ordered stimulus preference is maintained. This type of modulation is consistent with the observed responses for Neuron 1.
We also considered several models based on the second alternative that response modulations were stimulus-specific. These stimulus-dependent models included separate gain factors for: (1) each stimulus shape (Eq. 3a); (2) each stimulus color (Eq. 3b); (3) each stimulus shape and each stimulus color (Eq. 3c); and (4) each of the 9 unique shape–color combinations in the stimulus set (Eq. 3d). In the predictions of these models, both response magnitude and ranked stimulus preference could change during the discrimination trials. This type of modulation is consistent with observed responses for Neurons 2 and 3.
To fit regression coefficients for each of the models, we used individual trial responses to each of the 9 stimuli, and performed the regression using randomly paired fixation and discrimination trials across the recording session (for details, see Materials and Methods). Since our goal was to understand the difference in responses during discrimination trials relative to responses in fixation trials, we used the latter as the independent variable for the regression. Below, we first compare the performance of the 6 candidate models for the example neurons, and then, for the entire population of 83 neurons.
Figure 6 shows a comparison of the observed (open red symbols) and predicted (filled red symbols: averaged fit) responses during discrimination trials for example Neurons 1-3 (observed: same data as in Figs. 3–5). Since individual trials were used to fit the models, when averaged fit coefficients are used to produce the predictions in Figure 6 (see Materials and Methods), none of the models shows a prediction perfectly overlapping with the discrimination trials. Predictions using one set of coefficients from a model fitting iteration with the lowest RMSE (filled cyan symbols) are shown for illustration, but for overall model performance comparisons below, we emphasize the range of RMSE values from the averaged fits. For Neuron 1, the general trend in the observed responses during discrimination trials was reasonably well captured by the linear model, which had the lowest BIC value (Fig. 6A; RMSE: 4.8, BIC: 1048). The other models produced somewhat better qualitative predictions, but because of more parameters, they did not outperform the linear model as assessed by BIC (Figs. 6B–F; RMSE range: 4.7-4.8, BIC range: 1048-1076). Overall, the linear model provided the simplest and most parsimonious explanation of this neuron's response change during discrimination trials.
Comparison of predicted model responses for example neurons. A–F, Observed (red open symbols) and predicted responses (red filled symbols: averaged fit; cyan filled symbols: best fit) for the discrimination task based on regression models fit using fixation task responses (black symbols) of Neuron 1 as the independent variable; discrimination and fixation responses are same data as Figure 3B. The stimulus-agnostic linear model (A) provided the most parsimonious fit among the models. G–L, Observed and predicted responses for the discrimination task based on regression models fit using fixation task responses of Neuron 2; observed data (open symbols) as Figure 4B. The model in I has separate scaling coefficients for stimuli with different shapes, and its prediction matches the observed data best, with fewest parameters. The complex stimulus-dependent models generate a similar prediction with coefficients that are weighted toward contribution of stimulus shape. M–R, Observed and predicted responses for the discrimination task based on regression models fit using fixation task responses of Neuron 3; observed data as Figure 5B. The model in P has separate scaling coefficients for stimuli with different colors, and its prediction matches the observed data best, with fewest parameters. The complex stimulus-dependent models generate a similar prediction with coefficients that are weighted toward contribution of stimulus color.
For Neuron 2, the prediction of the shape-dependent model and the two complex stimulus-dependent models were similar to each other (Fig. 6I,K,L; RMSE range: 18.5-18.9) and better than the predictions of the other models (Fig. 6G,H,J; RMSE = 20.8). The simple shape-dependent model produced the lowest BIC value (BIC = 2220; range: 2220-2285). Looking closely at the regression coefficients of the more complex models, they weighted stimulus shape more strongly than stimulus color. Overall, this is consistent with the idea the change in responses during discrimination was dictated primarily by stimulus shape. Thus, the simple shape-dependent model provided the simplest explanation for the response modulation displayed by Neuron 2.
As a complementary case, for Neuron 3, the prediction of the color-dependent model and the two complex stimulus-dependent models were similar to each other (Fig. 6P,Q,R; RMSE range: 10.4-10.6), and better than the predictions of the other models (Fig. 6M–O; RMSE range: 11.4-11.5). The simple color-dependent model produced the lowest BIC value (BIC = 1747; range: 1747-1807). Again, looking closely at the regression coefficients of the more complex models, they weighted stimulus color more strongly than stimulus shape. This observation is consistent with the idea that stimulus color dictated the change in responses during discrimination; thus, the simple color-dependent model provided the best explanation for the response modulation displayed by Neuron 3.
These examples are generally representative for our recorded population of 83 neurons. Table 1 lists the proportion of best-fitting models for the population. For 16 of 83 (19.3%) neurons, the responses were best described by the stimulus-agnostic models consistent with Alternative 1: that responses during discrimination could be expressed as a scaled version of the responses during the fixation task. For the remaining 67 of 83 (80.7%) neurons, responses were best described by the stimulus-dependent models, consistent with Alternative 2: that response scaling during discrimination was dictated by stimulus identity. Among those neurons, responses of 39 of 83 (47.0%) were best captured by the simple stimulus-dependent models, and responses of 28 of 83 (33.7%) were best captured by the complex stimulus-dependent models.
Number and percentage of neurons best fit by each of the 6 candidate models, shown for the population as well as separately for each animal (M1 and M2)a
To rigorously consider the possibility that a greater proportion of neurons were best fit by stimulus-dependent models simply because we considered a larger number of stimulus-dependent than stimulus-agnostic models (4 vs 2), we restricted the choice of best model to the two stimulus-agnostic models and one of the stimulus-dependent models (the complex model combining shape and color). Even in this case, we found that 26 of 82 neurons (31%) were best fit by either of the stimulus-agnostic models, while 57 of 83 neurons (69%) were best fit by the sole stimulus-dependent model. Thus, the three broad groups of models [stimulus-agnostic, simple stimulus-dependent (color or shape), and complex stimulus-dependent (color+shape or unique stimulus)] cover the range of task-dependent modulations we observed. Below, we probe which neuronal characteristics are associated with these putatively distinct response profiles.
Correlation between task-dependent modulations and the strength of color/shape selectivity
Next, we asked whether there was a consistent relationship between the strength of a neuron's color and shape preferences and the type of response modulation observed during the discrimination task. For example, neurons that are less feature-selective may exhibit changes in responses that are most consistent with stimulus-agnostic models. Neurons that are strongly tuned for a specific stimulus feature (e.g., shape) may exhibit stimulus-dependent modulations, and these modulations may be along the more (or less) strongly tuned stimulus dimension. Thus, we asked whether the absolute or relative strengths of shape and color selectivity were different between neurons best fit by the stimulus-agnostic and stimulus-dependent models.
Absolute strength of feature selectivity
To assess the strength of shape or color selectivity, we used each neuron's responses to the 9 unique stimuli during fixation trials to calculate a selectivity metric, η2, for the influence of shape or color. We computed the η2 using a two-way ANOVA, as the partitioned sum of squares associated with stimulus shape (η2shape) or stimulus color (η2color), normalized by the total sum of squares. A higher value for η2shape corresponds to more of the response variance captured by shape, and thus stronger shape selectivity; and the converse for η2color.
Across the recorded population, most neurons were significantly feature-selective during fixation trials: 75 of 83 were significantly selective for shape, and 80 of 83 were significantly selective for color (p value associated with η2 < 0.05). Table 2 shows the η2 values for neurons best fit by each of the models (grouping the stimulus-agnostic models together, since responses of only 2 neurons were best explained by the polynomial model).
η2shape and η2color for neurons best fit by each of the candidate models (mean ± SEM) and separately for the three example neurons
We did observe a relationship between the mean strength of feature selectivity and the best-fitting model for each neuron. Across the group of neurons whose responses were best explained by the stimulus-agnostic models, responses tended to show weak selectivity for both shape and color, as quantified by the η2 metric (though we note that Neuron 1 is an outlier in this respect, showing weak shape selectivity but strong color selectivity). In contrast, responses of neurons best explained by the simple stimulus-dependent models tended to show strong selectivity for the corresponding feature. Neurons best explained by the shape-dependent model showed strong shape selectivity, but weak color selectivity (color selectivity not different from the stimulus-agnostic group; p = 0.227, Benjamini-Hochberg corrected with false discovery rate = 5%). The converse was true for neurons best explained by the color-dependent model: their responses showed strong color selectivity, but weak shape selectivity (shape selectivity not different from the stimulus-agnostic group; p = 0.756, Benjamini-Hochberg corrected with false discovery rate = 5%). Responses of neurons best explained by the complex stimulus-dependent models showed a range of selectivity strength, from moderate (shape+color model) to strong (unique stimulus model) selectivity for both shape and color. Overall, these results suggest that there is a relationship between the strength of selectivity for a given feature, and the influence of that stimulus feature on the difference in responses between fixation and discrimination tasks. That is, stimulus-agnostic models tend to best explain the response differences for neurons weakly tuned for both shape and color, but not for neurons more strongly tuned for one or both features. The simple stimulus-dependent models tend to best explain response differences for neurons strongly tuned for one feature and weakly for the other, and the complex stimulus-dependent models tend to best explain responses of neurons with strong tuning for both shape and color.
To more directly test whether stimulus-dependent modulations were related to selectivity strength, we performed an additional analysis. If responses during fixation trials are plotted against responses during discrimination trials, the line of best fit represents a stimulus agnostic prediction (the slope and intercept capture multiplicative and additive gain scaling). For each neuron, we calculated the root mean square deviation of responses during discrimination trials from the best fit line, and correlated this deviation metric with the selectivity metric for shape and color (η2) during fixation trials. The correlation was 0.23 for shape (p = 0.0377), and 0.11 for color (p = 0.3328). This analysis suggests that stronger tuning for shape tends to be associated with deviations from the prediction of the stimulus-agnostic model (i.e., stimulus-dependent changes in the response).
We performed the same assessment of feature selectivity strength during discrimination trials for each neuron, and found very similar results as for responses during fixation trials. For the population as a whole, there were no significant differences in the selectivity metric values computed during fixation trials and during discrimination trials. However, each neuron's responses can be relatively more selective for one feature and this imbalance is especially prominent in the shape-dependent and color-dependent neurons (Table 2, second and third row). Therefore, we next assessed relative feature selectivity changes in detail, and separately for the different subgroups of neurons.
Comparing relative strength of feature selectivity during fixation and behavior
As outlined in Figure 2A, B, stimulus-agnostic models predict no change in the strength of selectivity between fixation and discrimination trials. Stimulus-dependent models, on the other hand, can predict changes in absolute selectivity for one or both feature dimensions (see, e.g., Fig. 2C,D). When feature selectivity changes for just one dimension (e.g., shape selectivity increases) or if the feature selectivity for the two dimensions changes in opposite directions (e.g., shape selectivity increases but color selectivity decreases), then we would observe a change in relative selectivity (shape vs color) during discrimination relative to fixation. If the strength of selectivity for shape and color changes in concert between fixation and discrimination, then no change in relative selectivity would be expected. Given the large number of neurons that appeared to be best described by stimulus-dependent models, we wondered whether the neuronal data showed any systematic changes in relative feature selectivity during discrimination. For example, neurons may become more shape-selective during the shape discrimination task regardless of whether they are more shape- or color-selective during fixation. To address this question, we compared the ratio between η2shape and η2color for responses during the discrimination (Fig. 7, ordinate) and the fixation task (abscissa). As in Figure 2, values near 1 correspond to neurons with similar strength of shape and color selectivity while values >1 and <1 correspond to neurons with relatively higher shape selectivity and higher color selectivity, respectively.
Relationship between relative feature selectivity and best-fitting model. Relative feature selectivity, calculated as the ratio between partitioned variance associated with shape and color from two-way ANOVA (see Materials and Methods), is shown for discrimination trials (ordinate) compared with fixation trials (abscissa). Example Neurons 1-3 are marked. Relative selectivity of >1 indicates that responses were more shape- than color-selective; relative selectivity of <1 indicates that responses were more color- than shape-selective. A, Relative feature selectivity for neurons whose responses were well fit by the stimulus-agnostic models. B, Relative feature selectivity for neurons whose responses were well fit by the complex feature-dependent models. Black represents 6-parameter shape+color additive model. Red represents 9-parameter unique stimulus model. C, Relative feature selectivity for neurons whose responses were well fit by the shape- or color-dependent models. Many neurons well described by the shape-dependent model (black) became even more shape-selective during discrimination trials, although the difference between the groups failed to reach statistical significance (paired t test, p = 0.13). Likewise, many neurons well described by the color-dependent model (red) became more color-selective during discrimination trials, but the difference across the population failed to reach statistical significance (paired t test, p = 0.14). D, Modulatory ratio for neurons in C, comparing the changes in the feature that explains the stimulus-dependent response modulation during the discrimination task relative to the second feature. Thus, the modulation ratio measures shape selectivity relative to color selectivity for neurons best fit by the shape-dependent model, and color selectivity relative to shape selectivity for neurons best fit by the color-dependent model.
Figure 7 compares relative feature selectivity in fixation and discrimination trials for the different subclasses of cells. Neurons whose responses are best explained by the stimulus-agnostic models span across quadrants (Fig. 7A), reflecting that responses of neurons in this group were not consistently more selective for either shape or color. As expected from the simulation in Figure 2, relative feature selectivity was similar for responses during fixation and discrimination trials (paired t test using log-transformed ratios, p = 0.43). Although stimulus-agnostic neurons are expected to fall on the diagonal, several appear to deviate from this line. This is because neurons in this subgroup were very weakly selective for both stimulus shape and stimulus color, and small changes in feature selectivity appear amplified in this graphic representation.
Interestingly, neurons whose responses were best explained by the more complex stimulus-dependent models (Fig. 7B) showed a similar trend as the stimulus-agnostic neurons. Relative feature selectivity was similar during fixation and discrimination trials for both models (paired t test using log-transformed ratios, p = 0.59 and p = 0.94 for the 6-parameter and 9-parameter model, respectively). A few of these neurons became more shape-selective during discrimination, while a few others became more-color selective, but the vast majority were close to the diagonal, with similar relative feature selectivity during fixation and discrimination. Neurons in this subgroup were strongly selective for both shape and color, and did show changes in absolute selectivity for shape and for color during discrimination trials. However, these changes were proportionate such that relative feature selectivity was maintained across different task contexts.
Neurons best explained by the shape-dependent model (Fig. 7C, black) largely occupy the upper right quadrant (i.e., they are relatively more shape- than color-selective in both tasks). Many of these neurons fall above the identity line (in the blue zone, consistent with an increase in relative shape selectivity during discrimination) and a few fall below the identity line. Overall, there was a net increase in relative shape selectivity, but this increase did not reach significance (paired t test using log-transformed ratios, p = 0.13). Neurons best explained by the color-dependent model occupy the lower left quadrant because their responses are relatively more color- than shape-selective in both tasks (Fig. 7C, red). Although many of these neurons became more color-selective during discrimination (points falling below the identity line in the yellow zone), many others became less color-selective (falling above the identity line in the blue zone); thus, as for the shape-dependent group, there was not a significant difference in relative feature selectivity depending on task context (paired t test using log-transformed ratios, p = 0.14). To more broadly examine the changes in feature selectivity, we calculated an alternative ratio of selectivity as selectivity for the modulated feature relative to selectivity for the unmodulated feature. Here, the “modulated” feature refers to shape for neurons best fit by shape-dependent models and color for neurons best fit by the color-dependent models. Thus, this modulatory ratio captures more general changes in the group of neurons best explained by the simple stimulus-dependent models (Fig. 7D) and was significantly different during discrimination trials (paired t test using log-transformed ratios, p = 0.045). These results suggest that neurons whose responses were best explained by the model incorporating stimulus shape or by the model incorporating stimulus color also tended to become more strongly selective along that dimension.
In summary, neurons that were weakly shape- and color-selective showed minimal changes in feature selectivity between fixation and discrimination. Neurons that were strongly selective for shape or color tended to show an increase in selectivity for the preferred dimension. Neurons that were strongly tuned to both shape and color tended to show proportionate changes along both feature dimensions with no net change in relative selectivity. These results support the hypothesis that more selective neurons tend to be more flexible in their selectivity and neurons with an imbalance in selectivity (i.e., stronger selectivity for one stimulus dimension) tend to enhance this imbalance during behavioral discrimination.
Prior studies of V4 responses during visual search have reported that changes in selectivity attributable to feature-based attention emerge in the later portion of the neuronal response starting ∼50 after response onset (Ipata et al., 2012). To compare our observations more directly to this work, we performed the main analyses using an averaging window truncated earlier in time (50-125 ms from stimulus onset) to capture whether the selectivity changes we observe are present in the early portion of the neuronal response. Given V4 response latencies (Zamarashkina et al., 2020), we reasoned that the bulk of feature-attention based would commence after ∼125 ms from stimulus onset. Our main findings replicate with this new time window (compare with Table 1): the majority of neurons (53 of 83, 64%) were still best fit by the stimulus-dependent models, although the proportion of neurons best fit by the stimulus-agnostic models increased to 36% (30 of 83). This difference relative to the longer time window appeared to be because of fewer neurons whose responses were best described by the complex stimulus-dependent models with more parameters (17 of 83, compared with 28 of 83 for the original 50-200 ms window), especially the model with 9 parameters. Meanwhile, the proportion of neurons best described by simple stimulus-dependent models remained similar (36 of 83, compared with 39 of 83 for the original 50-200 ms window). The patterns of feature selectivity were very similar to those we report in Table 2. Overall, the results of the early time window analyses suggest that neurons whose responses are dependent on both stimulus attributes tend to be stimulus-agnostic in the early portion of the response, meaning any task-dependent modulations observed in these neurons are consistent with the effects of top-down attentional modulation seen in later portions of the neuronal response by Ipata et al. (2012). In contrast, neurons whose responses are dependent on stimulus shape or stimulus color alone show the stimulus-dependent modulation even in the early portion of the neural responses, deviating from the late top-down effects expected from Ipata et al. (2012).
Influence of other task-related factors
Next, we examined whether additional task-related factors contributed to the observed response modulations in our recorded population: the relationship between neuronal preference and the reference stimulus, and the saccadic eye movement reporting the animal's decision.
Influence of reference stimulus
Previous studies of feature attention have reported that response modulation may be dictated by whether a neuron's feature selectivity/preference for object features matches the stimulus feature being attended (the “feature-similarity gain model”) (Treue and Martínez-Trujillo, 1999; Bichot et al., 2005; Zhou and Desimone, 2011). For example, if a neuron's preferred shape is a square, and the reference stimulus in the discrimination task is also square, that neuron's responses would be enhanced. Although the present study compares two task conditions that are different from those typically used to study attentional effects, this interaction between stimulus and neuronal preference could contribute to the observed modulation in our data.
To consider this possibility, we conducted a two-way ANOVA on the responses during the test period with reference and test stimulus identity as factors. We found that 11 of 83 neurons showed a main effect of reference stimulus identity (77 of 83 neurons showed a main effect of test stimulus identity; 6 of 83 showed an interaction), suggesting that for a few neurons, there may indeed be some effect of the identity of stimulus being remembered or attended. To quantify such an effect, we compared the response modulation observed when the reference stimulus matched the preferred shape of the neuron (ref-pref) versus when it did not (ref-other). Our data suggest that V4 neurons did not show a consistent enhancement of responses in reference-preferred trials. We performed this analysis separately for preferred stimulus shape and preferred stimulus color. A relative effect close to 2 suggests that the observed modulation because of task context is largely because of a match between the neuron's preferred stimulus and the reference stimulus (see Eq. 4a). For these data, the average effect of a reference stimulus with the preferred shape was −2.9 ± 2.6 (not significantly different from zero, p = 0.27; median = −0.03; significantly different from 2 at α = 0.1, p = 0.0649). The average effect of a reference stimulus with the preferred color was −1.6 ± 1.5 (not significantly different from zero, p = 0.3; median = 0.009; significantly different from 2, p = 0.0215). Since the observed distribution of the effects was not significantly different from zero and was significantly different from 2, we posit that the contribution of any modulatory effect associated with the reference stimulus was minimal.
Influence of eye movements
Additionally, we considered whether the animal's motor plan contributed to the observed modulation. Eye movements have long been known to influence responses in extrastriate cortex (Fischer and Boch, 1981), and many attentional studies account for their effects (e.g., by comparing trials with saccades to a stimulus within a neuron's RF to trials with saccades to a stimulus outside of the RF). In contrast to classic attentional paradigms, the two saccade target dots in our study were both located outside of the RF, precluding such an analysis. Instead, we pursued the effect of other potential motor-related effects that could arise from the arrangement of target dots. In correct trials where the reference stimulus and the test stimulus had the same shape, the animal always made a saccade to the target location on the right side of the screen; when the two stimuli had different shapes, the saccade was to the left. Additionally, recording sites in both animals covered RF locations in the lower portion of the right visual hemifield; this imbalance could be reflected in the difference between rightward and leftward saccade trials since saccades toward the RF are associated with larger responses than saccades away from the RF (Han et al., 2009; Gee et al., 2010). To determine whether the eye movement requirements of the task contributed to the effects we observed, we performed analyses similar to the reference stimulus analysis above, with data separated into two groups based on the saccade outcome of each discrimination trial.
In a two-way ANOVA with interaction, 4 of 83 neurons showed a main effect of saccade direction (82 of 83 neurons showed a main effect of test stimulus identity; 13 of 83 showed an interaction). To quantify the potential contribution of saccade direction to the neural response, we examined the difference in the average neuronal responses in rightward-saccade and leftward-saccade trials, relative to the difference of responses in the fixation and discrimination trials (see Eq. 4b). The relative effect of saccade direction was, on average, 0.02 ± 0.22 (not significantly different from zero, p = 0.914; median = −0.08; significantly different from 2, p ≪ 0.001). Although some individual neurons showed some difference in response related to saccade direction, across the whole population there was no difference in response magnitude. These results suggest that there was no consistent relationship between responses in trials ending in rightward versus leftward saccades, as was intended by selecting an early spike averaging window of 50-200 ms.
Discussion
To determine how the feature selectivity of V4 neuron responses changes in different behavioral tasks, we compared responses of 83 neurons to the same stimuli presented in two task contexts: a passive fixation task and an active shape discrimination task. For 16 of 83 neurons in our population (19%), the difference in responses during these tasks could be explained by a straightforward stimulus-agnostic scaling of responses. For the other 67 of 83 neurons (81%), the difference in responses between active and passive tasks was stimulus-specific and depended on the shape and/or color of the stimulus. Across the population, we found that neurons that were strongly selective for either stimulus dimension were more likely to exhibit stimulus-dependent differences in responses between fixation and discrimination. For neurons best described by the simple stimulus-dependent models where only one feature explained the response modulation during discrimination, responses became more strongly selective for the modulated feature; and this selectivity change was present soon after neuronal response onset. Below we discuss how our data relate to other observations of tuning changes, and the potential advantages of stimulus-specific flexibility in tuning strength during behavior.
Relationship to prior studies: changes in response gain, tuning width, and peak
Many past studies have firmly established the idea that attention scales the gain of responses in sensory areas (McAdams and Maunsell, 1999; Maunsell and Treue, 2006). In our task design, because one task requires active shape-based discrimination and the other does not, it would be reasonable to expect results similar to prior attentional studies. Indeed, roughly a fifth of the neurons in our population displayed responses that were in line with prior V4 experiments involving attentional manipulations. A substantial number of neurons in our population showed response suppression during discrimination, rather than the enhancement commonly reported (though we believe this finding is underreported rather than unique to our paradigm, see, e.g., Maunsell et al., 1991). In either case (enhancement or suppression), the neurons with responses that were most consistent with gain-scaling were typically weakly selective for stimulus color and shape. For the majority of the neurons in our study that were strongly selective for color and/or shape, the responses during discrimination could not be explained on the basis of stimulus-agnostic gain scaling. Some prior studies have documented changes in tuning width and peak position in several cortical regions. For example, V4 orientation tuning becomes narrower in difficult discrimination tasks compared with easy discrimination tasks (Spitzer et al., 1988), as well as after perceptual learning (see Yang and Maunsell, 2004; Raiguel et al., 2006); and V4 spectral RFs shift toward the spectral features of a target object during feature-based search (David et al., 2008). Along the dorsal stream, neurons in lateral intraparietal area display tuning peak shifts for stimulus color and direction of motion, depending on the feature relevant for the task (Ibos and Freedman, 2014) and in MSTd, tuning for optic flow becomes narrower when this stimulus feature is relevant for the behavior, relative to when it is irrelevant (Dubin and Duffy, 2007). In our case, we find little evidence of peak shifts (e.g., the preferred stimulus feature for the example neurons was the same regardless of task; Figs. 3–5), and our results are more in line with previous work in area V4 showing stable orientation tuning across task contexts (Maunsell et al., 1991; though in that study, no notable stimulus-dependent changes in orientation selectivity were reported) and findings from IT cortex, where few neurons show a change in the peak of color tuning during task-context related modulation (Koida and Komatsu, 2007). We do, however, observe a variety of tuning width changes, measured in terms of changes in the strength of feature selectivity between passive fixation and active behavior. While some neurons exhibit stronger or weaker tuning along one of the stimulus dimensions (shape or color), others exhibit changes along both, either in concert (e.g., tuning along both becomes stronger, or in opposite directions). Evidence from PFC suggests distinct neuronal cell types are associated with changes in selectivity during discrimination tasks (narrow-spiking putative interneurons) (Hussar and Pasternak, 2009). Although we were unable to classify neurons in our recordings, cell type-specific functional flexibility remains an intriguing possibility for further exploration. Overall, our results provide the first demonstration that more strongly tuned V4 neurons exhibit more flexible tuning and raise the possibility that the more widely observed stimulus-agnostic scaling may be a function of stimulus choice.
Advantages of flexible stimulus-dependent tuning
While we observed a great diversity of changes in tuning strength across individual V4 neurons, we could identify several critical population trends. First, more strongly tuned neurons exhibited greater flexibility in tuning strength (i.e., stimulus-dependent modulation, and task-related changes in relative feature selectivity). Are such changes advantageous for discrimination behavior? Intuitively, stronger selectivity might improve the accuracy of readout: more selective responses are also more discriminable. However, we observed both strengthening and weakening of tuning among strongly tuned neurons. At the population level, information readout is not necessarily better when tuning is narrower, and it is possible that the heterogeneity in our recorded population was indeed advantageous. A direct way to assess this possibility in the future would be to examine noise covariance using simultaneous recordings from many neurons in passive and active tasks (Pouget et al., 1999; Zhang and Sejnowski, 1999).
Second, in neurons with an imbalance in selectivity, that is, more strongly tuned to one dimension compared with the other, tuning flexibility typically served to further enhance this imbalance in selectivity. For example, a V4 neuron more strongly tuned to shape than color during fixation could become even more strongly shape- than color-selective. Meanwhile, neurons strongly tuned for both dimensions maintained the balance despite changes in feature selectivity. This observation suggests that such neurons are subject to different modulations because they may be read out differently by downstream areas, or inform the behavior in disparate ways. Future studies might benefit from looking at the underlying source of the modulation. For example, if the imbalance in selectivity is amplified through the feedback from downstream areas, it might be revealed in the temporal dynamics of the shape- and color-selective responses (Fyall et al., 2017).
A more general observation from the current study is that while shape was the relevant feature in the discrimination task, both the shape and the color of the stimulus appeared to underlie the task-dependent changes in V4 responses. This finding suggests that the task-related modulation may not be associated with feature-based attentional processes, but rather object-based attention as proposed in prior studies (Roelfsema et al., 1998; Pooresmaeili et al., 2014; Cohen and Tong, 2015); and is consistent with a role for area V4 in building object-based representations (Pasupathy et al., 2020). Insights from studies of working memory and attentional control in PFC suggest that neurons carrying signals combining task- and memory-related information can subserve flexible representations of visual stimuli and thus guide behavior in a task-dependent manner (Rigotti et al., 2013; Panichello and Buschman, 2021), and indeed the differences we observe between the passive and active tasks could be related to the involvement of working memory in the discrimination task.
Relationship to other task-related factors: relevance of stimulus features, stimulus preferences, and eye movements
Are the observed changes in feature selectivity specific to shape discrimination tasks, or do they reflect more general differences between passive fixation and active tasks? In our population, the simple stimulus-dependent neurons showed an enhancement of selectivity for the modulated feature (shape OR color), and not the feature relevant for behavior (shape). We were also able to record data from 9 neurons in M1 during a color discrimination task, in addition to the shape discrimination task. The response changes were similar in both active tasks compared with the passive task. This suggests that our observations reflect a difference in responses during tasks requiring a stimulus-based judgment, compared with passively viewing. The behavioral relevance of the stimulus feature does not appear to be a factor influencing the response modulation. Based on our observations, when an animal is engaged in a behavioral judgment based on shape or color, relative selectivity for shape is enhanced in some neurons, and relative selectivity for color is enhanced in others, regardless of the relevant stimulus feature. However, we cannot completely rule out the possibility that the task-relevant feature does contribute in some way to the response modulations in different task contexts. For example, with a targeted experiment collecting responses from many neurons to the same stimuli in both a shape discrimination task and a color discrimination task, it would be possible to assess whether the former modulates only the portion of the V4 response encoding shape, and the latter modulates only the portion encoding color. Such putative modulations could be described by a stimulus-agnostic model, assuming that the overall neuronal response is a linear combination of the responses to the shape and color (consistent with Bushnell and Pasupathy, 2012). Further detailed study would shed light on the contribution of task-related factors that are associated with feature relevance.
We were able to rule out the contribution of two additional task-related factors: the relationship between neuronal preferences and the attended (or “reference”) stimulus in the behavioral task (the “feature-similarity gain model”) (Treue and Martínez-Trujillo, 1999; Martínez-Trujillo and Treue, 2004; Bichot et al., 2005), and the influence of eye movement planning (e.g., Han et al., 2009). Our observations appear to be specifically associated with responses to the stimulus in the RF, and not characteristics of the reference stimulus or saccade direction.
In conclusion, we have shown that responses of many V4 neurons are modulated by task context, and these modulations are stimulus-dependent for more strongly feature-selective neurons. The observed task-related changes largely pertain to tuning strength, without changes in other characteristics of tuning, such as the peak. Future studies will need to investigate whether these results generalize to other feature dimensions (e.g., texture), whether changes in feature selectivity improve information about stimulus features, and how such changes might inform behavior. Our results emphasize the importance of accounting for the influence of task context to understand how visual stimuli are represented in the responses of V4 neurons.
Footnotes
This work was supported by National Eye Institute Grant R01EY018839 to A.P.; National Eye Institute Center Core Grant for Vision Research P30EY01730 to the University of Washington; National Institutes of Health/ORIP Grant P51OD010425 to the Washington National Primate Research Center; and University of Washington Vision Training Grant National Eye Institute T32EY007031, National Science Foundation GRFP DGE-1256082, and University of Washington Computational Neuroscience Training Grant National Institutes of Health T90DA032436 to D.V.P. We thank Amber Fyall for expert animal care and two anonymous reviewers for suggestions helping connect this work to a wider context of the field.
The authors declare no competing financial interests.
- Correspondence should be addressed to Dina V. Popovkina at dina4{at}uw.edu
