Abstract
When spatial attention is directed toward a particular stimulus, increased activity is commonly observed in corresponding locations of the visual cortex. Does this attentional increase in activity indicate improved processing of all features contained within the attended stimulus, or might spatial attention selectively enhance the features relevant to the observer's task? We used fMRI decoding methods to measure the strength of orientation-selective activity patterns in the human visual cortex while subjects performed either an orientation or contrast discrimination task, involving one of two laterally presented gratings. Greater overall BOLD activation with spatial attention was observed in visual cortical areas V1–V4 for both tasks. However, multivariate pattern analysis revealed that orientation-selective responses were enhanced by attention only when orientation was the task-relevant feature and not when the contrast of the grating had to be attended. In a second experiment, observers discriminated the orientation or color of a specific lateral grating. Here, orientation-selective responses were enhanced in both tasks, but color-selective responses were enhanced only when color was task relevant. In both experiments, task-specific enhancement of feature-selective activity was not confined to the attended stimulus location but instead spread to other locations in the visual field, suggesting the concurrent involvement of a global feature-based attentional mechanism. These results suggest that attention can be remarkably selective in its ability to enhance particular task-relevant features and further reveal that increases in overall BOLD amplitude are not necessarily accompanied by improved processing of stimulus information.
Introduction
Attending to a spatial location typically results in improved visual performance at that location (Posner, 1980), such as improved stimulus detection, spatial resolution, contrast sensitivity, and orientation discrimination (Lee et al., 1997; Yeshurun and Carrasco, 1998; Carrasco et al., 2004; Baldassi and Verghese, 2005; Ling et al., 2009). Neuroimaging studies in humans (Brefczynski and DeYoe, 1999; Gandhi et al., 1999; Somers et al., 1999) and single-cell recordings in monkeys (Motter, 1993; Roelfsema et al., 1998; McAdams and Maunsell, 1999; Reynolds et al., 2000; Herrero et al., 2008) suggest that the behavioral benefits of spatial attention are mediated by stronger activity for attended than unattended stimulus locations in early visual areas. Thus, when subjects direct their attention to a spatial location, neural responses are boosted for stimuli presented at the attended location, allowing for improved visual performance.
It is generally assumed that spatial attention enhances the processing of all stimulus features appearing at the attended location attributable to an increase in the gain or strength of neuronal responses at that location (McAdams and Maunsell, 1999; Womelsdorf et al., 2008; Boynton, 2009; Reynolds and Heeger, 2009). For example, if a behavioral task required attention to the contrast of an oriented grating, enhanced processing of the orientation of the grating should also be observed. However, surprisingly few studies have tested this hypothesis directly in the visual cortex. Previous neurophysiological studies have measured the effects of spatial attention on neural tuning functions for a particular feature, such as orientation (McAdams and Maunsell, 1999), but only when that feature was relevant to the animal's behavioral task (e.g., in an orientation discrimination task). This raises the possibility that the observed enhancement in feature encoding may depend on task relevance (Treue and Maunsell, 1996; Treue and Martínez Trujillo, 1999; Martínez-Trujillo and Treue, 2004), rather than spatial attention per se.
In the first experiment reported here, we investigated the effects of spatial attention on the strength of orientation-selective responses using two behavioral tasks (see Fig. 1), one in which the orientation of the attended stimulus was task relevant (an orientation discrimination task) and one in which it was not (a contrast discrimination task). We reasoned that, if enhanced encoding of orientation were to be found for both behavioral tasks, this would suggest that spatial attention does indeed modulate the representation of all stimulus features at the attended location. However, if stronger orientation-selective responses were to be found only when orientation is a task-relevant feature, then this would challenge the assumption that spatial attention enhances the stimulus-driven response of all neurons whose receptive field overlaps with the attended location. In experiment 2, we further investigated the effects of top-down attention on feature-selective responses by presenting red or green oriented gratings and instructing observers to discriminate small variations in the color or orientation of the spatially cued grating. Using functional magnetic resonance imaging (fMRI), we consistently observed increases in BOLD activation with spatial attention in early visual areas V1–V4 across all behavioral tasks. However, multivariate pattern classification methods (Kamitani and Tong, 2005; Haynes and Rees, 2006; Norman et al., 2006) revealed that feature-selective responses were not always enhanced by spatial attention and that task relevance had an important modulatory role. The results further indicated that increases in BOLD amplitude were not always accompanied by improved encoding of stimulus information.
Materials and Methods
Subjects.
A total of nine healthy adult volunteers (aged 24–36 years, four female), with normal or corrected-to-normal vision, participated in this study. Six volunteers participated in experiment 1, which required performing an orientation discrimination task and a contrast discrimination task in separate MRI scanning sessions. Three of these volunteers and three additional volunteers participated in experiment 2, which directly compared the effects of performing orientation and color discrimination tasks on color-tinted oriented gratings in a single experimental session. Each scanning session was ∼2–2.5 h in duration. All subjects provided informed written consent. The study was approved by the Vanderbilt University Institutional Review Board.
Experimental design and stimuli.
Visual stimuli were generated by a Macbook Pro computer running Matlab and Psychophysics Toolbox software (Brainard, 1997; Pelli, 1997) and displayed on a rear-projection screen using a luminance-calibrated Eiki LC-X60 LCD projector with a Navitar zoom lens. Participants viewed the visual display through a mirror that was mounted on the head coil. A custom-made bite-bar system was used to minimize participant's head motion. Eye position was monitored using an MR-compatible Applied Science Laboratories EYE-TRAC 6 eye-tracking system.
In experiment 1, we measured fMRI responses in early visual areas for spatially attended and unattended gratings, while observers performed either an orientation or contrast discrimination task at the attended stimulus location. Participants were required to maintain fixation on a central bull's eye throughout each experimental run and to attend covertly to one of two laterally presented gratings (Fig. 1). A central cue instructed participants to shift attention from one grating location to the other at the beginning of each 16 s stimulus block. We used fMRI decoding methods to assess the reliability of orientation-selectivity activity patterns in early visual areas for attended and unattended stimuli (Kamitani and Tong, 2005), by training linear classifiers to predict which of two possible base orientations (∼55° or ∼145°) was shown at each stimulus location. In experiment 2, participants performed either a color or orientation discrimination task on one of two laterally presented gratings, and fMRI decoding was used to assess the reliability of both orientation- and color-selective activity patterns in early visual areas.
Stimuli and experimental procedure. Example of a trial sequence from experiment 1. Subjects fixated a central bull's eye target while gratings of independent orientation (∼55° or ∼145°) appeared in each hemifield. A compound white/black cue indicated whether subjects should attend to the left or right stimuli; in this example, the black circle indicates “attend right.” Subjects had to discriminate near-threshold changes in orientation (experiments 1 and 2), contrast (experiment 1), or color (experiment 2) between successive pairs of gratings presented at the cued location. Red circles depict the attended location and were not present in the actual display.
Each experimental run consisted of an initial fixation block followed by eight stimulus blocks and a final fixation block (block duration, 16 s). Each stimulus block consisted of four visual discrimination trials, during which counterphasing gratings of independent orientation (∼55° or ∼145°) were shown, centered at 5° of visual angle to the left and to the right of fixation (grating radius, 3.5°; spatial frequency, 1.0 or 1.5 cycles/° with randomized spatial phase; temporal frequency, 2 Hz sinusoidal contrast modulation). The orientation/contrast discrimination experiment (experiment 1) used grayscale luminance-defined sinusoidal gratings, the edges of which were attenuated by linearly decreasing the grating contrast over the distance from 3.0° to 3.5° radius (Fig. 1). Full contrast gratings were used in the orientation discrimination task, and a base contrast of 80% was used in the contrast discrimination task. For this experiment, the gratings successively presented at each location differed in spatial frequency (1.0 and 1.5 cycles/°). In the orientation/color discrimination experiment (experiment 2), we used red- and green-tinted square-wave gratings defined in Judd's CIE color space (Judd, 1951), for which chromaticity values are luminance independent (spatial frequency, 1.0 cycles/° with randomized spatial phase; temporal frequency, 2 Hz square-wave contrast modulation; edges, abrupt at 3.5° radius). The square-wave pattern was defined by modulations in the luminance dimension (contrast, 67%), and a uniform amount of red or green was added to the entire circular grating (mean chromaticity values for red grating: x, 0.57; y, 0.37; green grating: x, 0.37; y, 0.56). A minimum motion technique (Anstis and Cavanagh, 1983) was applied to produce luminance-equated tints of red and green for each individual observer.
We used a compound white/black cue that straddled the fixation point (±0.5°) to indicate with 100% validity which of the lateralized gratings should be attended for the visual discrimination task. The design of this compound central cue ensured balanced visual stimulation in the two hemifields. Subjects were instructed to attend to the grating on the same side of fixation as either the white or black portion of the compound cue; the relevant cue color was reversed every run in experiment 1 and across subjects in experiment 2. The cued location alternated between the left and right grating locations between every stimulus block.
Each trial consisted of a brief presentation of the central cue (250 ms on, 250 ms off), followed by a pair of gratings on either side of fixation (1000 ms), a 500 ms interstimulus interval, and a second set of gratings (1000 ms). After the second pair of gratings was removed, subjects had 1000 ms in which to make a two-interval forced-choice judgment about the gratings shown at the attended location (Fig. 1). For the orientation discrimination task, subjects had to report whether the second grating was rotated clockwise or counterclockwise relative to the first grating, by pressing a corresponding key on an MRI-compatible button box. In the contrast discrimination task, subjects had to report whether the second grating was of higher or lower contrast than the first grating. In the color discrimination task, subjects had to indicate whether the second grating appeared more reddish or greenish relative to the first grating. Note that, in the orientation experiment, subjects had to discriminate small changes in orientation of just a few degrees around two possible base orientations (55° or 145°), whereas fMRI decoding was used to predict which of the two base orientations was seen. Similarly, in the color experiment, subjects discriminated slight changes in hue between two successive gratings (more or less red or green), whereas the classifier simply predicted whether the overall hue was primarily red or primarily green. Thus, the subjects' behavioral responses regarding these fine discriminations were not predictive of the coarse stimulus changes.
These very small changes in orientation, contrast, or color for the two-interval forced-choice tasks were determined by an adaptive staircase procedure to maintain near-threshold performance at ∼80% accuracy (Watson and Pelli, 1983). For experiment 2, changes in color were computed in the L*a*b* color space of the 1986 Commission Internationale de l'Eclairage (CIE) for a common L*a*b* value, so that the chromaticity of the entire grating could be varied slightly across successive presentations, independently of the luminance-defined orientation pattern. It should be noted that human observers are somewhat more perceptually sensitive to chromatic variations for stimuli presented at higher luminance levels (Kaiser and Boynton, 1996). As a consequence, the higher luminance regions in the gratings might have been more informative for the observer's color discrimination task. Nonetheless, this stimulus design was the preferred option for separating the color and orientation components of the stimulus. Had we instead manipulated the changes in color separately for the high and low luminance regions of the grating, this would have introduced a physical color component in the orientation signal because of how the CIE L*a*b* color space is constructed. Orientation, contrast, and color variations of equal magnitude were applied at both attended and unattended stimulus locations; however, the interval of change (i.e., first or second grating) and direction of change (e.g., clockwise or counterclockwise) was randomly determined at each location to ensure independence. In the contrast discrimination experiment, a small amount of orientation jitter was added to match the average orientation change (for each subject) in the orientation task. In the orientation/color discrimination experiment, a small amount of orientation jitter (in the color discrimination task) or color jitter (in the orientation discrimination task) was introduced to match the average orientation or color change in the other task.
Participants completed 15–22 orientation runs and 16–22 contrast runs (experiment 1). In experiment 2, subjects performed 12 color discrimination runs and 12 orientation discrimination runs, with the task order counterbalanced across subjects. Before the actual experiment, subjects practiced the task to estimate the stimulus difference required to maintain ∼80% accuracy. In five runs (of 382 total), perceptual threshold estimates increased steadily over the course of the run, suggesting that subjects had misinterpreted the central cue and performed the task at the opposite spatial location. This was subsequently confirmed by subject self-reports, and these runs were therefore excluded from additional analysis.
Each scanning session also included two visual localizer runs, in which subjects viewed flickering checkerboard stimuli presented in the same spatial window as the lateral gratings (checker size, 0.4°; display rate, 10 images/s; edge, 0.5° linear contrast ramp). The checkerboard stimulus was alternately presented in the left and right hemifield for 12 s blocks, with fixation blank periods occurring at the beginning, between blocks, and at the end of each 300 s run.
Data acquisition.
MRI data were collected on a Philips 3.0 Tesla Intera Achieva MRI scanner at the Vanderbilt University Institute for Imaging Science, using an eight-channel head coil. A high-resolution 3D anatomical T1-weighted scan was acquired from each participant (FOV, 256 × 256; 1 × 1 × 1 mm resolution). To measure BOLD contrast, standard gradient-echo echoplanar T2*-weighted imaging was used to collect 28 slices perpendicular to the calcarine sulcus, covering the entire occipital lobe as well as the posterior parietal and posterior temporal cortex (TR, 2000 ms; TE, 35 ms; flip angle, 80°; FOV, 192 × 192; slice thickness, 3 mm with no gap; in-plane resolution, 3 × 3 mm). Subjects used a bite-bar system to minimize head movement.
Functional MRI data and preprocessing.
Data were initially motion corrected using automated image registration software in experiment 1 and FSL (for Functional MRI of the Brain Software Library) (McFlirt) in experiment 2. Brain Voyager QX (version 1.8; Brain Innovation) was used for subsequent preprocessing, including slice timing correction and linear trend removal. No spatial or temporal smoothing was performed. The functional volumes were aligned first to the within-session anatomical scan and then to the previously collected retinotopic mapping data, by rigid-body transformations. All automated alignments were inspected and manually refined when necessary. After across-session alignment, fMRI data underwent Talairach transformation and reinterpolation using 3 × 3 × 3 mm voxels.
Regions of interest.
Retinotopic mapping of visual areas was performed in a separate scan session using well-established methods (Sereno et al., 1995; DeYoe et al., 1996; Engel et al., 1997). Voxels used for decoding analysis were identified within each hemisphere for retinotopic areas V1, V2, V3, V3A, and V4. First, voxels near the gray–white matter boundary were identified within each visual area based on retinotopic maps delineated on a flattened cortical surface representation. Next, we identified the visually active voxels corresponding to the left and right grating locations, based on statistical activation maps obtained from the visual localizer runs. Decoding analyses were performed on three types of region of interest (ROI): contralateral ROIs, ipsilateral ROIs, and a foveal ROI. As an example, when decoding the orientation of an attended grating in the left visual field, contralateral ROIs would consist of voxels in the right hemisphere that receive direct input from the attended left grating location, ipsilateral ROIs would consist of voxels in the left hemisphere that receive direct input from the ignored right grating location, and the foveal ROI consisted of voxels in the foveal confluence of each hemisphere that responded to neither grating location.
In experiment 1, we selected 120 voxels from each of V1, V2, and V3, separately for each hemisphere, which were most significantly activated by the contralateral localizer stimulus (the selection of 120 voxels corresponds to an approximate activation threshold of p < 0.05). Because the V3A and V4 ROIs were substantially smaller than the earlier visual areas, for these analyses the two areas were combined into a single ROI, from which the 120 most significantly active voxels in each hemisphere were chosen (Kamitani and Tong, 2005). Classification accuracy was not strongly affected by further increasing the number of voxels selected, with similar accuracy levels found for up to 200 voxels from each visual area (supplemental Fig. S1, available at www.jneurosci.org as supplemental material).
We tested for spatial spreading of feature-selective activity beyond the cortical regions receiving direct input, by applying the same decoding analyses to ipsilateral ROIs and the foveal ROI. For the ipsilateral region of interest, we selected voxels ipsilateral to the stimulus location that were significantly activated by the localizer stimulus when presented in the mirror symmetric location in the contralateral visual field. For this region of interest, we excluded voxels that lay within 6 mm of the cortical midline as a conservative measure to avoid any possible inclusion of voxels encompassing portions of the contralateral hemisphere. The 120 voxels most significantly activated by the functional localizer were then chosen from within these restricted ROIs. Of the 480 voxels per hemisphere selected for the main analysis, only 17 voxels on average were excluded from the ipsilateral analysis because of proximity to the midline. For the foveal region of interest, we used voxels from the foveal confluence in areas V1–V3 combined. As a conservative measure, we only selected voxels in this ROI that were not significantly activated by the localizer stimulus, obtaining on average 30 voxels per hemisphere.
The color and orientation discrimination experiment (experiment 2) used similar procedures for ROI definition. However, far fewer training samples were available for training the classifier (12 runs of data per task condition compared with ∼20 on average for experiment 1), such that classification performance in individual areas was less reliable than in the previous experiments. Accordingly, for this experiment, 250 voxels were selected in each hemisphere from a combined V1–V4 ROI. To avoid biasing the voxel selection toward the earlier visual areas, which tended to show more significant responses to the localizer stimulus, these 250 voxels were chosen at random from all those in the combined ROI. This randomized selection procedure was repeated 10 times for each hemisphere. Classification accuracy remained stable with increasing numbers of voxels, up to 500 voxels (supplemental Fig. S2, available at www.jneurosci.org as supplemental material).
BOLD amplitude analyses.
Voxels selected for BOLD amplitude analyses were the same as those used for decoding. All data were transformed from MRI signal intensity to units of percentage signal change, calculated relative to the average level of activity for each voxel across the fixation rest periods at the beginning and end of each run. Mean responses per block (see Fig. 2B) were calculated by averaging the BOLD response within the relevant time window, after incorporating a 4 s delay to account for hemodynamic lag.
fMRI data samples used for decoding.
All fMRI data were transformed from MRI signal intensity to units of percentage signal change, calculated for each voxel in attended (unattended) blocks relative to its average amplitude across all attended (unattended) blocks within a given run. The fMRI time series were shifted by 4 s to account for hemodynamic delay. All data belonging to the same block were then temporally averaged, arriving at a spatial pattern of time-averaged activity for each block, consisting of the amplitude of every preselected voxel within the region of interest. All samples were labeled according to the stimulus orientation or color and analyzed separately for attended and unattended conditions.
Linear classifier.
We used multivariate pattern classification methods to decode the viewed orientation or color from fMRI activity patterns (Kamitani and Tong, 2005), enabling us to determine whether the strength of feature-selective fMRI responses is affected by spatial attention or behavioral task. Decoding performance gives an indication of the amount of orientation or color information available on the scale of fMRI voxels, such that relative changes therein can be informative about the effects of attention. This approach is illustrated here using orientation as an example, but we applied the same procedures to estimate the amount of color information carried by the fMRI activity patterns (Sumner et al., 2008; Brouwer and Heeger, 2009; Seymour et al., 2009). Individual fMRI voxels sampled from the visual cortex show a weak but reliable preference for particular orientations, presumably because of random millimeters-scale variations in the spatial distribution of orientation-selective cortical columns (Swisher et al., 2010). Pattern classification methods can effectively pool the information available across many such weakly tuned fMRI voxels, allowing a presented orientation to be decoded from coarse-scale population responses. An increase in the strength of orientation-selective fMRI responses is reflected by improved decoding performance.
Data samples were analyzed using a linear classifier to predict the viewed orientation (or color). We used linear support vector machines to obtain a linear discriminant function distinguishing between two orientations θ1 and θ2:
where xj is a vector specifying the BOLD amplitude of all n voxels on block j, xi is the amplitude of voxel i and wi is its weight, and w0 is the overall bias. Linear classifiers attempt to find weights and bias of this discriminant function such that, for a given linearly separable set of training data, the following relationship is satisfied:
Subsequent test patterns were assigned to orientation θ1 when the discriminant function became larger than 0 and to orientation θ2 otherwise. Samples obtained from all but one run were defined as the training set, and the remaining vector was defined as the test pattern (leave-one-run-out cross-validation). We repeated the cross-validation procedure until each run had served as a test run once and calculated the decoding accuracy across all test runs.
Eye tracking.
Eye position was monitored during scanning using an MR-compatible Applied Science Laboratories EYE-TRAC 6 eye-tracking system (60 Hz), for four subjects in the orientation discrimination task and six subjects in the contrast discrimination task (experiment 1). Eye-position data was collected for six subjects in the orientation/color discrimination experiment (experiment 2), but the data from two subjects were excluded from additional analyses as a result of technical difficulties with the eye-tracking system. Data were corrected for blinks and slow linear drift. Breaks from fixation were identified as deviations in eye position >1.5°. We used the mean x and y position for each block, as well as the product and the SD of all these values, as input to the eye-position-based orientation or color decoder (cf. Harrison and Tong, 2009).
Results
Subjects generally performed well at the visual discrimination tasks. Behavioral thresholds were quite stable throughout each experimental session, with mean thresholds of 2.9° in the orientation discrimination task (experiment 1) and a contrast shift of 6% in the contrast discrimination task (experiment 1). Color discrimination thresholds in experiment 2 averaged 2.7° in the ab color plane, and the mean orientation discrimination threshold in this experiment was 1.9°.
Spatial attention increases fMRI response amplitudes
First, we determined whether spatial attention led to stronger overall responses in corresponding regions of the visual cortex. Regions of interest consisted of visually active voxels in V1, V2, V3, V3A, and V4 that were reliably activated by contralateral stimuli in a functional localizer. In each ROI, we compared the amplitude of fMRI responses to attended versus unattended stimuli.
We first focused on the orientation and contrast discrimination tasks (experiment 1). Figure 2A shows the average time course of the BOLD response in V1 across subjects. In both tasks, we observed clear modulations in BOLD activity that followed the time course of the centrally cued shifts of spatial attention. Figure 2B shows mean response amplitudes plotted by visual area. Activity in early visual areas (V1–V4) was significantly higher for attended than unattended stimuli in both the orientation discrimination task (F(1,5) = 63.3, p < 1 × 10−3) and the contrast discrimination task (F(1,5) = 90.1, p < 1 × 10−3). Effects of spatial attention were very similar across visual areas, with no evidence of an interaction between attention and visual area (orientation, F(3,15) = 1.5, p = 0.25; contrast, F(3,15) = 2.2, p = 0.13). Moreover, the degree of attentional modulation was similar across the two tasks (Fig. 2C), as confirmed by a within-subjects ANOVA (F(1,5) = 0.1, p = 0.75). Similar results were found for the orientation/color discrimination experiment (experiment 2), in which spatial attention led to larger response amplitudes in both the color discrimination task (F(1,5) = 38.0, p < 1 × 10−2) and the orientation discrimination task (F(1,5) = 67.8, p < 1 × 10−3), with the degree of attentional modulation being comparable across the two tasks (F(1,5) = 0.25, p = 0.64). Thus, for all tasks and experiments, focal spatial attention resulted in similarly increased activity in corresponding regions of the visual cortex.
Amplitude of the BOLD response in experiment 1. A, Time course of BOLD activity in corresponding regions of area V1, averaged across subjects. Left, Orientation discrimination task; right, contrast discrimination task. Red, Attended stimulus location; blue, Unattended stimulus location. Dashed lines indicate the time period when subjects attended to the corresponding location (0–16 s), before and after which they attended to the other stimulus location. Alternating blocks of attending and ignoring the grating produced a periodic pattern of modulation. Error bars indicate ±1 SEM. B, Mean response amplitudes in areas V1–V4 for attended and unattended blocks. Response amplitudes were significantly higher for attended stimuli than unattended stimuli for both discrimination tasks. C, Magnitude of attentional modulation (attend − unattend) was very similar across the two experimental tasks for all early visual areas.
Attending to orientation enhances orientation-selective responses
Next we asked whether the spatial attentional enhancement of the BOLD response also resulted in stronger feature-selective responses at the attended stimulus location. Voluntary spatial attention involves top-down feedback signals that can increase the baseline activity of visually selective neurons, independent of their feature tuning (Luck et al., 1997; Kastner et al., 1999; Ress et al., 2000). In addition, it is assumed that spatial attention leads to a multiplicative increase in the gain of the response of the neuron, with greater attentional enhancement occurring at the preferred orientation of the neuron (McAdams and Maunsell, 1999; Womelsdorf et al., 2008; Boynton, 2009; Reynolds and Heeger, 2009). Here, we used multivariate pattern classification methods to investigate whether fMRI activity in the visual cortex shows evidence of such an attentional gain mechanism. To the extent that the orientation bias in the response of individual voxels reflects a biased distribution of orientation-selective neurons, it logically follows that voxel responses should be disproportionately boosted when their preferred orientation is spatially attended. Multivariate pattern classification methods can effectively pool the information available across many such orientation-tuned voxels, allowing a presented orientation to be decoded from fMRI activity (see Materials and Methods). If orientation classification performance is selectively enhanced for spatially attended gratings compared with unattended gratings, then this would imply that covert attention can increase in the strength of orientation-selective responses. For each ROI, we trained a linear classifier on the patterns of activity generated by the contralateral attended stimulus during one set of the runs and used this to predict the attended orientations shown during the remaining, independent test runs (Fig. 3A). A separate set of linear classifiers were trained and tested on activity patterns for contralateral unattended gratings, to compare relative performance across attended and unattended conditions. In experiment 2, we applied this same approach to investigate the effects of attention on color-selective fMRI responses.
Orientation decoding results for the contralateral stimulus in experiment 1. A, Illustration of the voxels used to decode the orientation of the contralateral (Contra) stimulus, when that stimulus was attended or unattended. B, Decoding performance in the orientation discrimination task for gratings presented at the attended (red) and unattended (blue) spatial location. Error bars indicate ±1 SEM. Reliably better decoding performance was found for attended than unattended stimuli in areas V1 and V2 (V1, t(5) = 3.1, p < 0.05; V2, t(5) = 2.7, p < 0.05) and approached significance in areas V3 and V3A/V4 (V3, t(5) = 2.2, p = 0.08; V3A/V4, t(5) = 2.3, p = 0.07). C, In the contrast discrimination task, orientation decoding performance was not reliably better for attended compared with unattended stimulus locations (p > 0.1 in all areas).
When subjects performed the orientation discrimination task in experiment 1, we found significantly better orientation decoding performance for attended than unattended stimulus locations throughout the early visual areas (Fig. 3B) (F(1,5) = 7.1, p < 0.05) (supplemental Fig. S1A, available at www.jneurosci.org as supplemental material). Thus, attention could effectively enhance the strength of these orientation-selective responses. We also assessed the similarity of orientation-selective activity patterns across the attended and unattended conditions, by training a classifier on orientations at a given location when it was spatially attended and testing it on orientations shown at that same location when that location was not attended. Generalization performance matched the level seen for training and testing on unattended locations alone, indicating that orientation-selective activity patterns were similar across the two conditions (supplemental Fig. S3A, available at www.jneurosci.org as supplemental material).
We next analyzed the data from the contrast discrimination task to ascertain whether or not the attentional enhancement of orientation processing found in experiment 1 was dependent on the behavioral task performed. Although spatial attention led to much greater BOLD amplitudes at the attended location (Fig. 2, right), decoding of the orientation presented at the attended location was not reliably better than that observed for the unattended location (Fig. 3C) (F(1,5) = 3.8, p = 0.11) (supplemental Fig. S1B, available at www.jneurosci.org as supplemental material). Direct comparison of the effects of attention across the two tasks revealed a statistically significant interaction effect (F(1,5) = 7.5, p < 0.05), indicating that attention better enhanced orientation-selective responses when orientation, rather than contrast, was the task-relevant feature. Decoding of unattended stimulus orientations did not reliably differ across the two tasks (F(1,5) = 0.1, p = 0.77), confirming that stimulus-driven orientation-selective activity was comparable across the two tasks. Thus, attending to the contrast of a grating did not automatically lead to improved encoding of the task-irrelevant feature of orientation.
To investigate the generality of this effect, in a second experiment, we directly compared the effects of attending to the orientation or color of lateralized gratings that varied in both orientation (∼55° or ∼145°) and surface color (red or green). This experiment was identical to previous sessions, except that subjects performed either a color or orientation discrimination task, using the grating presented at the attended location. Because far fewer training samples were available for classification in this experiment, such that classification performance in individual areas was less reliable than in the previous experiments, voxels were selected from a combined V1–V4 ROI.
When subjects performed the color discrimination task, decoding of stimulus color was significantly better at the attended location than the unattended location (Fig. 4, top) (t(5) = 3.4, p < 0.05) (supplemental Fig. S2A, available at www.jneurosci.org as supplemental material). Moreover, when subjects performed the orientation discrimination task, decoding of color at the attended location was no different than at the unattended location (t(5) = 0.2, p = 0.86). Direct comparison of the effect of attention on color responses across these two tasks revealed a statistically significant interaction effect (F(1,5) = 12.6, p < 0.05), indicating that color-selective responses were more discriminable when color, rather than orientation, had to be attended. Comparable decoding performance was found for the unattended colors in both behavioral tasks (t(5) = −0.3, p = 0.75), indicating similar stimulus-driven activity across the two tasks. These results indicate that color-selective responses at the attended location were enhanced only when color was relevant to the observer's task.
Color and orientation decoding results for the contralateral stimulus in experiment 2. Decoding accuracy for color (top) and orientation (bottom) responses in the orientation (left) and color (right) discrimination tasks, for attended (red) and unattended (blue) stimuli. Error bars indicate ±1 SEM. In both tasks, decoding of orientation was reliably better for attended than unattended stimuli in areas V1–V4 combined (bottom) (orientation discrimination task, t(5) = 3.5, p < 0.05; color discrimination task, t(5) = 3.0, p < 0.05). In the color discrimination task, decoding of color at the attended location was reliably better than at the unattended location in areas V1–V4 (top, right) (t(5) = 3.4, p < 0.05). Importantly, however, color decoding performance was not reliably better for attended compared with unattended stimulus locations in the orientation discrimination task (top, left) (t(5) = 0.2, p = 0.86), indicating selective enhancement of color-selective responses in the color discrimination task.
Curiously, in both the orientation and color discrimination tasks, decoding of orientation was significantly better at the attended location than at the unattended location (Fig. 4, bottom) (orientation task, t(5) = 3.5, p < 0.05; color task, t(5) = 3.0, p < 0.05), with no evidence of an interaction across the two tasks (F(1,5) = 0.4, p = 0.56). Why did attending to the color of the grating nonetheless boost the strength of orientation-selective responses? Although we manipulated the chromaticity of the entire grating, many subjects reported after the study that they found it more helpful to attend to the high luminance portions of the orientation grating, because the color differences in these regions seemed more perceptually salient (see Materials and Methods). This behavioral strategy of focusing more on the high luminance portions of the oriented grating, rather than the color of the grating as a whole, may have led to the allocation of attention to grating orientation as well as color. Nevertheless, cortical processing of the task-irrelevant feature of color did not improve when subjects performed the orientation discrimination task. This task-specific boost in color processing is consistent with the task-specific enhancement of orientation responses found in experiment 1. Thus, data from both experiments provide positive evidence that attention is capable of selectively enhancing task-relevant features, without necessarily boosting all of the features contained within the attended stimulus. Such selective enhancement of a specific visual feature could potentially reflect the involvement of a feature-based attentional mechanism (Treue and Maunsell, 1996; Treue and Martínez Trujillo, 1999).
Spatial spreading of feature-selective information across the visual field
We investigated the mechanism underlying the task-specific enhancement of feature-selective responses by testing the spatial specificity of these modulatory effects. Previous work has shown that attending to a specific visual feature can lead to spreading of feature-selective activity across the visual field (Treue and Maunsell, 1996; Treue and Martínez Trujillo, 1999; Saenz et al., 2002; Martinez-Trujillo and Treue, 2004; Serences and Boynton, 2007). We therefore asked whether information about the attended stimulus orientation or color remained confined to that stimulus location or spread to distal locations in the visual field, taking the latter as a signature of feature-based attention.
Specifically, we tested whether the decoder could predict the orientation or color presented at the attended spatial location based on activity patterns obtained from the ipsilateral visual cortex (Fig. 5A). Reliable decoding based on ipsilateral activity would suggest the spreading of top-down feature-specific information to the opposite hemifield, because stimuli confined to one hemifield are known to activate only the contralateral portions of areas V1–V3 (Tootell et al., 1998; Serences and Boynton, 2007). As a control, we also tested whether these activity patterns could predict the orientation or color of unattended ipsilateral stimuli. For these analyses, we selected those voxels in areas V1–V4 that were reliably activated by contralateral stimuli in the functional localizer. Thus, these voxels received strong feedforward input from the unattended contralateral stimulus (Figs. 3, 4).
Spatial spreading of orientation information to ipsilateral and foveal ROIs in experiment 1. A, Illustration of the voxels used to decode the orientation of the ipsilateral (Ipsi) stimulus, when that stimulus was attended or unattended. Here, successful decoding of attended orientations would indicate a global spreading of orientation-selective activity to the opposite hemifield. A similar analysis was performed on the unstimulated foveal region in areas V1–V3. B, Decoding performance in the orientation discrimination task for ipsilateral and foveal ROIs. Red, Attended stimuli; blue, unattended stimuli. Error bars indicate ±1 SEM. Decoding accuracy for attended ipsilateral gratings was significantly greater than chance in V1, V2, V3, and the foveal ROI (V1, t(5) = 6.2, p < 0.01; V2, t(5) = 5.7, p < 0.01; V3, t(5) = 5.7, p < 0.01; V3A/V4, t(5) = 2.0, p = 0.10; V1–V3 fovea, t(5) = 3.5, p < 0.05). Decoding accuracy for unattended gratings did not reliably differ from chance-level performance (all p > 0.1). C, Decoding performance in the contrast discrimination task. Orientation decoding performance was not reliably different from chance levels in any ROI (all p > 0.1).
We first focused on the orientation discrimination task of experiment 1. Results revealed that activity patterns in the visual cortex could reliably predict the orientation of attended ipsilateral stimuli (t(5) = 5.6, p < 0.01) but failed to predict the orientation of unattended ipsilateral stimuli (Fig. 5B) (supplemental Fig. S1C, available at www.jneurosci.org as supplemental material). Next, we investigated whether this spread of orientation information might depend on an orientation-specific interaction between attended and unattended gratings. One possibility might be that some form of automatic grouping or facilitation occurs when the unattended contralateral stimulus matches the orientation of the attended ipsilateral stimulus. To address this possibility, we determined the accuracy of decoding performance separately for trials in which the orientations in the two hemifields were the same or different (after training the decoder on both sets of data). Decoding of the attended ipsilateral orientation was significantly greater than chance in both cases (supplemental Fig. S4, available at www.jneurosci.org as supplemental material) (same orientation, t(5) = 5.3, p < 0.01; different, t(5) = 4.2, p < 0.01), and performance did not reliably differ in these two conditions (F(1,5) = 3.8, p = 0.11). We also tested for spatial spreading of orientation information in the foveal representation in the visual cortex, which consisted of voxels that were not reliably activated by either grating location. The orientation of the attended stimulus could be reliably predicted from voxels representing the foveal confluence of areas V1–V3 combined (t(5) = 3.5, p < 0.05). In contrast, the orientation of the unattended grating could not be decoded from the activity patterns in the foveal confluence (Fig. 5B) (t(5) = −0.2, p = 0.86). Such spatial spreading of orientation information to the ipsilateral and foveal ROIs, found specifically for the attended grating, is consistent with the involvement of a global feature-based attentional mechanism in this task.
We performed a similar set of analyses on the fMRI data collected from the contrast discrimination task in experiment 1. Previous analyses indicated no attentional enhancement of orientation responses in the contralateral ROI. Therefore, we predicted that there should also be no evidence of spatial spreading of orientation information to other regions of visual cortex. Analyses indicated that ipsilateral ROIs could not reliably predict the stimulus orientation of attended or unattended gratings (Fig. 5C) (supplemental Fig. S1D, available at www.jneurosci.org as supplemental material). Similarly, the foveal ROI led to chance levels of decoding performance in this task (Fig. 5C).
Finally, we tested for spatial spreading of orientation and color information in experiment 2, performing similar analyses on an ipsilateral region of interest (V1–V4 pooled) and the foveal ROI. When observers performed the orientation discrimination task, we observed reliable enhancement of orientation responses in the ipsilateral ROI (Fig. 6) (t(5) = 3.5, p < 0.05) (supplemental Fig. S2B, available at www.jneurosci.org as supplemental material), indicating reliable spatial spreading of the attended task-relevant feature. Decoding performance was also slightly higher in the foveal ROI, although not significantly so, presumably because of the lower number of samples available per condition in this experiment (decoding performance, 56%; chance-level performance, 50%; t(5) = 1.2, p = 0.3). In contrast, the orientation of unattended gratings could not be reliably decoded from ipsilateral ROIs, nor did we find evidence of spatial spreading of color information when observers performed the orientation discrimination task (Fig. 6).
Spatial spreading of color and orientation information to the ipsilateral ROI in experiment 2. Decoding accuracy for color (top) and orientation (bottom) responses in the ipsilateral (Ipsi) regions of areas V1–V4 combined, for both attended (red) and unattended (blue) stimuli, in the orientation (left) and color (right) discrimination tasks. Error bars indicate ±1 SEM. Orientation decoding accuracy exceeded chance-level performance for the attended stimuli in the orientation discrimination task (bottom, left) (t(5) = 3.5, p < 0.05), and color decoding accuracy was reliable for attended stimuli in the color discrimination task (top, right) (t(5) = 5.0, p < 0.01). Orientation decoding in the color discrimination task was marginally significant (bottom, right) (t(5) = 2.2, p = 0.08). For unattended gratings, decoding of orientation or color did not significantly differ from chance-level performance in any task (all p > 0.1).
Next, we analyzed the data for the color discrimination task. We found reliable spatial spreading of color information to the ipsilateral ROI (Fig. 6) (t(5) = 5.0, p < 0.01), with a trend toward significance for the foveal ROI (decoding performance, 53%; chance-level performance, 50%; t(5) = 2.5, p = 0.05), consistent with a feature-based attentional mechanism. Given that participants also showed some attentional enhancement of orientation-selective responses in the contralateral ROI for this task, we were curious as to whether orientation information about the attended grating might also show some evidence of spatial spreading, although this information was not explicitly relevant to the participant's task. We found a marginally significant effect of being able to predict the orientation of the attended grating for the ipsilateral ROI (Fig. 6) (t(5) = 2.2, p = 0.08) but no evidence of such information in the foveal ROI (decoding performance, 50%). The results for the ipsilateral ROI are potentially suggestive of spatial spreading of orientation information, although caution should be expressed in interpreting such marginal results, because the orientation-selective activity patterns in this condition were quite weak and variable.
Overall, experiments 1 and 2 provide considerable evidence of spatial spreading of feature-selective information to ipsilateral and foveal ROIs, with positive results obtained whenever a particular feature was relevant to the observer's discrimination task. The potential contributions of feature-based, spatial, and object-based attention are further considered in Discussion.
Eye movement control analyses
We also considered whether eye movements could potentially account for the observed fMRI results. Eye position was successfully monitored in the MRI scanner for the majority of our subjects, and analysis of these data confirmed that subjects maintained stable fixation throughout the visual tasks (supplemental Table S1, available at www.jneurosci.org as supplemental material). Mean eye position deviated by less than ±0.08° of visual angle between blocks when subjects attended to the left versus right location, and stability of fixation did not reliably differ between any of the tasks. We also evaluated whether eye movement signals might account for successful decoding of the attended stimulus orientation or color. Unlike cortical activity, eye-position signals failed to predict the attended orientation in any task [orientation decoding accuracy in the orientation and contrast tasks (experiment 1) or orientation and color tasks (experiment 2), 49 and 48%, or 50 and 50%, respectively; chance-level performance, 50%], nor were eye-position signals predictive of the attended color in either the orientation or color discrimination task of experiment 2 (color, 51%; orientation, 50%; chance-level performance, 50%). Therefore, it seems unlikely that the fMRI orientation decoding results might be attributed to eye movements.
Discussion
This study revealed distinct correlates of spatial and feature-selective attention, as measured with fMRI. Attending to a lateralized grating led to much stronger BOLD responses in early visual areas, but surprisingly, these facilitatory effects of spatial attention did not always result in stronger orientation- or color-selective responses to the attended stimulus. Instead, we found that attentional enhancement of feature-selective activity could be strongly modulated by the observer's task. In experiment 1, we found that cortical processing of orientation was enhanced when subjects attended to subtle changes in the orientation of the grating but not its contrast. In experiment 2, color-selective responses were enhanced by attention only when color was the task-relevant feature and not when the orientation of the grating had to be attended. In these experiments, spatial attention alone proved insufficient to facilitate feature-selective processing in early visual areas. Instead, the enhancement of orientation- and color-selective activity appeared to reflect the involvement of a feature-based attentional mechanism, which has been shown to operate globally across the visual field (Treue and Maunsell, 1996; Treue and Martínez Trujillo, 1999; Saenz et al., 2002; Martinez-Trujillo and Treue, 2004; Serences and Boynton, 2007). Enhancement of orientation- or color-selective activity was not confined to the attended stimulus location but rather was found to spread to distal locations, including the ipsilateral hemifield.
It should be noted that, in experiment 2, we found that performance of a color discrimination task led to attentional enhancement of both color- and orientation-selective responses. Why did we observe enhancement of a task-irrelevant feature in this instance? One possibility is that spatial attention led to the enhancement of both relevant and irrelevant features, but such an account does not adequately explain why no such enhancement occurred for color responses when subjects performed the orientation task in experiment 2, nor does it explain the task-specific improvement of orientation responses in experiment 1. Accounts of object-based attention suffer from similar difficulties, because these theories assume that attention directed to any feature of an object should automatically lead to the enhancement of all features of that object (Duncan, 1984; Blaser et al., 2000). Instead, the interpretation we favor is suggested by the behavioral strategy reported by our subjects of attending more to the high luminance portions of the grating when discriminating small changes in overall chromaticity. Such an attentional strategy seems plausible given that human observers are somewhat more sensitive to chromatic differences at higher luminance levels (Kaiser and Boynton, 1996). We speculate that this behavioral strategy resulted in enhanced feature-based processing of both color and orientation information. Regardless of the underlying behavioral strategy, we find positive evidence in two separate experiments that task relevance can modify the attentional enhancement of feature-selective responses.
Our findings run counter to the commonly held assumption that spatial attention automatically facilitates the processing of all features presented at the attended location (McAdams and Maunsell, 1999; Boynton, 2009; Ling et al., 2009; Reynolds and Heeger, 2009). Only a few neurophysiological studies have investigated this issue in detail. In a highly influential study, McAdams and Maunsell (1999) characterized the orientation tuning functions of individual neurons in area V4 while monkeys performed an orientation discrimination task involving one of two lateralized gratings. They observed strong modulatory effects of attention and suggested that spatial attention leads to an increase in the gain of the stimulus-driven response of the neuron. Such a gain increase should enhance the amount of feature-selective information carried by the neuron, at both the individual level and the population level (Seriès et al., 2004; Ma et al., 2006). However, here we varied the subject's behavioral task and found that spatial attention did not always lead to stronger feature-selective responses. For example, attending to orientation was important for observing stronger orientation-selective responses. Our results suggest that the modulation of orientation-selective responses found in previous studies might be, at least in part, attributed to the task relevance of orientation, and not to the effects of spatial attention alone.
Our results indicate that subjects are often capable of restricting their attention to a single task-relevant feature of a stimulus, contrary to theories of object-based attention. These theories assume that attending to one feature of an object should result in the automatic selection of the whole object, including its task-irrelevant features (Duncan, 1984; Blaser et al., 2000). Neural evidence in favor of this view comes from an fMRI study by O'Craven et al. (1999). Their stimuli consisted of an overlapping face and house, with one moving and the other stationary, and subjects had to monitor the identity of either the faces or houses. Attending to a moving face led to higher activity not only in face-selective areas but also in motion-sensitive area MT+, indicating enhanced processing of motion although this feature was not directly relevant to the observer's task. Why might attention have spread to task-irrelevant features in this study? One explanation is that motion, although not explicitly relevant to the observer's task, nonetheless served as a useful cue for distinguishing the moving object from the stationary one, thereby resulting in top-down enhancement of these implicitly relevant features as well. Perhaps consistent with this notion, in experiment 2, we found that attending to the color of a luminance-defined orientation grating led to enhancement of both color- and orientation-selective responses. Perhaps orientation served as an implicitly relevant cue for the selection of color signals in this situation, for the reasons described above. It will be interesting for future studies to investigate the experimental conditions that favor the selection of a specific task-relevant feature or an entire object for the purposes of performing a behavioral task.
We conclude that the task-specific enhancement of feature-selective processing found here cannot be readily explained in terms of spatial attention alone or object-based attention. Instead, it appears to reflect a strong contribution of feature-based attention (McAdams and Maunsell, 2000; David et al., 2008; Hayden and Gallant, 2009; Patzwahl and Treue, 2009). Previous studies indicate that, when observers must attend to one of two overlapping orientations or motion directions, feature-selective activity in early visual areas is biased in favor of the attended feature (Treue and Maunsell, 1996; Kamitani and Tong, 2005, 2006; Liu et al., 2007). Furthermore, attending to a feature at one location can lead to global biases across the visual field (Treue and Maunsell, 1996; Treue and Martínez Trujillo, 1999; Saenz et al., 2002; Martinez-Trujillo and Treue, 2004; Serences and Boynton, 2007). Consistent with these observations, we found information about attended orientations and attended colors not only in cortical regions corresponding to the attended stimulus but also in the opposite hemifield containing the unattended stimulus and in the foveal representation as well. Such task-related activity was found even when the attended stimulus differed in orientation from the unattended stimulus, indicating that this top-down orientation-selective signal can operate independently of the bottom-up input.
This study also revealed a dissociation between the strength of the BOLD response and the amount of information found in the pattern of BOLD activity. Spatial attention led to stronger BOLD responses in all three experiments, yet this was not automatically accompanied by improved strength of orientation- or color-selective activity. Thus, greater activity in the visual cortex does not necessarily indicate stronger feature-selective responses. Moreover, reliable information might sometimes be found in brain regions that show weak BOLD activity. A recent study demonstrated that information about an orientation maintained in working memory could be decoded during the blank delay period, even when overall activity in the visual cortex fell to baseline levels (Harrison and Tong, 2009). The relationship between BOLD amplitude and information contained in the activity pattern appears to be far from straightforward. Dissociations between these two measures are important to consider, especially given that most previous fMRI studies have focused exclusively on changes in BOLD amplitude as an index of specific cognitive or neural processes. An advantage of the fMRI decoding approach is its emphasis on measuring the amount of information contained in fMRI activity patterns (Kamitani and Tong, 2005; Haynes and Rees, 2006; Kriegeskorte et al., 2006; Norman et al., 2006), which could potentially reveal a different interpretation of the brain activity.
In conclusion, most theories of attention assume that spatial attention leads to enhanced processing of all stimulus features at the attended location, regardless of the task performed. However, the present results suggest that, although spatial attention enhances the overall BOLD response in the human visual cortex, this overall enhancement is not necessarily accompanied by improved encoding of stimulus features. Rather, feature-selective responses may be selectively enhanced according to their relevance to the observer's behavioral task.
Footnotes
This work was supported by a Rubicon grant from the Netherlands Organization for Scientific Research (J.F.M.J.) and National Eye Institute (NEI) Grants R01 EY017082 (F.T.) and P30-EY008126. We thank Ben Wolfe for technical assistance, the Vanderbilt University Institute of Imaging Science for MRI support, Sang Wook Hong for assistance with color calibration, and Jascha Swisher and Sam Ling for many helpful comments and discussions.
- Correspondence should be addressed to Janneke Jehee, Donders Institute for Brain, Cognition, and Behavior, Center for Cognitive Neuroimaging, Kapittelweg 29, 6525 EN Nijmegen, The Netherlands. janneke.jehee{at}donders.ru.nl
