Abstract
Selective attention is known to interact with perceptual organization. In visual scenes, individual objects that are distinct and discriminable may occur on their own, or in groups such as a stack of books. The main objective of this study is to probe the neural interaction that occurs between individual objects when attention is directed toward one or more objects. Here we record steady-state visual evoked potentials via electrocorticography to directly assess the responses to individual stimuli and to their interaction. When human participants attend to two adjacent stimuli, prefrontal and parietal cortex shows a selective enhancement of only the neural interaction between stimuli, but not the responses to individual stimuli. When only one stimulus is attended, the neural response to that stimulus is selectively enhanced in prefrontal and parietal cortex. In contrast, early visual areas generally manifest responses to individual stimuli and to their interaction regardless of attentional task, although a subset of the responses is modulated similarly to prefrontal and parietal cortex. Thus, the neural representation of the visual scene as one progresses up the cortical hierarchy becomes more highly task-specific and represents either individual stimuli or their interaction, depending on the behavioral goal. Attention to multiple objects facilitates an integration of objects akin to perceptual grouping.
SIGNIFICANCE STATEMENT Individual objects in a visual scene are seen as distinct entities or as parts of a whole. Here we examine how attention to multiple objects affects their neural representation. Previous studies measured single-cell or fMRI responses and obtained only aggregate measures that combined the activity to individual stimuli as well as their potential interaction. Here, we directly measure electrocorticographic steady-state responses corresponding to individual objects and to their interaction using a frequency-tagging technique. Attention to two stimuli increases the interaction component that is a hallmark for perceptual integration of stimuli. Furthermore, this stimulus-specific interaction is represented in prefrontal and parietal cortex in a task-dependent manner.
- attention to multiple objects
- electrocorticography
- perceptual integration
- prefrontal cortex
- steady-state visual evoked potential
Introduction
Selective attention is a top-down process that modulates neural activity depending on task demand and surrounding context (Kim and Verghese, 2012, 2014). Electrophysiological studies report that simultaneous attention to several isolated objects enhances corresponding responses in visual cortex (Müller et al., 2003; McMains and Somers, 2004; Morawetz et al., 2007; Drew et al., 2009; Niebergall et al., 2011; Itthipuripat et al., 2013; Störmer et al., 2014). Thus, attentional modulation of isolated objects is generally thought of as top-down modulation of individual stimulus representations. However, objects in daily life often occur in the presence of clutter, such as buildings in a city and vegetables in a grocery store. How is attentional selection of multiple objects implemented in such situations? Does attention highlight individual neural representations simultaneously without affecting the potential neural interaction between objects, or does attention modulate the neural interaction between targets? Furthermore, how does attention interact with perceptual integration? When two objects are in a configuration that allows perceptual integration, how does attention to one or both influence their representation? Previous studies show that, when adjacent stimuli are integrated into a perceptual whole, there is interaction between the representations of individual stimuli (Zemon and Ratliff, 1984; Victor and Conte, 2000; Aissani et al., 2011; Tsai et al., 2012; Gundlach and Müller, 2013; Boremanse et al., 2014). Does attentional selection override perceptual integration when one of the objects is selected, and does it allow perceptual integration when both objects are selected?
Furthermore, we examine how attentional selection modulates the degree of interaction. The “executive control” of information and behavior and attention allocation is known to depend critically on prefrontal and parietal cortices (Duncan, 2010). To determine how these higher-level areas modulate responses depending on the task, we use electrocorticography (ECoG) to record responses in prefrontal, parietal, and visual cortex in humans. To investigate the neural correlates of attending simultaneously to multiple stimuli, we use steady-state visual evoked potentials (SSVEPs) elicited by periodic, flickering stimuli. The SSVEP paradigm produces evoked responses to a train of stimulus presented at a fixed rate. When two stimuli flicker at frequencies F1 and F2, the SSVEPs consist of harmonics of the driving frequencies (mF1 and nF2 where m, n are integers), or “self-terms,” and intermodulation (IM) components (mF1 ± nF2). In particular, the IM components represent nonlinear interactions between the two input frequencies and can only arise from neural mechanisms that receive input from both frequencies and only if they interact in a nonlinear fashion (Regan and Regan, 1988; Norcia et al., 1999; Victor and Conte, 2000). Thus, IM response can serve as a direct and unequivocal marker of neural interaction. This technique allows us to disentangle responses to individual objects and to their interactions, an analysis not amenable to traditional neurophysiology and neuroimaging approaches that measure aggregate responses (Desimone and Duncan, 1995; Beck and Kastner, 2005).
Our data show a robust representation of differential stimulus-driven components in prefrontal and parietal area under different attentional demands. Here, only the first harmonic of the attended stimulus is evident when attention is directed to one flickering stimulus, and only the nonlinear neural interaction term is evident when attention is deployed to both flickering stimuli. In contrast, early visual areas show all stimulus-driven components with differential enhancement of the attended component. We demonstrate that the integration of two stimuli is enhanced when they are both attended, and that prefrontal and parietal cortices preferentially manifest this combined representation. Thus, we show that, in addition to representing task-relevant stimuli, prefrontal and parietal cortex plays an important role in the parsing of visual information.
Materials and Methods
Observers.
A total of 6 patients (1 male and 5 females) participated in our study; data from one female observer were not analyzed due to excessive eye movements. All observers underwent placement of intracranial electrode arrays to localize seizure foci before surgical treatment of medically refractory epilepsy. The arrays were typically in place for 3–7 d, with the location of the electrodes and duration of implantation determined independently by clinical needs alone. None of these patients had photosensitive epilepsy or had a seizure focus in the occipital lobe. In general, antiseizure medications were withheld as part of the inpatient monitoring to induce seizures. Patients were given medications for postoperative pain or benzodiazepines as needed for seizures. We waited until the acute effects of these medications wore off, and patients felt that they were at baseline, before proceeding with experiments. Additional characteristics of the subjects are shown in Table 1. Institutional review boards at University of Washington approved the experimental protocols, to which the participants consented. All participants had normal/corrected-to-normal visual acuity.
Information for each patient
Display and visual stimuli.
Stimuli were presented on a CRT monitor (Dell P780 Ultrascan) placed 60 cm in front of the patient. Monitor refresh rate was 100 Hz. The CRT monitor was brought in to the patient's room on a mobile platform. The stimuli consisted of a pair of wedge-shaped gratings flickering at two different frequencies (one at 16.7 Hz and the other at 12.5 Hz for Participants 1, 2, and 3; 8.33 and 3.33 Hz for Participants 3 and 4) whose contrast was modulated on-off in a square-wave fashion, and two static wedge-shaped gratings located diametrically opposite the flickering wedges. Each wedge was a portion of a concentric grating pattern. We used two sets of flickering frequencies (16.7 and 12.5 Hz; 8.33 and 3.33 Hz) to examine whether faster and slower sets of flickering frequencies differentially drive networks of brain areas that span different spatial scales. Wedges extended from 2° to 8° eccentricity radially and covered 40° in polar angle. Concentric gratings had a spatial frequency of 2 cycles per degree and were presented in the visual field contralateral to electrode placement. We chose to present stimuli in either the upper or lower quadrant depending on the location of the posterior electrodes. If a majority of the posterior electrodes were below the calcarine fissure, we chose to present stimuli in the upper quadrant. In separate blocks, we explored two stimulus configurations with gratings either within the same quadrant or within the same hemifield. Participants took part in both configurations (or blocks), as they were able (for a patient who was able to participate in both, see Fig. 3). All flickering stimuli were phase-locked to cue onset. At cue onset, a photodiode detected a 10 ms flash of a small white square in the top left corner of the monitor and sent a time marker to the g.USBamp amplifier (Guger Technologies). The photodiode and the white square patch were covered so that the flashes were not visible to the participant. In a separate control experiment, two superimposed vertical gratings, with the same spatial frequency (2 cpd) and spatial phase, were modulated at 16.7 and 12.5 Hz, respectively, and shown at the center of the screen in a square window of 6.5 deg. The α transparency levels were set at 50% so that both gratings were visible. Two patients (Participants 1 and 5) participated in the control experiment. All stimuli were delivered by using Psykinematix (Kybervision Japan). For a summary of experimental conditions for each participant, see Table 2.
Experimental conditions tested for each participant
Experimental procedure.
The observer initiated each trial by a button press. At the start of each trial, cues (red lines) pointing to individual elements indicated the stimuli to be attended. A block of trials consisted of four interleaved conditions: attend to both flickering stimuli (see Fig. 1, attend-both), to both static stimuli (see Fig. 1, attend-away), and to each of the two flickering stimuli (see Fig. 1, attend-one; attend-F1 or attend-F2). To avoid transients due to stimulus onset, the stimuli appeared 0.48 s before the cue for the pair of gratings flickering at 16.7 and 12.5 Hz (0.6 s for the pair of gratings flickering at 8.33 and 3.33 Hz). During the trial, which lasted 2.4 s for the pair of gratings flickering at 16.7 and 12.5 Hz (3 s for the pair flickering at of 8.33 and 3.33 Hz flickering gratings), the observer attended to the cued stimuli to detect a brief contrast decrement that occurred at a random time on the cued stimuli while maintaining fixation at the central point (for a schematic of the timeline of the tasks, see Fig. 1a). At the end of a trial, participants responded with joystick by pressing the left button if they detected a contrast change and the right button if they detected no contrast change. The cue was a 100% valid (i.e., the contrast change never occurred in a noncued stimulus). For the attend-both and attend-away conditions, the contrast decrement occurred on one of two cued stimuli, selected at random. Half of total number of trials had a contrast decrement and the other half did not have a contrast decrement. The contrast decrement was tailored to individual subjects. Before the main experiments, the subject ran several practice trials so that she or he was able perform the task correctly some of the time. Because of time constraints, we did not measure the contrast threshold precisely but intentionally leaned toward making the task more rather than less difficult. The mean contrast decrement values across observers were −28% for faster (8.33 and 16.7 Hz) and −31% for slower (3.33 and 12.5 Hz) frequencies, and ranged from −13% to −18% for the static stimuli. Each experimental session consisted of as many blocks as the observer was able to participate. In each block, we tested a single configuration for the stimuli (e.g., same hemifield or same quadrant). Each block had 80 trials, with 20 trials for each of the four cue conditions. Eye movements were monitored throughout the trial with horizontal EOG electrodes attached to the side of each eye to ensure that the observer maintained fixation. We used criteria from the FASTER (Fully Automated Statistical Thresholding for EEG Artifact Rejection, http://www.mee.tcd.ie/neuraleng/Research/Faster) package to eliminate ECoG channel and epoch artifacts. We also monitored an HEOG channel to reject trials with horizontal eye movement. On average, 10% of the trials were rejected. These trials included ones with and without the contrast decrement, regardless of whether the observer responded correctly.
Electrocorticographic recording.
Platinum electrode arrays (Ad-Tech) were configured as combinations of a rectilinear “grid” (4 × 8 or 5 × 8 or 8 × 8) and strip arrays (2 × 8). The electrode contacts had 4 mm diameter (2.3 mm exposed), 1 cm interelectrode distance, and were embedded in SILASTIC. These arrays were surgically placed on the subdural brain surface. Reference and ground electrodes were on the scalp, close to the midline and slightly contralateral to the site of craniotomy. ECoG signals were physically split into two identical sets. One set was fed into the clinical system and the other set was recorded using g.USBamp amplifiers (Guger Technologies) at 1.2 kHz sampling frequency. ECoG signals were bandpass-filtered from 0.1 to 500 Hz using general-purpose BCI software (Schalk et al., 2004). Electrodes containing epileptiform activities were excluded from further analyses.
Cortical rendering and electrode localization.
The relationship between electrode position and gyral anatomy was determined by first rendering the cortical surface from a preoperative MRI, using Freesurfer software (https://surfer.nmr.mgh.harvard.edu). Then electrode positions were calculated with respect to the preoperative MRI from postoperative computational tomography using Bioimage Suite software (http://www.bioimagesuite.org).
Signal analysis.
Response waveforms of individual trials were epoched from cue onset to the end of trial at all ECoG electrodes and averaged over trials. This averaged waveform was converted to the frequency domain via the discrete Fourier transform. The signal-to-noise ratios (SNRs) of SSVEP components were computed by dividing peak amplitudes by the associated noise, which is defined for a given frequency f by the average amplitude of the two neighbor frequencies (i.e., f − δf and f + δf) where δf gives the frequency resolution of the Fourier analysis, which was 0.4167 Hz for the pair of gratings flickering at 16.7 and 12.5 Hz (Participants 1, 2, 4, and 5) and 0.3333 Hz for the pair of gratings flickering at 8.33 and 3.33 Hz (Participants 3 and 4) in our studies. In the main attention experiment with gratings flickering at 16.7 and 12.5 Hz, we chose three SSVEP components (F1 = 16.7 Hz, F2 = 12.5 Hz, and F1 − F2 = 4.17 Hz) for further analysis because these components showed the most robust activation. Only a couple of electrodes in posterior area showed the second harmonics (2F1 = 33.3 Hz and 2F2 = 25 Hz), and no electrode showed the sum component (F1 + F2 = 29.2 Hz). The sum component was only seen in the passive viewing condition when the two flickering gratings overlapped completely (see Fig. 7). The dominant first-harmonic component is consistent with previous reports that on-off flicker generates stronger first-harmonic responses, whereas counter-reversal flicker generates stronger second-harmonic responses (Di Russo et al., 2007; Kim et al., 2011). To determine whether SSVEP responses and their modulation depend on the range of flicker frequency, we tested lower frequencies (8.33 and 3.33 Hz). In these experiments, the same SSVEP components (F1 = 8.33 Hz, F2 = 3.33 Hz, and F1 − F2 = 5 Hz) were chosen to be consistent with the analyses in the main experiment with the pair of gratings flickering at 16.7 and 12.5 Hz.
To assess the presence of the three SSVEP components of interest in any channel, we used a combination of SNR (Meigen and Bach, 1999) and Tcirc2 statistic (Victor and Mast, 1991) criteria. The SNR allowed for direct comparison of trial-averaged data across subjects. The Tcirc2 statistic used both Fourier amplitude consistency and Fourier phase consistency across individual trials to assess whether stimulus-driven activity significantly differed from noise. Thus, it provided a powerful statistic for identifying significant SSVEP above background EEG. First, all channels with SNR > 3 in any of the three SSVEP components of interest were identified. Second, the Tcirc2 statistic was calculated for each SSVEP component at each identified channel.
For each attentional condition, channels that manifested significant (Bonferroni corrected) SSVEP response in any of the three components were then considered to have significant response and selected for further analysis.
We analyzed the attentional modulations of SSVEP components for each experimental condition (attend-both or attend-one [F1 or F2]). For statistical power, all channels that passed statistical criteria described above were pooled across observers. We separately analyzed data from channels located in prefrontal, parietal, and occipital regions as determined by surface anatomy.
We performed a planned linear trend analysis on the patterns of the three SSVEP components to examine how they were comodulated by attention (Howell, 1997; Myers and Well, 2003; Cardinal and Aitken, 2006). A linear contrast is defined as a linear combination of each component's mean SNR above the baseline taken as the response in the “attend-away” condition across all pooled channels in a cortical area of interest. Each mean μj has a weight wj as follows:
such that wF1−F2 + wF1 + wF2 = 0.
We performed trend analysis within experimental conditions and within spectral components separately. First, we chose one set of linear contrasts to compare the spectral components within each attention condition: the attended components versus the remaining components in the “attend-both” (see Fig. 6a) and “attend-one” (see Fig. 6b). Thus, we fit the SNR using weight values 2 for the attended component and −1 for the remaining two components as follows:
Specifically, when attention was directed to both flickering stimuli, we tested the prediction that F1 − F2 was modulated more than F1 or F2 (see Fig. 6a). When attention was directed to one component, we tested the prediction that the attended component was modulated more than the ignored component or the IM term F1 − F2 (see Fig. 6b). For statistical testing, the F statistic was defined as Fϕ = 
, where MSψ is the mean squared error of the contrast (with degree of freedom of 1) and the denominator is the residual error. The residual error was obtained from a repeated-measures ANOVA with degree of freedom set to (A − 1) × (N − 1), where A is the number of groups and N the number of channels.
Analogously, we chose a second set of linear contrasts to evaluate how attentional manipulation affected specific spectral components: the IM (see Fig. 6c) and the first harmonic responses (see Fig. 6d). For example, linear contrast was defined using weight values of 2 for the “attend-both” and −1 for the “attend-F1” and “attend-F2” responses in the case of the IM response.
We additionally performed an analysis using a more restricted subset of occipital channels (see Fig. 6, fourth triplet) that manifested the same set of SSVEP components as seen in the prefrontal and parietal areas. The rationale for this approach was to determine whether the occipital area manifested a similar response selectivity as the higher areas.
Results
We elicited SSVEPs by using wedge-shaped gratings flickering at two different frequencies (for the frequencies used, see Table 1) and two static wedge-shaped gratings located diametrically opposite the flickering wedges. To emulate the real world where similar objects tend to be nearby, we used two adjacent flickering gratings that appeared simultaneously. Because the electrode locations were determined solely on clinical grounds, we tried two configurations to obtain good responses, with the gratings either within the same quadrant or within the same visual hemifield (in separate blocks). Four types of cues were used to direct the participant's attention: to both flickering stimuli (Fig. 1, attend-both), to one of the two flickering stimuli (Fig. 1, attend-one; attend-F1 or attend-F2), or to both static stimuli that served as a baseline (Fig. 1, attend-away).
Attentional modulation of SSVEPs in Participant 1 (P1) to stimuli presented in the bottom right quadrant. a, Timeline of each trial. Two wedge-shaped flickering gratings (F1 flickered at 16.7 Hz, F2 flickered at 12.5 Hz) and two wedge-shaped static gratings (S1, S2) were presented simultaneously. A cue at fixation indicated the stimuli to attend to. Participants were asked to detect a brief contrast decrement that appeared at a random time on the cued stimulus with 50% probability. The four cues were as follows: attend to static wedges (attend-away), to both flickering wedges (attend-both), or to one flickering wedge (attend-F1 or attend-F2). b, Black dots indicate ECoG electrode locations in P1. c–f, Amplitude spectra of responses in the attend-away, attend-both, and attend-F1 and attend-F2 conditions, respectively. First and second columns represent modulations of SSVEPs from a representative prefrontal and an occipital channel, respectively (yellow circles). Red and green bars represent components related to the attended and ignored stimuli, respectively, in that condition. In the attend-both condition, the nonlinear interaction term at the difference frequency (F1 − F2 = 4.17 Hz) was significant (by SNR criteria [SNR > 3] and Tcirc2 statistic) in both prefrontal and occipital channels. In the attend-one condition, attended first harmonic responses were significant only in the occipital channel.
Behavior analysis
Table 3 shows behavioral data in each of the four attention conditions. Proportion correct (hits + correct rejections) was highest when observers attended to the contrast decrements on the slower frequency grating, compared with when they attended to decrements on the higher frequency grating (t(6) = 6.481, p = 0.0006). However, the performance for these two conditions was not significantly different from that for the attend-both condition. Importantly, we showed that, in the attend-both condition, observers attended to both gratings as cued, and not just to a single grating. An estimate of the true probability of detection for each grating in the attend-one condition (after correction for guessing) shows that the observed performance in the attend-both condition was much better than if the observer had attended to only the faster grating (t(6) = 6.835, p < 0.0007) or to the slower grating (paired t test; t(6) = 5.198, p < 0.002).
Behavioral performance data for each patient: proportion correct
Attend-both condition
Clear-cut SSVEPs are illustrated in two representative occipital and prefrontal channels for different attentional conditions (Fig. 1c–f). The first harmonic responses (F1 = 16.67 Hz and F2 = 12.5 Hz) and the difference IM response (F1 − F2 = 4.17 Hz) were most robust and were the focus of our subsequent analysis. The presence of SSVEP in the prefrontal channels was not due to volume conduction because their response phases differed from those of the occipital channels (Watson–Williams test, F(1,278) = 86.1, p < 10−10). The occipital channel shows clear attentional modulation of all three components (the two first harmonics and the IM term). When attention is directed to both flickering stimuli, the two first harmonic responses as well as the IM response are significant. Attending to one of two flickering stimuli produces a significant first harmonic at the flicker frequency corresponding to the attended stimulus. Furthermore, whereas the occipital channel exhibited prominent IM response regardless of which stimuli were attended, the prefrontal channel showed a significant IM response only when the participant attended to both stimuli. Figure 2 shows data on other prefrontal and occipital electrodes that showed significant activity, demonstrating that electrodes in prefrontal cortex typically exhibit IM responses in the attend-both condition, while occipital electrodes show both first harmonic and IM terms.
Amplitude spectra for other electrodes that manifested significant response in at least one of the three SSVEP components of interest F1, F2, or F1 − F2 (indicated by an asterisk above the spectra) for Participant 1. Each vertical column of panels represents the spectra for each of the four attention conditions on a particular electrode: attend-away, attend-both, attend-F1 and attend-F2. Note the different scales of the y-axis.
We identified the effect of selective attention on the first harmonic responses and IM response in all recorded channels. Significant response at each frequency of interest was labeled at channel locations rendered on the individual anatomy of four participants (see Figs. 3⇓–5, 7). Across the brain, a pattern of responses emerged that became progressively more specific going up the hierarchy. In early visual areas, attending to both stimuli elicited strong first harmonic responses (Fig. 3, green labels), or first harmonic responses and one IM response (Fig. 3, yellow labels). In contrast, prefrontal and parietal areas only manifested a significant IM response (Fig. 3, red labels), consistently seen in multiple observers.
ECoG channels with significant SSVEP components in the attend-both condition for Participants 1–3. Channels were classified by patterns of significant activity across the three steady-state components. Colors represent different combinations of components. a–d, Stimulus configuration with flickering wedges in right and left visual hemifield for P1, P2, and P3, respectively. e–l, Channels with significant SSVEP responses in different views for P1, P2, and P3, respectively.
Attend-one condition
In contrast to the attend-both condition where prefrontal and parietal cortex showed a clear IM response, the attend-one condition yielded a less robust first harmonic response to the attended stimulus (F1 = 16.67 Hz or F2 = 12. 5 Hz) in these areas (Fig. 4, circled green labels). While both frequencies elicit strong first harmonic responses in early visual areas (Fig. 4, circled green and yellow labels), the response in prefrontal and parietal areas appeared to decline with increasing stimulus frequency, across participants. This trend of decreasing activation of prefrontal and parietal cortex as flicker frequency increases raises the possibility that higher frequency responses are attenuated in these regions. To investigate this possibility, we ran the experiments using lower flicker frequencies 8.33 and 3.33 Hz in two participants (Participants 3 and 4). Participant 3 had ventral coverage of visual areas, as well as prefrontal and parietal cortex (Fig. 4d,h). Participant 4 had no coverage of visual areas but had electrodes covering prefrontal and parts of parietal cortex (Fig. 5). With these lower stimulation frequencies, we found that the first harmonic responses (F1 = 8.33 Hz and F2 = 3.33 Hz) and IM response (F1 − F2 = 5 Hz) appeared in prefrontal and parietal regions (Fig. 5). Importantly, these stimulus-driven SSVEP components appeared differentially depending on the attention condition. In prefrontal and parietal regions, attention to one of two stimuli resulted in one first harmonic response at the flicker frequency of the attended stimulus and attention to both stimuli resulted in only the IM response. This suggests that low-pass temporal frequency filtering likely contributed to the absence of the first harmonic responses at higher stimulus frequencies (especially 16.67 Hz) in prefrontal and parietal regions in the attend-one condition. It is possible that polysynaptic communication (Breakspear et al., 2010; Bridwell and Srinivasan, 2012) across distant brain areas causes this high-frequency steady-state ECoG component to be less phase-locked across trials for sustained periods so that high-frequency steady-state ECoG components are localized to posterior visual areas whereas only lower frequency components are seen in parietal and prefrontal areas. However, it is important to note that a low-pass filter alone cannot explain the selective representation of only one of the three SSVEP components when slower flicker frequencies (8.33 and 3.33 Hz) were used.
ECoG channels with significant SSVEP components in the attend-one (F1 or F2) conditions for Participants 1–3. Channels were classified by patterns of significant activity across the three steady-state components for the corresponding attention and stimulus configuration depicted in the first row. Green circles indicate significant first harmonic component specific to the attended stimulus. a–d, Stimulus configuration with flickering stimuli in right and left visual hemifield and attention directed to F1. e–h, Stimulus configuration with flickering wedges in right and left visual hemifield and attention directed to F2.
ECoG channels with significant SSVEP components in the attend-one wedge (attend to flickering frequencies F1 = 8.33 Hz or F2 = 3.33 Hz) and attend-both conditions for Participant 3. This participant has no occipital lobe coverage. Flickering wedges were presented either in the right hemifield spanning the horizontal meridian or in the right lower quadrant. Channels with significant SSVEP responses in prefrontal and parietal areas are marked in color. a, Attend-F1 (right hemifield). b, Attend-F2 (right hemifield). c, Attend-both (right hemifield). d, Attend-F1 (bottom right quadrant). e, Attend-F2 (bottom right quadrant). f, Attend-both (bottom right quadrant).
Task specificity of responses
To systematically determine whether attention differentially modulates the two first harmonic and the IM responses, we performed a planned linear trend analysis of these three SSVEP components. Attentional modulation of each component was quantified as SNR difference by subtracting a baseline, wherein attention was directed toward a pair of static gratings presented at the diametrically opposite location (the “attend-away” condition). The results did not change when SSVEP amplitudes at flickering frequencies were used instead of SNRs (data not shown). Figure 6 summarizes the results, grouped by anatomic location, pooling all channels that showed any significant SSVEP responses from 4 participants. We did not include the temporal lobe in the analysis because only a small number of temporal channels were significant in 1 subject (Fig. 3). The linear contrast used for each trend analysis was denoted by a symbol above each triplet (see Materials and Methods). When observers attended to both flickering stimuli, the IM response in prefrontal and parietal cortex was enhanced, but the first harmonic responses were unchanged (Fig. 6a). Trend analysis identified a significant difference between the IM and the mean of the two first harmonic responses (F(1,40) = 34.7, p < 10−6 in prefrontal area; F(1,28) = 39.97, p < 10−6 in parietal cortex), whereas occipital channels showed no such pattern. Even when we consider a subset of occipital channels that showed the same modulated response (IM) as prefrontal and parietal regions, there was no significant trend (F(1,24) = 2.40, not significant). These findings indicate that attention to two nearby stimuli selectively enhances the IM response in prefrontal and parietal cortex but is much less selective in early visual areas, where only a subset of channels show significant IM enhancement.
Attentional modulation of SSVEP components (first harmonic and IM responses) within an experimental condition for the attend-both (a) and attend-one (b) conditions. y-axis indicates the mean SNR difference between attend-both (a) or attend-one (b) versus attend-away, pooled across the number of channel indicated. In each plot, the first three triplets of bars included all channels with any significant SSVEP components in prefrontal, parietal, and occipital cortex. The fourth triplet is a subset of occipital channels that manifested the same components as the prefrontal and parietal areas. A linear trend analysis examined whether (a) the IM response alone was significantly modulated compared with the two first harmonic responses in the attend-both condition and (b) whether the attended first harmonic response alone was significantly modulated compared with the IM response and ignored first harmonic response in the attend-one condition. The icon (
or 
) shape above each set of three bars indicates the linear trend that was tested. Asterisk indicates whether the trend was significant. To analyze the same data across the three attention conditions, we performed a similar trend analysis for a specific SSVEP component: IM response in c and the first harmonic response in d, keeping the same channel selection as in a and b, respectively. Here the trend analysis in c determines whether the IM response was modulated more in the attend-both relative to the attend-one condition, and in d whether the attended first harmonic response was modulated more when it was attended than when it was ignored or when both stimuli were attended.
The specificity of attentional modulation also was present when attention was directed toward one stimulus (Fig. 6b). In this case, prefrontal and parietal areas showed a selective enhancement of the “attended” relative to the ignored first harmonic response and the IM response (F(1,40) = 8.02, p < 0.007 in prefrontal area, F(1,36) = 47.13, p < 10−6 in parietal area), whereas occipital channels as a whole did not show such a pattern (Fig. 6b, third triplet). When we considered only occipital channels with significant attended first harmonic response, there was a significant difference between the attended first harmonic response and the other two components (F(1,160) = 7.42, p < 0.02; Fig. 6b, fourth triplet). Our results are consistent with recent studies demonstrating a target-specific response to an isolated stimulus in prefrontal and parietal cortex (Stokes et al., 2013; Mendoza-Halliday et al., 2014; Ester et al., 2015). Moreover, attention to two abutting stimuli enhances only the neural interaction between the two stimuli without any first harmonic responses in prefrontal and parietal cortex.
The preceding analysis revealed a specific pattern of attention modulation across SSVEP components. We next examined whether the pattern persisted when we analyzed the same data within a component across conditions. We tested whether the IM response is selectively enhanced during the “attend-both” condition and whether the first harmonic response is selectively enhanced during the “attend-one” condition. The IM response in prefrontal and parietal, but not in the occipital cortex, was enhanced when observers attended to both flickering stimuli (Fig. 6c). Trend analysis identified a significant difference in the IM response between attend-both and attend-one conditions (F(1,40) = 45.5, p < 10−7 in prefrontal area; F(1,28) = 64.6, p < 10−8 in parietal cortex), whereas occipital channels showed no such significant difference (F(1,82) = 0.6649, not significant in the third triplet; F(1,24) = 1.4275, not significant in the fourth triplet). On the other hand, the first harmonic response in prefrontal and parietal cortex was enhanced when observers attended to the corresponding stimulus (Fig. 6d; F(1,40) = 41.9, p < 10−6 in prefrontal area; F(1,36) = 27.5, p < 10−5 in parietal cortex), but not in the occipital area (F(1,192) = 0.546, not significant in the third triplet; F(1,160) = 0.7331, not significant in the fourth triplet in Figure 6d). These results confirmed a double dissociation between the experimental condition and SSVEP response in the prefrontal and parietal, but not the visual cortex.
Attention and IM in prefrontal cortex
We examined whether IM response in prefrontal cortex arises merely from an interaction between two adjacent stimuli. We performed a control experiment using overlapped gratings (flickering at 16.7 and 12.5 Hz), which generate strong interaction between the two gratings. In two participants (Participants 1 and 5), the sum IM response (F1 + F2 = 29.2 Hz) is localized to posterior visual areas (Fig. 7, pink and cyan labels), whereas the difference IM response (F1 − F2 = 4.17 Hz) appears in both visual areas as well more anteriorly (e.g., the temporal pole; Fig. 7, red labels), consistent with low-pass filtering in these higher level cortical areas. Critically, overlapped flickering gratings did not show any difference IM response in prefrontal (and parietal) cortex. This result strongly suggests that the appearance of difference IM response in prefrontal cortex is not simply stimulus-driven but rather a response to selective attention to two behaviorally relevant stimuli.
ECoG channels with significant SSVEP components responding to passive viewing of two overlapped gratings for Participants 1 and 5. a, Stimulus configuration with two overlapped flickering gratings at the middle. The two gratings flickered at frequencies F1 and F2 (16.7 and 12.5 Hz, respectively). b–e, Red, pink, and cyan dots indicate channels with significant difference IM response F1 − F2, sum IM response F1 + F2, and both responses, respectively, for Participant 1 (b, d) and Participant 5 (c, e). For both participants, the sum IM response is mostly localized to the posterior area while the difference IM response appears progressively from posterior to temporal areas.
Discussion
To date, neural mechanisms for attending to multiple objects has been studied largely with targets that are isolated from nontargets either in time or in space (Müller et al., 2003; McMains and Somers, 2004; Morawetz et al., 2007; Drew et al., 2009; Niebergall et al., 2011; Itthipuripat et al., 2013; Mendoza-Halliday et al., 2014; Störmer et al., 2014; Ester et al., 2015). Here we show that the IM component of SSVEPs in prefrontal and parietal cortex is a singular and specific marker of simultaneous attention to adjacent stimuli in a stimulus configuration conducive to perceptual grouping. Specifically, cortical SSVEPs recorded from parietal and prefrontal areas show a selective increase in IM response when two stimuli are simultaneously attended, in contrast to occipital responses, which are less selective. This progression of response selectivity is also observed when one stimulus is attended. Two critical observations relate to this finding. First the high signal fidelity of ECoG facilitated the observation of SSVEPs outside of posterior visual areas, which are not typically reported in human EEG/MEG studies by (for exceptions, see Perlstein et al., 2003; Aissani et al., 2011; Baldauf and Desimone, 2014). Second, measurement of the activity corresponding to nonlinear interaction is crucial to fully describe the neural correlate of attention to multiple objects, rather than simply tracking the representation of each object separately (Müller et al., 2003; McMains and Somers, 2004; Morawetz et al., 2007; Drew et al., 2009; Niebergall et al., 2011; Itthipuripat et al., 2013; Störmer et al., 2014).
What is the origin of the SSVEP IM response in prefrontal and parietal cortex? The pattern of IM response in these cortical areas and their dependence on attention task cannot be attributed simply to bottom-up processes. First, while loss of coherence from propagation in the visual pathway appears to attenuate higher frequency activities, this effect alone does not account for the modulation of SSVEP by manipulation of attention task. Using a set of lower stimulus-tagging frequencies, we observed selective modulation of the IM response in attend-both and of the first harmonic responses in attend-one conditions, respectively (Fig. 5). This result demonstrates that high-frequency attenuation does not account for the response selectivity in prefrontal and parietal cortex. In addition, the absence of IM response in the prefrontal cortex during passive viewing of overlapped gratings argues against a simple low-pass filtering explanation (Fig. 7). Second, the appearance of the IM response in parietal and prefrontal areas might be due to the larger neuronal receptive size of these areas, which allow greater potential for interaction between the two stimuli than a smaller V1 neuron. A previous human ECoG study (Yoshor et al., 2007) that used electrodes of a similar size to ours (2.3 mm in diameter), estimated the receptive field size of an electrode in early visual cortex to be <1° in the fovea and larger in the periphery. Thus, an individual “wedge” in our stimulus probably was larger than the coverage of a single occipital electrode, and few of these electrodes might have had receptive fields overlapping both wedges. It is plausible that larger receptive fields in prefrontal and parietal cortex would be more likely to encompass both wedges, allowing the possibility of nonlinear interactions. However, this increase in receptive field size alone cannot explain how the SSVEP response to identical stimuli depends on the attention condition. Instead, we suggest that task-specific SSVEP response in prefrontal and parietal cortex reflects both stimulus-driven activity and goal-oriented processing as we will discuss below. Because the IM response can only arise from nonlinear interaction between two inputs, the presence of the IM response in the absence of the first harmonic responses in prefrontal and parietal cortex implies that SSVEP components are propagated from other brain regions to prefrontal and parietal cortex, where the first harmonic components are gated out. The greater degree of response selectivity in these regions compared with early visual areas is consistent with the known roles of prefrontal and parietal lobe in the attention network (Suzuki and Gottlieb, 2013; Jacob and Nieder, 2014).
Previous SSVEP studies have shown that the IM response is shaped by both mid-level processes, such as perceptual binding (Aissani et al., 2011; Gundlach and Müller, 2013; Boremanse et al., 2014) and low-level processes, such as lateral spatial interaction (Zemon and Ratliff, 1984; Victor and Conte, 2000). The selective representation of specific SSVEP components in prefrontal and parietal cortex is likely affected by attention as well as stimulus configuration. Attention becomes more target-selective and excludes a contiguous irrelevant surround when target location is known in advance (Boehler et al., 2009; Kim and Verghese, 2012). Conversely, attention spreads within a perceptually linked texture especially when target location is unknown (Duncan, 1984; He and Nakayama, 1995; Driver and Baylis, 1998; Valdes-Sosa et al., 2000; Driver et al., 2001; Kim and Verghese, 2014). In our paradigm, when the contrast change occurs at a single cued location, attention is narrowly focused to the cued stimulus, resulting in a selective enhancement of the response to the target stimulus. When a contrast change occurs randomly in one of two locations, the observer needs to monitor both locations as a whole rather than separately, a strategy that appears to enhance the integrated signal from both stimuli over separate signals from individual stimuli. Thus, the perceptual integration of the two cocircular gratings due to configuration and task demand may contribute to generation of IM response.
The increase in the specificity of attentional modulation of SSVEP components from occipital to prefrontal and parietal regions clearly demonstrates that prefrontal and parietal cortex encode stimulus-specific responses relevant to the attentional task. Two recent studies of working memory provide clear evidence that prefrontal and parietal cortex encode stimulus-specific information (Mendoza-Halliday et al., 2014; Ester et al., 2015). Mendoza-Halliday et al. (2014) showed that, in a match-to-sample task, neurons in primate lateral prefrontal cortex had a sustained activation to the test during the delay period between test and sample. The other study using fMRI (Ester et al., 2015) showed that human prefrontal cortex has a similar decoded representation during a memory task for a particular orientation as it does during stimulation with that orientation. However, the results of these studies were based upon representations of the target feature presented in relative temporal or spatial isolation from the distractor stimulus. A drawback of this approach of decoding simple visual features or spatial positions of an isolated stimulus is that it does not capture a common situation in daily life where multiple similar objects are in proximity. Particularly, when attention is simultaneously deployed to multiple nearby objects, it is not clear what pattern of stimulus-specific responses will be represented in prefrontal and parietal cortex. Our results for attending to multiple objects are the first to show that prefrontal and parietal areas encode exclusively the interaction component when two adjacent stimuli are simultaneously attended, while the early visual areas represent both the individual stimuli and their interaction. These results are consistent with the interpretation of a recent neural pattern classification study in monkey prefrontal cortex that initial stimulus-specific population responses evolved to different final decision-related states depending on the current task demand (Stokes et al., 2013).
There are known limitations to ECoG studies in epilepsy subjects. First, the placement of electrodes is not under experimental control and may not cover all areas of interest. Second, the physical and cognitive state of the patients may vary, although the behavioral performance (Table 3) was similar in our subjects. Third, patients with focal epilepsy may have atypical brain organization related to the seizure focus. We tried to minimize this issue by discarding data from sensors that recorded epileptiform activities. Despite these limitations, studies in this population have yielded results consistent with the literature, including that on selective attention (Gunduz et al., 2012). Beyond sensory cortices, complex sensory features are integrated in associative areas of prefrontal and parietal cortex. Because these areas receive afferent connections from multiple areas of sensory association, prefrontal and parietal cortex, they can play a critical role in the organization of sensory stimuli. Therefore, it is possible that prefrontal and parietal regions carry information about what is currently relevant by combining stimulus-specific responses and contextual rules (Stokes et al., 2013). This unified system for the representation of sensory information and the processes of selecting, transforming, and using that information might allow us to flexibly and stably deal with a dynamic external environment.
Footnotes
This work was supported by National Science Foundation Grant 0963914 to P.V., National Institutes of Health Grant EY020876 to J.J.T., and Institute for Basic Science Grant R001-D1. We thank Jonathan D. Victor and Jonathan Winawer for discussion; and Kurt Weaver, Lise Johnson, Jeremiah Wander, and Wonsuk Kim for technical assistance.
The authors declare no competing financial interests.
- Correspondence should be addressed to Dr. Yee-Joon Kim, Center for Cognition and Sociality, Institute for Basic Science, Daejeon 34141, Republic of Korea. joon{at}ibs.re.kr
