Abstract
Perceptual difficulty is sometimes used to manipulate selective attention. However, these two factors are logically distinct. Selective attention is defined by priority given to specific stimuli based on their behavioral relevance, whereas perceptual difficulty is often determined by perceptual demands required to discriminate relevant stimuli. That said, both perceptual difficulty and selective attention are thought to modulate the gain of neural responses in early sensory areas. Previous studies found that selectively attending to a stimulus or increasing perceptual difficulty enhanced the gain of neurons in visual cortex. However, some other studies suggest that perceptual difficulty can have either a null or even reversed effect on gain modulations in visual cortex. According to Yerkes–Dodson's Law, it is possible that this discrepancy arises because of an interaction between perceptual difficulty and attentional gain modulations yielding a nonlinear inverted-U function. Here, we used EEG to measure modulations in the visual cortex of male and female human participants performing an attention-cueing task where we systematically manipulated perceptual difficulty across blocks of trials. The behavioral and neural data implicate a nonlinear inverted-U relationship between selective attention and perceptual difficulty: a focused-attention cue led to larger response gain in both neural and behavioral data at intermediate difficulty levels compared with when the task was more or less difficult. Moreover, difficulty-related changes in attentional gain positively correlated with those predicted by quantitative modeling of the behavioral data. These findings suggest that perceptual difficulty mediates attention-related changes in perceptual performance via selective neural modulations in human visual cortex.
SIGNIFICANCE STATEMENT Both perceptual difficulty and selective attention are thought to influence perceptual performance by modulating response gain in early sensory areas. That said, less is known about how selective attention interacts with perceptual difficulty. Here, we measured neural gain modulations in the visual cortex of human participants performing an attention-cueing task where perceptual difficulty was systematically manipulated. Consistent with Yerkes–Dodson's Law, our behavioral and neural data implicate a nonlinear inverted-U relationship between selective attention and perceptual difficulty. These results suggest that perceptual difficulty mediates attention-related changes in perceptual performance via selective neural modulations in visual cortex, extending our understanding of the attentional operation under different levels of perceptual demands.
Introduction
In a wide range of behavioral tasks, performance is modulated by how much priority is placed on processing information about a relevant stimulus and how hard is it to detect and discriminate relevant stimulus features (Urbach and Spitzer, 1995; Ahissar and Hochstein, 1997; McDowd and Craik, 1988; Lavie et al., 2004; Prinzmetal et al., 2009; Carrasco, 2011; Giesbrecht et al., 2014). These two factors are known as selective attention and perceptual difficulty, respectively. While the effects of attention on neural gain in early visual cortex (Moran and Desimone, 1985; Mangun and Hillyard, 1987; Desimone and Duncan, 1995; Johannes et al., 1995; Connor et al., 1997; Hillyard et al., 1998; Mangun and Buck, 1998; McAdams and Maunsell, 1999; Reynolds et al., 2000; Martínez-Trujillo and Treue, 2002; Reynolds and Chelazzi, 2004; Maunsell and Treue, 2006; Lee and Maunsell, 2009; Reynolds and Heeger, 2009; Lee and Maunsell, 2010; Anton-Erxleben and Carrasco, 2013; Buschman and Kastner, 2015; Luo and Maunsell, 2015; Foster et al., 2021), and the concurrent attention-related benefits in perceptual performance, have been extensively studied (Mangun and Hillyard, 1988, 1990; Störmer et al., 2009; Pestilli et al., 2011; Itthipuripat et al., 2014a, b, 2017; Itthipuripat and Serences, 2016), it is still unclear how perceptual difficulty mediates neural responses in visual cortex and how perceptual difficulty interacts with selective attention.
First, prior studies that only manipulated perceptual difficulty without systematically manipulating attention have found mixed effects on neural activity in visual cortex. Some early studies in nonhuman primates found that increasing perceptual difficulty increased neural firing rates in early visual areas and sharpened neural tuning functions in mid-level visual areas (Spitzer et al., 1988; Spitzer and Richmond, 1991). In contrast, a recent study reported that increasing perceptual difficulty lead to increased firing rates in some neurons and decreased firing rates in others, resulting in null effects of perceptual difficulty on the population-level activity in early visual cortex (Ruff and Cohen, 2014a). We hypothesized that different effects of perceptual difficulty on neural activity were partially because of a nonlinear interaction between perceptual difficulty and the effects of selective attention on gain modulations in visual cortex.
This hypothesis is hard to evaluate based on prior work because of differences in task design and most studies did not independently manipulate perceptual difficulty and attention. In a few studies that did jointly manipulate these factors, perceptual difficulty was manipulated across a limited range, limiting the ability to observe nonmonotonic modulations in attentional gain with increased perceptual difficulty (Chen et al., 2008; Dodson, 1915; also see Motter, 1993; Handy and Mangun, 2000; Handy et al., 2001; Boudreau et al., 2006; Prinzmetal et al., 2009). Moreover, data linking difficulty-related changes in attentional modulations of neural activity and behavioral performance are still lacking.
According to the influential Yerkes–Dodson's Law, arousal state should mediate behavioral performance and the underlying neural processes in line with a nonlinear inverted-U function (Yerkes and Dodson, 1908; Dodson, 1915; see also Hull, 1943; Broadhurst, 1957; Duffy, 1957; Anderson et al., 1989; Diamond et al., 2007). Therefore, increasing perceptual difficulty from easy to an intermediate level could enhance overall arousal, leading to larger attentional gain modulations. On the other hand, when a behavioral task is too difficult, arousal might be too high, leading to no further increase or even a drop in attentional gain. Based on this framework, it is possible that some behavioral tasks were already challenging for the subjects to perform, resulting in no gain modulation with increasing difficulty (e.g., from ∼80% to ∼63% accuracy in Ruff and Cohen, 2014a). However, if baseline perceptual difficulty was low, then increasing difficulty might lead to large gain modulations (e.g., from ∼93% to ∼73% accuracy in Spitzer et al., 1988). In either case, the unique contribution of each of these two factors can only be assessed by parametrically varying the perceptual demand of behavioral tasks across the entire range of difficulty levels while independently manipulating the focus of attention.
Here, we designed a paradigm to manipulate perceptual difficulty to assess possible nonmonotonic effects on attentional gain in human visual cortex. We concurrently measured behavioral performance and the amplitudes of steady-state visually evoked potentials (SSVEPs) while human participants performed a contrast discrimination task (Fig. 1). Moreover, we employed a linking model based on the signal detection theory (SDT) to describe the interaction between attention and perceptual difficulty based on brain-behavior modulations (Tanner and Swets, 1954; Pestilli et al., 2011; Itthipuripat et al., 2014a, 2017; Itthipuripat and Serences, 2016).
Materials and Methods
Subjects
We recruited seven neurologically healthy male and female human subjects (19–23 years old) from the community at University of California San Diego (UCSD). All subjects had normal or corrected-to-normal vision and participated in 6 d of experiments, including 1 d of behavioral training and 5 d of EEG recording. After the first day of training, one subject decided not to continue, resulting in the complete datasets from six subjects in the final analysis (three female, three male, all right-handed, mean age ± SD = 20.5 ± 1.217 years old). Before their participation, each subject signed an informed consent form approved by the Institutional Review Board at UCSD. They were compensated for $10 and $15 per hour for participating in behavioral and EEG experiments, respectively. The use of six subjects is within the typical range for studies using similar multiple session approaches to measure behavioral and neural contrast response functions (CRFs) in human subjects (Di Russo et al., 2001; Morrone et al., 2002, 2004; Carrasco et al., 2004; Pestilli and Carrasco, 2005; Ling and Carrasco, 2006; Pestilli et al., 2007, 2009, 2011; Herrmann et al., 2010; Itthipuripat et al., 2014a, b, 2017).
Experimental design and statistical analysis
Stimuli and tasks
Stimuli and behavioral tasks were controlled by a PC running Windows XP using MATLAB (The MathWorks) and the Psychophysics Toolbox (version 3.0.8; Brainard, 1997; Pelli, 1997). Subjects sat 60 cm from a CRT monitor with a gray background of 34.51 cd/m2 and the refresh rate of 120 Hz. We conducted the experiments in the dark room in a quiet experimental area for the behavioral training sessions and in a dark, sound-attenuated, and electromagnetically shielded room (ETS Lindgren) for the EEG recording sessions.
In the main EEG experiment, subjects fixated at the center of the screen while performing different variants of the attention-cued contrast discrimination task with differing levels of task difficulty (easy, medium, and hard; Fig. 1). In our tasks, each trial began with either a focused-attention or a divided-attention cue, followed by an array of two checkerboard wedge stimuli (i.e., a target and a nontarget stimulus) appearing 400–600 ms after the cue onset for a duration of 1500 ms. The focused-attention cue pointed directly to the target stimulus appearing on either the left or the right lower visual quadrant. The divided attention cue pointed to both quadrants and the target was equally likely to appear in one of these locations.
The target stimulus contained a contrast increment in either the parafoveal or peripheral segments of the stimulus. Subjects reported which of these segments had the contrast increment as accurately and fast as possible by pressing one of the two corresponding buttons on the keyboard using their right index and middle fingers, respectively. Note that this contrast increment appeared for the entire stimulus duration of 1500 ms. The nontarget stimulus had the same contrast value throughout the entire stimulus region. The spatial frequency of the wedge stimuli was adjusted along the foveal-to-peripheral axis to emulate the cortical magnification factor measured in human V1 (Cowey and Rolls, 1974; Sereno et al., 1995; Engel et al., 1997; Duncan and Boynton, 2003). The inner and outer edges of the wedge stimuli were placed 7.03° and 13.46° visual angle away from fixation, respectively. The two lateral edges of these wedge stimuli were 7.944° apart.
To measure behavioral and neural data as a function of stimulus contrast, we pseudo-randomized the pedestal contrasts (or baseline contrasts) of the target and nontarget stimuli independently from six contrast levels: 0%, 3.75%, 7.5%, 15%, 30%, and 60% Michelson contrasts. To simultaneously monitor SSVEPs evoked by individual stimuli, the visual stimuli on the lower left and lower right quadrants were flickered on-off at 24 and 20 Hz, respectively. Then, placeholders came up 300 ms after stimulus offset on both sides of the contrast increment segment of the target stimulus. These placeholders would change from black to blue, red, or yellow to provide feedback about whether the subject responded correctly, incorrectly, or too slowly (i.e., slower than 1500 ms after stimulus onset), respectively. The intertrial interval was pseudo-randomly drawn from a uniform distribution spanning 300–500 ms.
In order to study the effect of perceptual difficulty on the attentional modulations of behavioral and EEG data, we adjusted the contrast thresholds for each of the attention conditions and the pedestal contrast levels across blocks of trials so that the accuracy levels of the behavioral tasks were maintained at ∼91%, ∼76%, and ∼65% accuracy for the easy, medium, and hard blocks, respectively (see behavioral results in Fig. 2A). On the first day, subjects underwent a behavioral training session for 2.5 h, where we implemented the method of constant stimuli to estimate initial contrast discrimination thresholds to be used for the easy, medium, and hard blocks on the first day of EEG recording. On each of the 5 d of the EEG experiment, subjects underwent six easy blocks, six medium blocks, and six hard blocks. Block types changed every three blocks and the order of blocks types were pseudo-randomized across subjects. Collectively, the entire EEG experiment contained 4320 trials where all experimental conditions were counterbalanced: two attention conditions × two target locations × six pedestal contrast levels of target × two contrast increment locations × three difficulty level × 30 repeats. Trial order was also pseudo-randomized so that trial types were not predictable.
Behavioral analyses
The contrast increment values averaged across all EEG sessions were plotted as a function of pedestal contrasts, yielding threshold versus contrast (TvC) functions for all difficulty levels (easy, medium, and hard) and attention conditions (focused and divided attention). The within-subject SEMs associated with these contrast increment values were computed and plotted for each level of stimulus contrast (Loftus and Masson, 1994). There were 120 trials in total for each of these 36 experimental conditions. 3-way repeated-measures ANOVAs were used to examine the effects of perceptual difficulty (easy/medium/hard), attention (focused-attention/divided-attention), and contrast (0–60%) on hit rates, mean reaction times on correct trials (correct RTs) and contrast discrimination thresholds (Fig. 2). For each difficulty level we also performed two-way repeated-measures ANOVAs to test the main effects of attention and interactions between attention and contrast on the perceptual threshold data. For each ANOVA that yielded a significant main effect of attention or a significant interaction between attention and contrast, we performed subsequent pairwise t tests to examine the effects of attention on the contrast thresholds across individual contrast levels (one-tailed because of the predicted direction of attentional modulations, i.e., attention decreasing contrast thresholds). Multiple comparisons for 6 individual contrast levels were corrected using the Holm–Bonferroni method (Holm, 1979).
In addition, we used a combination of
Here, c is stimulus contrast and
Here,
In this equation,
Here, Z is the inverse cumulative distribution function of the standard normal. For each subject, we fit the TvC data separately for individual difficulty levels and attention conditions with Equations 1–4 using MATLAB's fminsearch function (Nelder–Mead method; nonlinear least squares) with
Where min(
EEG data acquisition, preprocessing, and analyses
EEG data were recorded using a 64-channel ActiveTwo system (Biosemi Instrumentation) with the sampling rate of 512 Hz. There were 8 additional external electrodes: a pair of reference electrodes placed on the left and right mastoids, a pair of electrodes affixed near the outer canthi of the left and right eyes for monitoring horizontal eye movements, and two pairs of electrodes affixed above and below the left and right eyes for monitoring blinks and vertical eye movements. The EEG data were online-referenced to the CMS-DRL electrode, and the data offset in all electrodes were maintained 20 V, which is a standard criterion for this active electrode system.
We used custom MATLAB scripts and EEGLab11.0.3.1b (Delorme and Makeig, 2004) to preprocess and analyze EEG data. First, the EEG data from all electrodes were re-referenced to the algebraic mean of the data recorded from the left and right mastoid electrodes. Then, we applied the 0.25-Hz high-pass and 55-Hz low-pass Butterworth filters (third order). The continuous EEG data were segmented into epochs extending from 500 ms before to 3000 ms after cue onset. Independent component analysis was performed to remove prominent eye blinks (Makeig et al., 1996). Next, we used visual inspection and threshold rejections to further reject trials confounded by residual eye movements, muscle activity, drifts, and other artifacts. This step led to the removal of 7.38 ± 4.43% SD of trials across all six subjects.
Next, the epoched data were time-locked to stimulus onset and the baseline activity was subtracted from −100 to 0 ms relative to the stimulus onset. The data were then sorted into the following experimental bins: focused-attention target stimuli, focused-attention nontarget stimuli (e.g., ignored stimuli), and divided-attention stimuli separately for the easy, medium, and hard blocks. For each of these bins, trials were also sorted into 12 subbins depending on the contrast and the location of the stimulus of interest (six contrast levels × two target locations). Following this data sorting step, the EEG data in all subbins were then averaged to obtain event-related potentials (ERPs) for each condition. Next, the ERPs were filtered with a Gaussian wavelet function with a 0.1 fractional bandwidth and center frequencies of 20 and 24 Hz, resulting in frequency-domain coefficients at the stimulus driving frequencies. Given this small fractional bandwidth, the frequency-domain standard deviation of the SSVEP signals were 0.8493 and 1.0192 Hz for 20- and 24-Hz flickers, respectively. In turn, we obtained the amplitudes of the SSVEPs evoked by individual stimulus-flicker frequencies by computing the absolute value of the coefficients. For each subject, the SSVEP amplitudes in individual subbins were then averaged across five contralateral posterior occipital electrodes where the SSVEP amplitudes collapsed across contrast levels, stimulus types, difficulty levels, and attention conditions were maximal. The data were then averaged across a 0–500 ms poststimulus window to avoid potential confounds from including data after a response had been made which was around 550–850 ms across all conditions (Fig. 2B). Here, the within-subject SEMs associated with the averaged SSVEP were computed for each level of stimulus contrast (Loftus and Masson, 1994). The mean SSVEP amplitudes and their SEMs were then plotted as a function of stimulus contrast, yielding SSVEP-based CRFs for each of the difficulty levels and attention conditions. On the contrast axis of the CRFs, the stimulus contrast values for the ignored stimulus were fixed at 0%, 3.75%, 7.5%, 15%, 30%, and 60% Michelson contrasts. However, the contrast values in the focused-attention and divided-attention conditions were obtained from the average contrast values between the pedestal and increment segments because the target stimuli contained both pedestal and contrast increment segments. Since the fit parameters were obtained from individual CRFs with true mean physical contrasts of all stimulus segments (i.e., instead of directly comparing SSVEP amplitudes for each pedestal contrast), any task-related and attention-related modulations of fit parameters associated with neural CRFs should be minimally affected by low-level physical differences in visual stimuli.
Next, we fit SSVEP-based CRFs for each difficulty level and attention condition of individual subjects with the Naka–Rushton equation (Eq. 2), where (
There were three main sets of statistical comparisons we employed to test the effects of attention and their interactions with perceptual difficulty. First, we tested the overall differences across focused-attention, divided-attention stimuli, focused-attention nontarget (or ignored) stimuli from the data collapsed across all difficulty levels. To do so, we computed the probability at which the resampled
Note that in the current study, the stimulus duration of 1500 ms and the time window of the main analysis (0–500 ms) are not perfectly aligned. We chose this analysis window based on the protocol from a recent SSVEP from our group that used a similar contrast discrimination task (Itthipuripat et al., 2018). In this prior study, we knew that subjects made a behavioral response ∼500 ms after stimulus onset, and that motivated the choice of the analysis window. That said, we opted to keep the stimulus duration at 1500 ms because we also manipulated perceptual difficulty which could result in longer RTs. As it turned out, responses were made ∼550–850 ms after stimulus onset in the present study. So, restricting the SSVEP window to 0–500 ms helped minimize confounds from motor-related activity and ensured that we only examined attentional modulations while subjects were still actively attending to the stimuli. That said, we also conducted the similar analyses for the data averaged across 500–1000 ms and across 1000–1500 ms to investigate the effects of attention and perceptual difficulty at these later time windows.
Although we chose the stimulus flickers (20 and 24 Hz) above the α range (∼10 Hz) to avoid contaminations from alpha band oscillations, one of our stimulus flickers (i.e., 20 Hz) still overlapped with the second harmonic of endogenous alpha band activity. In order to rule out the potential confound from the second harmonic of α, we conducted an additional analysis where we examined the effects of perceptual difficulty and attention on the CRFs based on the mean amplitude of induced EEG activity at 20 Hz. We first applied a wavelet filter at 20 Hz to the trial-by-trial EEG data and then computed the amplitude of the wavelet coefficients. We then averaged the amplitude data across trials to compute the induced EEG activity at 20 Hz as a function of stimulus contrast separately for different difficulty levels and attention conditions. Note that this method is different from the analysis performed to obtain SSVEP, where we averaged trial-by-trial EEG data first to isolate the 20-Hz steady-state response before applying the wavelet filter. We then bootstrapped data using the similar resampling and fitting procedures performed in the main SSVEP analysis.
Correlation analysis
We examined how difficulty-related changes in attention-induced response gain modulations observed in the SSVEP data related to those predicted by modeling the TvC data at the intersubject level. To do so, we first computed differences in attention-induced changes in the rescaled
Data sharing
The stimulus presentation and analysis code as well as the behavioral and EEG data have been made available on The Open Science Framework (see the following links: https://osf.io/bc3hj/?view_only=326b8c4ffeb842dcbb81673f7f3a852c and https://osf.io/rbnm6/?view_only=aee48f6f9fae44d6a4e917af99aa709d).
Results
The present study used the attention-cueing contrast discrimination task shown in Figure 1 to simultaneously monitor attentional modulations of perceptual thresholds and SSVEPs measured as a function of stimulus contrast (the threshold vs contrast functions, or TvCs, and the SSVEP-based contrast response functions, or CRFs, respectively). Each trial started with a focused-attention cue pointing to the target stimulus or a divided-attention cue pointing to both target and nontarget stimuli. Subjects determined whether the target contained a contrast increment either in the more foveal or the more peripheral region of the target stimulus as accurately and quickly as possible using a button-press response. The pedestal (or baseline) contrasts for the visual stimuli were pseudo-randomly selected from 0%, 3.75%, 7.5%, 15%, 30%, and 60% Michelson contrasts. The left and right stimuli were flickered at 20 and 24 Hz, respectively, so that we could simultaneously monitor SSVEPs evoked by both stimuli. To manipulate perceptual difficulty, we adjusted the magnitude of contrast increments every three blocks to control hit rates in the easy, medium, and hard blocks.
The attention-cueing contrast discrimination task. Each trial started with a focused-attention or divided-attention cue. The focused-attention cue pointed directly to the target stimulus, but the divided-attention cue pointed to both target and nontarget stimuli. Subjects had to determine whether the target contained a contrast increment at the inner or outer region of the stimulus. Once it occurred, the contrast increment was presented for the duration of the stimulus presentation period. Feedback, which indicated the exact position of the contrast increment, was presented in blue for correct responses and in red for incorrect responses. The pedestal contrasts were independently and pseudo-randomly selected from 0%, 3.75%, 7.5%, 15%, 30%, and 60% Michelson contrasts. The left and right stimuli were flickered at 24 and 20 Hz, respectively, to allow the simultaneous monitoring of SSVEPs evoked by both stimuli. Perceptual difficulty was systematically manipulated across blocks of trials by adjusting the magnitude of the contrast increments so that hit rates were maintained at ∼91%, ∼76%, and ∼65% for easy, medium, and hard blocks, respectively.
Behavioral results
Perceptual difficulty for the easy, medium, and hard blocks was successfully controlled at ∼91% (mean = 91.34%, ±1.60% SEM), ∼76% (mean = 76.27%, ±0.97% SEM), and ∼65% accuracy (mean = 65.27%, ±2.95% SEM), respectively (Fig. 2A). A repeated-measures ANOVA showed a significant main effect of perceptual difficulty on accuracy (F(2,10) = 174.88, p < 0.001). Since perceptual difficulty was intentionally equated across attention tasks and contrast levels, there was no main effect of attention (F(1,5) = 0.22, p = 0.659), no main effect of contrast (F(5,25) = 2.04, p = 0.108), and no interaction between attention and perceptual difficulty on hit rates (F(5,25) = 1.99, p = 0.115). Increasing perceptual difficulty significantly increased correct RTs (F(2,10) = 57.46, p < 0.001) and divided attention increased correct RTs compared with focused attention (F(1,5) = 34.87, p = 0.002; Fig. 2B). In addition, we found no main effect of contrast (F(5,25) = 2.33, p = 0.072), and no interaction between attention, perceptual difficulty, and contrast on correct RTs (ps ≥ 0.305).
Hit rates, response times on correct trials (correct RTs) and perceptual thresholds in the contrast discrimination task plotted as a function of pedestal contrast. A, The accuracy levels were successfully controlled at ∼91%, ∼76%, and ∼65% for the easy, medium, and hard runs, respectively. Hit rates were also equated across the focused-attention and divided-attention conditions. B, Correct RTs increased with increasing perceptual difficulty and in the divided compared with focused attention condition. C, The threshold versus contrast (TvC) functions for the easy, medium, and difficult blocks. The TvC data were fit using a combination of Naka–Rushon and d' equations (Eqs. 1–4) that describe the link between behavior and the hypothetical neural contrast response functions (CRFs; Fig. 3). Error bars indicate ±1 within-subject SEM. Diff*** labels in A and B indicate significant main effects of perceptual difficulty on hit rates and correct RTs with ps < 0.001. Att** in B shows a significant main effect of attention on correct RTs with p < 0.01. Att*** and Att x Ct* labels in C indicate a significant main effect of attention and a significant interaction between attention and contrast on the perceptual thresholds with ps < 0.001 and 0.05, respectively.
The manipulation of perceptual difficulty across blocks of trials led to significant decreases in contrast discrimination thresholds as a function of perceptual difficulty (Fig. 2C; a main effect of perceptual difficulty on contrast discrimination thresholds: F(2,10) = 21.30, p < 0.001). Compared with divided attention, focused attention significantly reduced contrast discrimination thresholds (a main effect of attention on contrast discrimination thresholds: F(1,5) = 10.83, p = 0.023), consistent with previous results (Huang and Dobkins, 2005; Pestilli et al., 2011; Hara and Gardner, 2014; Itthipuripat et al., 2014a, 2017, 2019). Note that these changes in contrast thresholds were not because of differences in perceptual difficulty across attention tasks because accuracy levels were successfully equated across the focused-attention and divided-attention conditions and across the 6 pedestal contrasts (see the statistics in the previous paragraph).
The attention-induced decreases in contrast thresholds were mainly driven by a significant main effect of attention in the medium difficulty blocks (F(1,5) = 23.14, p = 0.005). At this difficulty level, there is no interaction between focused/divided attention and contrast (F(1,5) = 2.26, p = 0.080). Post hoc paired t tests revealed significant decreases in contrast discrimination thresholds in the focused-attention compared with the divided-attention conditions for all of the 6 pedestal contrast levels at this difficulty level (t(5)s = −2.76, −5.75, −3.78, −3.28, −3.31, and −2.50 with ps = 0.020, 0.001, 0.006, 0.011, 0.011, and 0.027 for 0–60% contrast levels, respectively; one-tailed because of the known direction of the main effect of attention without the interaction with attention and contrast; i.e., the reduced thresholds with focused attention, passing the Holm–Bonferroni corrected threshold of 0.05; Fig. 2C). This finding is consistent with data reported in previous studies where perceptual difficulty was controlled at a similar accuracy level (Huang and Dobkins, 2005; Pestilli et al., 2011; Hara and Gardner, 2014; Itthipuripat et al., 2014a, 2017, 2019). In contrast, we observed no significant main effects of attention on contrast thresholds in the easy or in the hard blocks (F(1,5) = 4.17, p = 0.097, and F(1,5) = 3.88, p = 0.106, respectively). That said, in the hard blocks, there was a significant interaction between attention and pedestal contrast on the contrast threshold data, which was driven by a significant decrease in threshold for the highest contrast level; i.e., 60% contrast (t(5) = −4.02, p = 0.005, one-tailed, Holm–Bonferroni corrected) with no significant and some marginal attentional modulations at the lower contrast levels (t(5)s = 0.33, −2.66, −2.15, −1.25, and 0.54 with ps = 0.377, 0.022, 0.042, 0.134, 0.205 for 0%−30% contrast levels, respectively; one-tailed, not passing Holm–Bonferroni corrected threshold of 0.0083).
To estimate how much neural attentional gain is needed to account for the attention-induced changes in contrast thresholds, we employed a linking model based on the signal detection theory (SDT; Tanner and Swets, 1954; Legge and Foley, 1980; Boynton et al., 1999; Gorea and Sagi, 2001; Huang and Dobkins, 2005; Pestilli et al., 2011; Hara and Gardner, 2014; Itthipuripat et al., 2014a, 2017; Itthipuripat and Serences, 2016; Fig. 3). Under this modeling framework, the perceptual contrast threshold (
Quantitative linking model based on the signal detection theory (SDT; Tanner and Swets, 1954; Boynton et al., 1999; Pestilli et al., 2011; Itthipuripat et al., 2014a, 2017; Itthipuripat and Serences, 2016). The level of perceptual contrast threshold (
Based on the model fitting procedure, we obtained estimates of the response gain (
Individual fits for the threshold versus contrast (TvC) functions across all subjects.
The corresponding best fit parameters of the TvC functions (see Figs. 2, 4). The first column shows the mean values (empty symbols) and 68% confidence intervals (error bars) from the bootstrapped distributions of the response gain (
For the behavioral data collected in the medium and hard blocks, the model predicted an increase in
The response gain parameters predicted from the TvC data were expressed in arbitrary units. Thus, we rescaled the predicted response gain parameters so that overall predicted response gain parameters across individual subjects spanned a range of 0–1. Since this rescaling step was performed using the minimum and maximum values of the response gain predicted by the behavioral data across all task and attention conditions (not within individual conditions), it did not produce substantial effects or interact with task-related and attention-related modulations. Specifically, the normalized
Unlike
SSVEP results
To determine whether the actual pattern of attentional modulations in neural responses followed the pattern predicted by the SDT model (Fig. 3), we measured SSVEP responses over human visual cortex. SSVEP responses are the phase-locked EEG responses generated from early visual areas in response to rhythmically presented stimuli (Regan, 1989; Andersen et al., 2012; Norcia et al., 2015). Many past studies have shown that attention can either increase response or contrast gain of the SSVEP-based CRFs (Hillyard and Anllo-Vento, 1998; Di Russo et al., 2001; Lauritzen et al., 2010; Wang and Wade, 2011; Itthipuripat et al., 2014a,b, 2018), making it a sensitive measure to test this specific prediction. In the current study, we computed the SSVEP amplitudes at the driving stimulus frequencies of 24 and 20 Hz for the left and right stimuli, respectively (see Materials and Methods, EEG data acquisition, preprocessing, and analyses). According to the full frequency spectrum of the EEG data from the contralateral posterior occipital electrodes, we found that our stimulus flicker rates of 20 and 24 Hz induced robust SSVEP signals that peaked around the central frequencies throughout the entire stimulus presentation period (i.e., 0–1500 ms; Fig. 6). The amplitudes of the SSVEP responses scaled with stimulus contrast. Importantly, the amplitude at the SSVEP frequencies were far higher than the amplitude associated with the frequencies of early visually evoked potentials (VEPs; 8–9 Hz) that appeared ∼0–500 ms poststimulus onset. This ensured that our SSVEP results were minimally confounded by modulations of slow-going VEPs and α oscillations.
The full frequency spectrum of the EEG data from the posterior occipital electrodes contralateral to the stimulus flickers of 20 and 24 Hz, shown in cold and hot colors, respectively. The SSVEP responses were centered at the flicker frequencies of 20 and 24 Hz for the duration of the stimulus presentation (0–1500 ms). There was also early stimulus-evoked activity at low frequencies (8–9 Hz) from 0 to 500 ms poststimulus onset. The amplitude of the SSVEP responses scaled with stimulus contrast. Importantly, the SSVEP frequencies were far above the VEP and α frequencies and they were thus not overlapping.
We then averaged the SSVEP amplitudes across contralateral posterior occipital electrodes where the responses, collapsed across contrast levels, stimulus types, difficulty levels, and attention conditions, were maximal. Finally, we plotted the mean amplitude of the SSVEP responses as a function of stimulus contrast, producing SSVEP-based CRFs for all difficulty levels (easy, medium, and hard) as well as attention conditions [i.e., the focused-attention target, the focused-attention nontarget (or ignored), and the divided-attention stimuli; see Fig. 7].
A, The SSVEP-based CRFs from 0 to 500 ms poststimulus across different attention conditions and difficulty levels (the top and bottom panels are the same data organized in different layouts). Error bars indicate within-subject SEM. Focused attention produced the most robust effects on SSVEP-based response gain in the medium, compared with the easy and hard blocks (see corresponding fit parameters in Fig. 9). Note that in the focused-attention and divided-attention conditions, the stimulus contrast values on the x-axis reflect the mean values computed across all segments of the target stimulus (both pedestal and incremental contrasts). In contrast, the stimulus contrast values in the ignored condition were based solely on the pedestal contrast because the ignored nontarget stimuli contained no contrast increments. B, Topographical maps of SSVEP data elicited by left and right stimuli of 24- and 20-Hz flicker frequencies, respectively, plotted separately for different contrast levels, attention conditions, and difficulty levels. The data were averaged from 0 to 500 ms poststimulus onset. For both left and right stimuli, focused attention produced the highest effects on SSVEP amplitudes at high contrast levels in the medium difficulty compared with the easy and hard blocks.
Overall, we found a nonmonotonic inverted-U relationship between perceptual difficulty and attentional gain modulations in the SSVEP-based CRFs. In the medium difficulty blocks, there was a robust increase in response gain in the SSVEP-based CRFs in the focused-attention target compared with the divided-attention and the focused-attention nontarget (ignored) stimuli (see Figs. 7, 8 for the grand-averaged SSVEP data and individual subjects' data, respectively). However, attention-related response gain modulations of the SSVEP-based CRFs were smaller in the easy and hard blocks. This pattern of results was also consistent at the individual-subject level (Figs. 8, 9).
Individual fits for the SSVEP-based contrast response functions (CRFs) across all subjects.
A, Response gain (
To more precisely characterize the impact of perceptual difficulty and attention on response and contrast gain modulations of these SSVEP-based CRFs, we next fit the data with a Naka–Rushon equation (see Materials and Methods, EEG data acquisition, preprocessing, and analyses; and Eq. 2). We used the
There were three main sets of statistical comparisons we employed to test the effects of attention and possible interactions with perceptual difficulty. First, we tested for overall differences across focused-attention, divided-attention stimuli, focused-attention nontarget (or ignored) stimuli using data collapsed across all difficulty levels. Second, we examined these differences separately for each of the difficulty levels. Finally, we tested for an interaction between attention and perceptual difficulty level. Specifically, we examined whether the attentional modulations (focused minus divided attention) were different across the medium and easy blocks, across the medium and hard blocks, and across the hard and easy blocks, respectively.
Collapsed across all difficulty levels, we observed significant increases in
Specifically, we observed significant increases in
Importantly, when examining the interaction between attention and perceptual difficulty, we found significant differences in attentional modulations (focused minus divided attention) between the medium and easy blocks and between the hard and easy blocks but not between the medium and hard blocks (ps = 0.010, 0.014, and 0.573, respectively, one-tailed because of the known direction of the modulations, Holm–Bonferroni corrected). Taken together, the observed response gain patterns suggest nonlinear changes in attentional modulation of response gain as a function of perceptual difficulty.
For the contrast gain data, we found a decrease instead of an increase in contrast gain. Specifically, focused attention shifted the CRFs rightwards instead of leftwards. There were significant increases in the
An auxiliary analysis that examined changes in baseline (b) of the CRFs showed no significant differences between the focused-attention target and the divided-attention stimuli, between the focused-attention target and the ignored stimuli, or between the divided-attention stimuli and the ignored stimuli (p = 0.042–0.910; Fig. 9B, bottom panels). That said, the patterns of response and contrast gain patterns were highly similar to those observed with the baseline subtraction methods (compare Fig. 9A and B).
Notably, nonlinear changes in attentional modulations of response gain as a function of perceptual difficulty in the SSVEP data (Fig. 9A,B, top panels) were only observed in the time period before (from 0 to 500 ms poststimulus), but not during or after subjects, made responses (from 500 to 1500 ms poststimulus; Fig. 10). Specifically, from 500 to 1000 ms poststimulus, there was a significant difference in the
A, The SSVEP-based CRFs from 0 to 1500 ms poststimulus across different attention conditions and difficulty levels. Error bars indicate within-subject SEM. Focused attention produced the most robust effects on the SSVEP-based response gain in the medium difficulty block, compared with the easy and hard blocks only before (0–500 ms in the left column), but not during (500–1000 ms in the middle column) or after subjects responded (1000–1500 ms in in the right column). B, Corresponding response gain (
For contrast gain, there was no significant difference in the
Although we chose to flicker our stimuli in frequencies (i.e., 20 and 24 Hz) that were far above the α range (∼10 Hz), the amplitude of the SSVEP response at 20 Hz could potentially be mediated by the harmonic of endogenous alpha band oscillations. In order to test this possibility, we conducted an additional analysis where we examined the effects of perceptual difficulty and attention on the CRFs based on the mean amplitude of induced (total) EEG activity at 20 Hz. Unlike the SSVEP results, we found no significant modulations of perceptual difficulty (ps = 0.941–0.969) and attention (ps = 0.228–0.918) and no interactions between these two factors on the response gain parameters describing the neural CRFs based on the 20-Hz-induced EEG oscillation (ps = 0.384–0.667, all ps are two-tailed; Fig. 11). Accordingly, it is unlikely that the inverted-U-shaped SSVEP results were contaminated by modulations of the harmonic of endogenous α oscillations.
A, The induced, or total, EEG amplitude at 20 Hz as a function of task difficulty and attention. Error bars indicate within-subject SEM. There was no main effect of perceptual difficulty, no main effect of attention, and no interaction between the two factors on the response gain parameters describing the shapes of the neural CRFs based on the amplitudes of the induced EEG oscillations at 20 Hz. B, The corresponding fit parameters of the neural CRFs. The empty symbols and error bars represent the mean values and 68% confidence intervals from the bootstrapped distributions of the fit parameters (resampling subject labels with replacement).
Correlations between behavioral and SSVEP data
Overall, the effect of attention and its interaction with perceptual difficulty on the SSVEP-based CRFs was consistent with modulatory patterns of response gain predicted by the behavioral data. This suggests a robust connection between the effects of perceptual difficulty on attentional gain modulations and changes in behavioral performance. To more formally evaluate this relationship, we evaluated between-subject correlations between the difficulty-related changes in attentional modulations of response gain predicted by the SDT-based modeling of the TvC data and those observed in the SSVEP-based CRFs. To do so, we examined whether difficulty-related changes in attention-related response gain in the SSVEP data positively correlated with those predicted by the TvC data on a between-subject basis. We first calculated the magnitude of the response gain modulations predicted by the modeling of the TvC data (focused minus divided attention) for individual subjects (same as above), and then computed the differences between the medium and easy blocks (Fig. 11, triangles), between the medium and hard blocks (Fig. 11, diamonds), and between the hard and easy blocks (Fig. 11, circles). Next, we repeated these steps for the normalized response gain modulations observed in the SSVEP data separately for 0–500, 500–1000, and 1000–1500 ms poststimulus. Finally, we correlated these difficulty-related differences in attention-induced response gain modulations obtained from the behavioral and SSVEP data using a repeated-measures correlation method. We observed a significant positive correlation between the difficulty-related differences in response gain modulations in the behavioral and SSVEP data measured only before response period (0–500 ms poststimulus; Rho = 0.689, p = 0.009; Fig. 12). That said, no significant correlations were found between the behavioral and SSVEP data measured during and after response period (Rhos = 0.115 and 0.103 and ps = 0.708 and 0.737 for 500–1000 and 1000–1500 ms poststimulus, respectively). Note that the correlation results reported here should be interpreted with caution because of a small sample size.
Repeated measures correlations between difficulty-related differences in attentional modulations of response gain parameters obtained from fitting the SSVEP-based CRFs and behavioral TvCs. There was a significant correlation only in the time window before response execution (0–500 ms poststimulus onset; Rho = 0.689, p < 0.001) but not after (500–1500 ms; Rhos = 0.708–0.737, ps > 0.05). Different color dots represent differences in attentional modulations between the medium and easy blocks (triangles), between the medium and hard blocks (circles), and between the hard and easy blocks for individual subjects (diamonds). The data from the same subject are assigned the same color, with corresponding lines showing the repeated measures correlation fit for each subject.
Together, the data suggest a robust link between neural and behavioral attentional modulations across changes in perceptual difficulty and that the interactive effects between attention and perceptual difficulty on selective sensory information processing only occur at the time period relevant to behavior.
Discussion
The present study tested how perceptual difficulty influenced attentional modulations of the population-based neural activity in human visual cortex. We found that perceptual difficulty had a nonlinear inverted-U-shaped impact on response gain modulations of the behavioral TvC functions and on neural CRFs. Importantly, this nonlinear relationship could be accounted for by SDT-based quantitative modeling of perceptual contrast discrimination thresholds. Furthermore, positive correlations between these metrics suggest a close relationship between difficulty-related differences in attention-related response gain in the behavioral and neural data. Together, our results suggest that perceptual difficulty regulates behavioral performance in part by mediating the magnitude of attentional gain modulations in visual cortex.
Past behavioral studies have investigated the influence of perceptual difficulty on selective attention (Dosher and Lu, 2000; Lu and Dosher, 2000; Prinzmetal et al., 2009). One study compared the effects of perceptual difficulty on endogenous and exogenous attention (Prinzmetal et al., 2009). They found that increasing perceptual difficulty increased the beneficial effect of endogenous attention on behavioral performance but produced the opposite effect on exogenous attention (Prinzmetal et al., 2009). Our study used endogenous attention cues and showed similar behavioral effects: increasing perceptual difficulty from easy to intermediate levels improved perceptual performance. That said, making the task harder did not lead to a better performance and actually led to smaller attentional effects on behavior.
To assess attentional gain of population-level activity in visual cortex, we measured the effect of attention on SSVEPs. SSVEPs reflect the synchronized activity in visual cortex that oscillates in synchrony with periodic sensory inputs (for review, see Andersen et al., 2012; Norcia et al., 2015; also see Regan, 1989; Müller et al., 1997; Srinivasan et al., 2006; Di Russo et al., 2007; Andersen et al., 2008, 2012; Fuchs et al., 2008; Andersen and Muller, 2010; Störmer et al., 2013; Störmer and Alvarez, 2014). Note that the flicker frequencies of the driving stimuli (20 and 24 Hz) in the present study were far above those of the slow-going VEPs and alpha power. The full frequency spectrum of the EEG data also showed robust SSVEP signals peaked at the flicker frequencies. Thus, task-related attentional modulations of SSVEP activity reported here should be minimally influenced by those of the VEPs and alpha activity. Indeed, an analysis of induced power at 20 Hz (i.e., power not phase locked to the driving stimulus) revealed no attentional modulations. This suggests that our data were not contaminated by modulations of the second harmonic of endogenous alpha band oscillations at ∼10 Hz.
Consistent with the single-unit data reported by Chen and colleagues (Chen et al., 2008), we found that increasing perceptual difficulty from an accuracy level of 91% to 76% significantly increased response gain of SSVEP-based CRFs. However, when perceptual difficulty was increased further to 65% accuracy, there was no further increase in response gain. Instead, response gain was modestly attenuated, consistent with an inverted-U relationship between perceptual difficulty and attentional modulations. We speculate that we observed an inverted-U relationship, in contrast to a monotonic increase, because the difficulty range in our experiment was broader compared with previous studies (65–91% accuracy vs 86–98% accuracy in Chen et al., 2008).
Since contrast increments were independently adjusted for the focused-attention and divided-attention conditions in the present study, there were low-level differences in the target stimuli across these two conditions. However, it is unlikely that these low-level differences substantively contributed to the observed task-related attentional modulations of the SSVEPs. First, instead of using the pedestal contrast as contrast values to plot neural CRFs, we used the mean contrast values computed across all segments of the visual stimulus (both pedestal and incremental contrasts). Since the fit parameters were obtained from individual CRFs with true mean physical contrasts of all stimulus segments (i.e., instead of directly comparing SSVEP amplitudes for each pedestal contrast), we believe that any task-related and attention-related modulations of fit parameters associated with neural CRFs were minimally affected by low-level physical differences in visual stimuli. Second, overall mean stimulus contrast in the divided attention condition was higher than in the focused attention condition and contrasts in the easy block were higher than in the medium and hard blocks. If differences in our SSVEP data were because of these low-level physical differences, we should have observed task-related and attention-related modulations of the SSVEP data in opposite directions. For example, focused attention should have reduced the overall SSVEP amplitudes compared with divided attention. Moreover, the easy block should have generally produced the highest response amplitudes, compared with the medium and hard blocks. However, these were not the patterns of results we observed.
The nonlinear inverted-U relationships between perceptual difficulty and attentional gain modulations observed in our study are consistent with Yerkes–Dodson's Law, which describe how global cognitive factors and arousal states regulated behavioral performance (Yerkes and Dodson, 1908; Dodson, 1915; see also Broadhurst, 1957; Duffy, 1957; Anderson et al., 1989; Diamond et al., 2007). Indeed, a recent study conducted by Fischer and colleagues (2008) found that increasing task demands while human participants were racing in a driving simulator imacted overall cortical excitability measured in the frontal cortex following an inverted-U function (Loveless and Sanford, 1975; Elbert and Rockstroh, 1987; Birbaumer et al., 1990; Aeschbach et al., 1999; Nagai et al., 2004). That said, it is unclear how changes in perceptual difficulty, known to mediate arousal levels and cortical excitability, affect changes in selective information processing in the early sensory areas. The present study provides evidence suggesting that perceptual difficulty indeed regulates attention-induced response gain modulations at the earliest stages of visual cortical information processing.
Similar to the mixed effects of perceptual difficulty on gain modulations reported in previous studies, other global cognitive factors such as stress, anxiety, emotion, aversive stimuli, and expected rewards have been shown to either up-regulate or down-regulate the level of early sensory responses and in turn led to either an improvement or an impairment in performance (Gray et al., 2003; Keil et al., 2005; Phelps et al., 2006; Bakardjian et al., 2011; Hajcak et al., 2013; Schönwald and Müller, 2014; Song and Keil, 2013, 2014; Wieser et al., 2014; Baruni et al., 2015; Bekhtereva et al., 2015; Bekhtereva and Müller, 2015; Paczynski et al., 2015; Zhang et al., 2016; Chang et al., 2020). It is likely that some of these mixed results could be in part explained by Yerkes–Dodson's Law (Yerkes and Dodson, 1908; Dodson, 1915; see also Broadhurst, 1957; Duffy, 1957; Anderson et al., 1989; Diamond et al., 2007; Hull, 1943). However, there is still a paucity of research studies that parametrically manipulate the strength of these cognitive factors. Future studies could adopt our approach to determine whether similar inverted-U relationships can be generalized to explain the influence that other cognitive factors may have on selective information processing and perceptual performance.
Recent studies have proposed that heightened arousal states could desynchronize local neural activity and thus reduce correlated variability between neurons. In turn, reducing noise correlations may increase the efficiency of information processing in visual cortex (Ruff and Cohen, 2014b; Beaman et al., 2017). The difficulty-related changes in attentional gain observed here could in part reflect arousal-dependent changes in the synchrony of local neural activity, which could disrupt the SSVEP response (cf. Ruff and Cohen, 2014a; Kohn et al., 2016; Cohen and Kohn, 2011; Cohen and Maunsell, 2009; Mitchell et al., 2007, 2009). At the systems and neurotransmitter levels, arousal is also thought to increase local glutamate levels, which interact with norepinephrine and acetylcholine to amplify high priority sensory representations and inhibit low priority sensory representations (for review, see Mather et al., 2016). These neurotransmitters have been found to regulate receptor pathways that independently mediate neural gain amplification and neural noise modulations (Kirkwood et al., 1999; Herrero et al., 2008, 2013; Polack et al., 2013). That said, the links between neural mechanisms that support arousal-dependent attention at the neuromolecular, systems, physiological, and behavioral levels are not clear. This is a challenging research question that will likely require combining multimodal methods (e.g., pharmacological intervention, single-unit recording, local field potential, and EEG) and using computational models to integrate results across different levels of analysis (Wyler et al., 1982; Herrero et al., 2008, 2013; Purcell et al., 2013; Cosman et al., 2018; for review, see Itthipuripat and Serences, 2016).
In conclusion, we found that perceptual difficulty regulates attentional gain measured at the population level in human visual cortex: attentional gain was highest when perceptual difficulty was at an intermediate level and lower when the task was easier or harder. Moreover, the interaction between perceptual difficulty and attentional gain in visual cortex was correlated with difficulty-related modulations in perceptual performance. Taken together, our data suggest an inverted-U relationship between perceptual difficulty and selective processing of early sensory information, partly reconciling contradictory results of perceptual difficulty on attentional gain modulations in the early visual cortex.
Footnotes
This work was supported by the National Institutes of Health Grant R01-EY025872 and a James S. McDonnell Foundation award (to J.T.S.). This work was also supported by the National Research Council of Thailand grant (fiscal years 2021-2024, under project numbers 102976 and 118711); the Thailand Science Research and Innovation (TSRI) Basic Research Fund: fiscal year 2022 under project number FRB650048/0164, fiscal year 2021 under project number FRB640008, and fiscal year 2020 under project number 62W1501; the Program Management Unit (PMU) grant (fiscal year 2023); the Asahi Glass Foundation grant; the research grant from the Research & Innovation for Sustainability Center, Magnolia Quality Development Corporation Limited, Thailand; the KMUTT Partnering initiative grant (fiscal year 2021) and the startup fund for junior researchers at King Mongkut's University of Technology Thonburi (KMUTT); and the KMUTT's Frontier Research Unit Grant for Neuroscience Center for Research and Innovation (to S.It.).
The authors declare no competing financial interests.
- Correspondence should be addressed to Sirawaj Itthipuripat at itthipuripat.sirawaj{at}gmail.com
