Abstract
Previous studies have indicated that both increased physical salience and increased reward-value salience of a target improve behavioral measures of attentional selection. It is unclear, however, whether these two forms of salience interact with attentional networks through similar or different neural mechanisms, and what such differences might be. We examined this question by separately manipulating both the value-driven and physical salience of targets in a visual search task while recording response times (RTs) and event-related potentials, focusing on the attentional-orienting-sensitive N2pc event-related potential component. Human participants of both sexes searched arrays for targets of either a high-physical-salience color or one of two low-physical-salience colors across three experimental phases. The first phase (“baseline”) offered no rewards. RT and N2pc latencies were shorter for high-physical-salience targets, indicating faster attentional orienting. In the second phase (“equal-reward”), a low monetary reward was given for fast correct responses for all target types. This reward context improved overall performance, similarly shortening RTs and enhancing N2pc amplitudes for all target types, but with no change in N2pc latencies. In the third phase (“selective-reward”), the reward rate was made selectively higher for one of the two low-physical-salience colors, resulting in their RTs becoming as fast as the high-physical-salience targets. Despite the equally fast RTs, the N2pc's for these low-physical-salience, high-value targets remained later than for high-physical-salience targets, instead eliciting significantly larger N2pc's. These results suggest that enhanced physical salience leads to faster attentional orienting, but value-driven salience to stronger attentional orienting, underscoring the utilization of different underlying mechanisms.
SIGNIFICANCE STATEMENT Associating relevant target stimuli with reward value can enhance their salience, facilitating their attentional selection. This value-driven salience improves behavioral performance, similar to the effects of physical salience. Recent theories, however, suggest that these forms of salience are intrinsically different, although the neural mechanisms underlying any such differences remain unclear. This study addressed this issue by manipulating the physical and value-related salience of targets in a visual search task, comparing their effects on several attention-sensitive neural-activity measures. Our findings show that, whereas physical salience accelerates the speed of attentional selection, value-driven salience selectively enhances its strength. These findings shed new insights into the theoretical and neural underpinnings of value-driven salience and its effects on attention and behavior.
Introduction
A rapidly expanding body of research has begun documenting how attention is biased toward reward-associated features, a phenomenon known as value-driven attentional capture (VDAC) (for reviews, see Anderson et al., 2011b; Chelazzi et al., 2013; Failing and Theeuwes, 2018). This biasing of attention tends to be involuntary and nonstrategic (Hickey et al., 2010; Hickey and van Zoest, 2012), which means its impact on behavior can vary substantially depending on the context. If the reward-associated feature is tied to a task-relevant parameter, then it can facilitate the shift in attention toward this item, improving behavioral performance (Raymond and O'Brien, 2009; Failing and Theeuwes, 2014). However, if it is tied to a task-irrelevant parameter, then it can draw attention away, ultimately impairing behavioral performance (Anderson et al., 2011b; Theeuwes and Belopolsky, 2012). This suggests that associating a stimulus with a reward enhances its pertinence to the visual system, increasing its overall salience in some key ways.
VDAC's variable impact on behavioral performance bears considerably similarity to that of a different modulator of attention: physical salience. Physical salience is generated from the distinctiveness of the stimulus's physical properties (e.g., color, shape) with respect to other objects in a scene (Theeuwes, 1994, 2010; Itti and Koch, 2001; Wolfe, 2007). These distinctive physical features capture attention quickly and automatically; targets imbued with high physical salience relative to its surrounding neighbors generate faster response times (RTs), and the onset of electrophysiological markers of visual selection occurs earlier (Luck et al., 2006; Töllner et al., 2011). Conversely, salient distractors can slow RTs and generate later neural markers of visual selection (Theeuwes, 1991, 1992; Hickey et al., 2006).
In summary, both physical salience and salience formed from reward value (i.e., value-driven salience) share many important characteristics. Cognitively, objects high in either type of salience appear to draw more attention, and their behavioral consequences are both dependent on the task relevance of the salient item. Although these similarities have been noted in early reports of VDAC research (Hickey et al., 2010), various theoretical accounts of these processes have argued that they are distinct phenomena (Awh et al., 2012), suggesting that there are key differences at some level in their underlying neural mechanisms. This idea is supported by findings that value-driven salience is related to fMRI activity not only within the visual cortex but in the striatum as well (Anderson et al., 2014; Hickey et al., 2015). A recent review highlighted that value-driven salience does involve an extended network of brain regions but also suggested that it should ultimately modulate attentional allocation in an analogous, bottom-up way just as if it were more physically salient (Anderson, 2019). Yet, to our knowledge, this view has yet to be specifically and directly tested.
Most prior investigations of value-driven salience or physical salience were designed to control for one form of salience while manipulating the other, so that changes in behavior could be appropriately attributed to the specific type of salience of interest. Consequently, few studies have compared the impact of these two forms of salience on attentional allocation. A handful of behavioral studies have begun investigating these differences and have generally concluded that both forms of salience have independent contributions to behavior (Anderson et al., 2011a; Wang et al., 2013; Gong and Liu, 2018). While this provides some support to the idea that the mechanisms by which these forms of salience do differ, the lack of corresponding neuroscience studies means that the exact differences in the underlying neural mechanisms remain unknown.
The current study was designed to directly compare the effects of physical salience and value-driven salience on visual selection, while measuring the underlying neural processes. To this end, we used a visual search task composed of uniquely colored pop-out targets in an array and manipulated both the physical salience and value-driven salience of the targets by varying the physical distinctiveness of the target color and its associated monetary reward value, respectively. Scalp EEG was recorded to provide high-temporal-resolution measures of the associated electrical brain activity, allowing us to gain insight into the underlying neural-processing mechanisms, including the key question as to whether these manipulations differentially affect the strength (amplitude) or speed (latency) of the attentional selection process, or both.
Materials and Methods
Participants
Twenty-eight healthy volunteers participated in the study. Of this set, 4 participants were rejected from further analysis due to excessive EEG/horizontal electro-oculogram (EOG) artifacts (as described in EEG acquisition and processing, below). This left 24 participants for further analysis (mean age ± SD: 23.4 ± 4.0 years; 11 female). Our sample size was based on the general standards in the field, particularly for value-driven effects on the attention-related N2pc component, studies of which generally have had 20-30 participants in the EEG dataset (e.g., Hickey et al., 2010; van den Berg et al., 2014; Oemisch et al., 2017). The behavioral studies that inspired this manuscript have also used similarly sized groups (Anderson et al., 2011a; Wang et al., 2013; Gong and Liu, 2018). Power calculations or prespecified stopping criteria a priori were not conducted for this study. All subjects gave their written informed consent to participate, and were paid $15 per hour plus a bonus ranging from $9 to $22 (mean ± SD: $14.0 ± 3.3) based on performance. This study was approved by the Duke Medical Center Institutional Review Board for human subjects. All participants were right handed, had normal color vision (Ishihara Test for Color Blindness) (Ishihara, 1917), and had normal or corrected-to-normal visual acuity.
Experimental design
Stimuli were programmed and presented using the Presentation software suite (Neurobehavioral Systems Inc., Albany, CA, www.neurobs.com). On each trial, a circular array of 42 ellipses was presented against a light gray background (Fig. 1A). Each array contained 41 distractor ellipses and one uniquely colored target ellipse. The stimuli's colors were manipulated in the Hue-Saturation-Lightness color space. All distractors were of the same hue at 180° (teal) in the Hue-Saturation-Lightness color space. The physical salience of targets was controlled by manipulating the target-distractor hue difference. For high-physical-salience targets, the target-distractor hue difference was 180°, and thus the target hue was 0° (or 360°, both red). For low-physical-salience targets, the hue deviated from the distractors by a relatively small amount, namely 36°, so that the hue of the target color was either 216° (blue) or 144° (green) (for the relative positions of the stimuli's colors in a schematic color wheel, see Fig. 1B). The color saturation of all stimuli was kept at 100%. The lightness of all colors was initially set as 100% and then adjusted using a flicker-fusion program (Simonson and Brožek, 1952) such that all colors were perceived as having the same level of luminance. This procedure equates the global sensory input energy level for each target color, removing any potential biases in any of the event-related potentials (ERPs) that are particularly sensitive to low-level sensory information (e.g., the N1; for previous applications of this technique in EEG studies, see Donohue et al., 2016; Harris et al., 2016). The target could appear randomly in 1 of 10 locations on the lower portion of the array, although never at midline locations (Fig. 1A).
Example stimulus display and illustration of target colors. A, Sample stimulus display. Multiple teal ellipses represent distractors. Red ellipse represents the color singleton target. Participants oriented attention to the target and discriminated its orientation (vertical vs horizontal) by pressing the corresponding button (in this example, “vertical”). B, Illustration of target colors. In the schematic color wheel, the distractor hue is 180°, high-salience target color is in the diametrically opposite position (0°, or 360°), and the two low-salience target colors are 36° apart from the distractor color to the left (216°) and to the right (144°), respectively.
The experimental session consisted of three phases: baseline, equal-reward, and selective-reward (Fig. 2). In the baseline phase, participants searched for a color-singleton target ellipse among an array of uniformly colored distractor ellipses. On each trial, the target could be of either high-physical-salience (red) or of low-physical-salience (blue or green). The search array was presented for 100 ms, followed by a 900-1100 ms fixation. Participants were instructed to report the orientation of the uniquely colored target ellipse as quickly and accurately as possible. The aspect ratio of all ellipses was 5:2, which is highly asymmetric; thus, the targets were easily discriminated as horizontal or vertical once identified as the target. Responses were made using the left and right index fingers on a game controller to indicate “vertical” or “horizontal” orientations, respectively. (This was thus an orthogonal discrimination relative to which side the singleton target was on.) No reward or reward information was given in the baseline phase, but the reward threshold for the next two phases was based on the RTs in the baseline phase (i.e., reward threshold = mean RT + SD of correct responses for that target color during the baseline phase). In the equal-reward and selective-reward phases, participants could earn reward points when they made a correct response faster than the reward threshold. Participants were explicitly told that the reward points were proportional to the speed of correct responses. The earned points would be accumulated and be converted to an extra cash bonus on the completion of the experiment. In the equal-reward phase, reward points could be earned for correct and fast responses to targets of any color. After each response, a written feedback informing the number of reward points earned on that trial (e.g., 0156) was presented below the fixation for 300 ms followed by a 200 ms fixation cross. Reward points were calculated based on the following algorithm: Reward points = Reward threshold (mean reaction time for each target type in the baseline phase) – the RT in the current trial. Participants also saw written feedback of “slow” or “wrong” following trials in which they failed to respond within the reward threshold or made the wrong button press, respectively. In the selective-reward phase, one of the low-salience target colors was selected as the high-value color. Participants could earn 10 times the reward points for a correct and fast response to targets in the high-value color (i.e., for high-value targets, Reward points = 10 × (Reward threshold – RT in the current trial)). Accordingly, there were three target types in the selective-reward phase: (1) high-physical-salience, low-value, (2) low-physical-salience, low-value, and (3) low-physical-salience, high-value. For half of the participants, the high-value target color was blue; for the other half, it was green. In both reward phases, once the participant earned another half dollar, a written notification of the accumulated reward amount (for example, “You've earned 2.5 dollars.”) was presented for 1500 ms, after the trial-by-trial feedback.
Experimental design in each phase. During each of the three phases, participants completed five blocks of 72 trials, resulting in 360 trials in each phase and a total of 1080 trials per participant. Within each block, the target position, color, and aspect orientation were randomly mixed. Feedback regarding accuracy and response latency was provided at the end of each block.
EEG acquisition and processing
Participants were seated ∼60 cm from a 24 inch monitor in a dimly lit, sound-attenuated, and electrically shielded room. EEG was recorded continuously using a custom extended-coverage elastic cap with 64 equally spaced active-electrode channels (actiCAP; Brain Vision), which covered the full head from slightly above eyebrows to below the inion (Woldorff et al., 2002). All electrodes were referenced to the right mastoid during recording, and electrode impedances were kept to <15 kΩ. EEG was recorded at a sampling rate of 500 Hz using a three-staged cascaded integrator-comb filter with a corner frequency of 130 Hz (actiCHamp, Brain Products GmbH, Gilching, Germany). A central white fixation-cross was visible at the center of the screen throughout each experimental block. Participants were instructed to maintain fixation on the central cross during the stimulus presentation. Eye movements were monitored with vertical and horizontal EOG channels and a closed-circuit zoom-lens camera.
All offline analyses were conducted using the ERPLAB Toolbox (Lopez-Calderon and Luck, 2014) (http://erpinfo.org/erplab/) and EEGLAB Toolbox (Delorme and Makeig, 2004). The EEG signals were bandpass filtered from 0.1 to 30 Hz using a noncausal Butterworth infinite impulse response filter with a 12 dB/octave roll-off. Independent component analysis (ICA) was used to correct for eye blinks. Sporadic artifacts, such as EMG bursts or strong electrode drifts, were manually rejected before running ICA. The ICA-corrected data were segmented into trials from 200 ms before stimulus onset to 800 ms after stimulus onset, and baseline-corrected relative to the 200 ms prestimulus period. Trials were automatically excluded if they contained an incorrect response, if the RTs were <200 ms or >1200 ms, if the EEG range exceeded ±60 µV, or if horizontal eye movements (horizontal electro-oculogram) exceeded ±20 µV. (Thresholds were minimally adjusted from these values for some participants.) The data were then rereferenced to the algebraic average of the left and right mastoids. Among the final set of 24 participants, artifacts led to the rejection of an average of 12.3% of trials (range = 0.9%-23.1%). None of the participants had residual eye movements that deviated >0.2° (i.e., horizontal electro-oculogram > 3.2 µV) toward the target after rejection criteria was applied (Lins et al., 1993).
Behavioral analyses
RTs to each target type were collected for each phase. Only correct trials were included in the RT analyses. Responses were considered accurate if the participant made a correct button press and the RT was between 200 and 1200 ms following the onset of the search array. All RTs reflect mean averages of raw, untransformed data.
ERP components
Because each behavior response reflects the outcome of processing in multiple cognitive stages, and change in behavioral responses can reflect modulation at any point in this sequence, measures such as RTs cannot provide direct examination into the cognitive stages affected by the experimental manipulations. For this reason, we also measured ERPs while participants were performing the task, whose high temporal resolution which allows us to be able to identify changes in discrete cognitive processing stages. In the current study, we focused on three ERP components: (1) the posterior N1, a negative-polarity wave (latency ∼100-200 ms), which reflects early sensory-evoked processing; (2) the posterior-contralateral N2pc component (latency ∼200-300 ms), which reflects the shift of attention to a lateralized stimulus location; and (3) the centro-parietal P3 (latency ∼350-550 ms), which is sensitive to the top-down controlled process of outcome evaluation. Prior investigations of physical and value-driven salience have largely suggested that both influence the attentional-shift-related N2pc (e.g., Hickey et al., 2006, 2010). However, we included the analysis of these other ERPs to first confirm that there were no low-level sensory influences driving our results, and to also understand how these attentional changes influenced processing later in the cognitive cascade. By comparing these ERP components elicited by different types of targets in the three phases, we assessed the effects that our salience manipulations had on the amplitude and/or latency of these ERP components, and thus on the cognitive processes they reflect.
N1
To examine early sensory processing, the N1 ERP component was measured at four parietal-occipital electrode sites around POz (see Fig. 4A). The onset latency of the N1 for each participant and each condition was determined as the time point at which the voltage reached 50% of the largest negative value found within a 100-200 ms post-target interval (Luck, 2014). It was necessary to slightly adjust this time window for 2 participants to maintain an accurate extraction of the N1 onset latency in each condition; this was achieved by slightly adjusting the window to 75-175 ms for 1 subject and 125-225 ms for the other. To estimate the N1 amplitude for each participant in each condition, the N1 peak latency was first extrapolated from the grand-average waveform of each condition within a 100-200 ms post-target interval. N1 amplitude was measured as the mean amplitude found within a ±20 ms time window from this peak latency in the grand-averaged waveform. The N1 amplitude was measured in this manner for each participant and each condition.
N2pc
The attentional-shift-related N2pc was derived from the difference between contralateral minus ipsilateral activity (relative to the target and thus to the attentional shift) (see Fig. 5). The ipsilateral waveform was computed as the average of the activity from the left-side electrodes when the target was presented to the left visual field and the right-side electrodes when the target was presented to the right. The contralateral waveform was computed as the average of the activities from the left-side electrodes to the targets in the right visual field and the right-sided electrodes to the targets in the left visual field. Analyses focused on activity within in the posterior electrodes PO7 and PO8, where N2pc activity tends to be largest (Luck and Kappenman, 2011). The onset latency and amplitude of the N2pc for each participant were measured in the same way as the N1, except that a 150-350 ms post-target interval was used.
P3
The P3 was measured at seven electrode sites around CPz (see Fig. 6). The amplitude of the P3 was calculated as the mean amplitude over the 350-550 ms latency window after target onset, for each participant and each condition. The latency of the P3 was estimated in each condition as the time point at which the voltage reached 50% of the largest positive value found within a 250-550 ms latency window. For 1 subject, it was observed that this window resulted in picking out a clearly incorrectly early peak of the P3 (i.e., it was detecting only a local positive “shoulder” on the positive-going deflection); thus, the window was adjusted to 300-600 ms for that subject. The P3's for another participant did not have an identifiable peak, and thus this participant was excluded from all the P3 analyses.
Statistical analysis
The RTs, ERP amplitudes, and ERP latencies for each target type were submitted to repeated-measures analyses ANOVAs. In order to understand the changes in our behavioral and neural measures across each new phase, we generated two 2 (phase) × 3 (target type) repeated-measures ANOVAS for each of our measures (e.g., RT, N1 amplitude, etc). The first repeated-measures ANOVA for each measure was conducted between the baseline and equal-reward phases, whereas the second repeated-measures ANOVA was conducted between the equal- and selective-reward phases. These two planned repeated-measures ANOVAs for each of our seven behavioral/neural measures resulted in 14 repeated-measures ANOVAs in total. We have organized the reports of our ERP analyses by each phase. In other words, we will first report how moving from the baseline to the equal-reward phase modulated each of our measures, and then we report the results from the equal- and selective-reward phases in a similar manner. Significance for each repeated-measures ANOVA was defined as p < 0.05. Any significant main effects of phase or target type were followed by Bonferroni-corrected pairwise comparisons. Significant interaction terms were assessed using post hoc paired-sample t tests where significance was Bonferroni-corrected to p < 0.025. All statistics were conducted using SPSS 24 (IBM, released 2016, RRID:SCR_002865).
Results
No differences between the two low-physical-salience targets in the baseline- and equal-reward phases were expected, given that they had equivalent physical salience and their reward values did not differ in those first two phases. In other words, none of the targets was rewarded in the baseline phase or they were all rewarded with an equal rate in the equal-reward phase. However, we measured and plotted responses (both behavioral and ERPs) to the two low-physical-salience targets to maintain a consistent structure for each statistical analysis throughout all three phases.
Behavioral results
The average RTs across subjects within each of the three phases are shown in Figure 3A, and the RTs specifically for each block are shown in Figure 3B. In the baseline phase, no reward information was provided, and we therefore expected similar RTs for the two low-physical-salience targets and faster RTs for the high-physical-salience targets, which is what we found: RTs to the two types of low-salience targets were nearly indistinguishable (Δ 1.15 ms, t(23) = 0.46, p = 0.649), and responses to the high-salience targets were significantly faster than either of the low-salience targets (Δ 23.85 ms, t(23) = 7.989, p < 0.001; Δ 22.70 ms, t(23) = 7.501, p < 0.001; respectively). These findings are consistent with what would be expected from successful manipulation of the physical salience of the targets.
Behavioral results of RTs. A, Average RTs across subjects in each phase. B, RTs separated out for each of the five blocks in each phase. Error bars indicate SEM. For full results, see Table 1.
In the equal-reward phase, all fast and correct responses were rewarded at an equal rate, regardless of targets' physical salience. RTs to the high-physical-salience targets were still faster than either of the low-physical-salience targets within that phase (Δ 21.91 ms, t(23) = 10.02, p < 0.001; Δ 20.61 ms, t(23) = 9.69, p < 0.001; respectively), and again no differences were observed between the two low-physical-salience targets (Δ 1.29 ms, t(23) = 0.60, p = 0.557). RTs between the no-reward/baseline phase and the equal-reward phase, analyzed using a 2 × 3 repeated-measures ANOVA (phase: baseline vs equal-reward) × 3 (target types) (Table 1), indicated a main effect of target type, driven by the significantly faster RT to the high-physical-salience target (Δ 22.88 ms, p < 0.001; Δ 21.66 ms, p < 0.001). However, there was also a main effect of phase, showing that participants responded faster overall in the equal-reward phase (Δ 70.29 ms, p < 0.001). No phase × target type interaction was found, indicating that the reward context speeded responses to all types of targets to the same extent.
Full results of the RT repeated-measures ANOVAs
Another possible contributor for the overall improvement in RTs from the baseline to the equal-reward phase is that we are observing a practice effect over time. A closer examination on the RTs in each block (Fig. 3B), however, ruled out this possibility: RTs decreased from the first to the second block in the baseline, showing a clear practice effect, but leveled off gradually over the later blocks in the baseline. Thus, the abrupt and substantial drop in RTs from baseline- to equal-reward phase was unlikely a practice effect, but instead the effect of reward as a general motivator for better performance in the equal-reward phase.
In the selective-reward phase, one of the low-physical-salience targets was associated with 10 times the reward value compared with the other targets. RTs to the low-physical-salience, high-value targets in this phase now became as fast as those to the high-physical-salience targets, such that the difference between them was no longer significant (Δ 4.85 ms, t(23) = 1.80, p = 0.086); moreover, the RTs for each of these two conditions were both faster than RTs to the low-physical-salience, low-value targets (Δ 13.46 ms, t(23) = 5.68, p < 0.001; Δ 18.31 ms, t(23) = 7.85, p < 0.001; respectively). Compared with the equal-reward phase, the 2 (phase: equal- vs selective-reward) × 3 (target type) repeated-measures ANOVA (Table 1) yielded a significant phase × target type interaction, confirming that selectively imparting high value (i.e., high reward) to physically low-salient targets boosted the target detection process such that the response to high-value (but low-physical-salience) targets was as fast as the high-physical-salience (but low-value) targets.
Electrophysiological markers
Early visual sensory processing: N1 effects
The first repeated-measures ANOVA on N1 amplitude (Fig. 4A,B; Table 2) revealed only a significant main effect of session phase, indicating an overall increase in N1 amplitude from the baseline to the equal-reward phase (Δ 0.68 µV, p = 0.008). The repeated-measures ANOVA on N1 latency revealed no significant differences (Table 2).
ERP traces and distribution of the N1 component. A, Top, ERP traces of the N1. Bottom left, Red dots indicate the posterior-occipital ROI of the four channels centered at POz that was used to assess for effects on the N1. B, Left, The mean and SEM of the N1 amplitude in each condition. There was a significant increase in amplitude from the baseline to equal-reward phase. Right, The mean and SEM of the N1 50% onset latency in each condition. There were no significant differences in either target type or phase. C, Scalp distribution of the N1 across phases. The increase in amplitude from the baseline- to equal-reward was centered on the chosen cluster of electrodes. The statistical comparisons indicated in brackets were derived from within-subject repeated-measures contrasts: ***p < 0.001. For full statistical results, see Table 2.
Full results of the N1 repeated-measures ANOVAs
The second repeated-measures ANOVAs on N1 amplitude (Table 2) and N1 latency (Table 2) yielded no significant effects. The general amplitude increase of the N1 component in the equal-reward phase suggests a generalized enhancement in visual sensory processing of the targets when there is an overall reward context, but there was no evidence that the selective high-value reward in the selective-reward phase differentially affected the early visual processing reflected by the N1.
Attentional allocation: N2pc effects
The first repeated-measures ANOVA of N2pc amplitude (Fig. 5A,B; Table 3) revealed a significant effect of phase, where the N2pc increased in amplitude in the equal-reward phase relative to the baseline phase (Δ 0.46 µV, p = 0.004). However, there was no main effect of target type or interaction term, indicating that this change in N2pc amplitude was similar for each type of target. The repeated-measures ANOVA on N2pc latency (Table 3) indicated a significant main effect of target type, where the onset latency of the N2pc was earlier for the high-physical-salience targets than either of the two low-physical-salience targets (Δ 31.75 ms, p > 0.001; Δ 28.13 ms, p > 0.001). There was also a significant effect of phase but no significant target type × phase interaction, indicating that the N2pc generally arose slightly earlier (Δ 6.81 ms, p = 0.001) for all target types in the equal-reward phase relative to the baseline phase. These results suggest that the reward context in the equal-reward phase equivalently enhanced the attentional allocation for all of the target type.
ERP traces of the attention-sensitive N2pc component and its distribution over the scalp in the selective-reward phase. A, Top, ERP traces of the N2pc. Bottom left, Red dots indicate the posterior-parietal ROI of the two channels used to measure the N2pc. B, Left, The mean and SEM of the N2pc amplitude in each condition. There was a general increase in amplitude between the baseline- and equal-reward phase, and the amplitude of the high-value target was further increased in the selective-reward phase. Right, The mean and SEM of the N2pc 50% onset latency in each condition. Although the onset latency for all target types became earlier as participants moved through each phase, the onset latency for the high-physical-salience target was consistently earlier than either of its two low-physical-salience counterparts. C, Distribution of N2pc-related activity over the scalp in the selective-reward phase. The statistical comparisons indicated in brackets were derived from within-subject repeated-measures contrasts: **p < 0.01; ***p < 0.001. For full statistical results, see Table 3.
Full results of the N2pc repeated-measures ANOVAs
The second N2pc amplitude repeated-measures ANOVA (Table 3) resulted in a significant main effect of target type as well as a signification interaction term. t tests revealed that the N2pc to the low-physical-salience targets now associated with high-value counterparts during the selective-reward phase was significantly larger than both the high-physical-salience target (Δ 1.06 µV, t(23) = 4.97, p < 0.001) and the low-physical-salience target associated with low value (Δ 1.17 µV, t(23) = 5.65, p < 0.001). We also performed an additional set of paired-sample t tests comparing the N2pc amplitude of like conditions between the equal- and selective-reward phases. There was no significant difference found between phases for the low-value target (Δ −0.09 µV; t(23) = −0.47, p = 0.640). There was a significant decrease in amplitude for the high-physical-salience target (Δ −0.54 µV; t(23) = −3.60, p = 0.002); however, there was an even larger increase in amplitude for the high-value-target (Δ 0.75 µV; t(23) = 3.84, p = 0.001), suggesting that this change was the primary driver of this interaction effect. The second N2pc onset latency repeated-measures ANOVA (Table 3) also had a main effect of target type and main effect of phase, suggesting that, while the N2pcs for all target types still arose faster in the selective-reward phase (Δ 5.08 ms, p = 0.001), the N2pc to the high-physical-salience targets was still occurring significantly earlier than both of the low-physical-salience targets (Δ 32.50 ms, p > 0.001; Δ 25.92 ms, p > 0.001). Importantly, there was no significant interaction term, suggesting that the selective changes in amplitude observed for the high-value target in the selective-reward phase did not impact the speed at which it occurred.
Target processing: P3 effects
The amplitude of the P3 (Fig. 6A,B; Table 4) did not differ between target types within either the baseline or the equal-reward phase, but there was a significant main effect of phase, suggesting an overall increase in P3 amplitude from baseline- to equal-reward phase (Δ 3.43 µV, p < 0.001). There was no significant interaction between target type and phase. The onset latency of the P3 (Fig. 6B, right) revealed a main effect of target type. This was driven by the relatively earlier onset latency for the high-physical-salience targets relative to the two low-physical-salience targets as indicated by a main effect of target type (Table 4), although this difference was only significant relative to one of the two targets (Δ 12.00 ms, p = 0.027) and not the other (Δ 7.57 ms, p = 0.180). There was no significant main effect of phase and no significant interaction term.
ERP traces of the P3 component and analysis of its amplitude and latency. A, Top, ERP traces of the P3. Bottom left, Red dots indicate the central-parietal ROI of the seven channels centered around site CPz that was used to assess the P3. B, Left, The mean and SEM of the P3 amplitude in each condition. There was a general increase in amplitude between the baseline and equal-reward phases. The amplitude of all targets additionally increased during the selective-reward phase but was further increased for the high-value target. Right, The mean and SEM of the P3 50% onset latency in each condition. The onset latency for the two low-physical-salience targets tended to be slower than the high-physical-salience target in the first two phases. In the selective-reward phase, the low-physical and low-value targets were significantly slower than both the high-physical and high-value target. The statistical comparisons indicated in brackets were derived from within-subject repeated-measures contrasts: *p < 0.05; **p < 0.01; ***p < 0.001. For full statistical results, see Table 4.
Full results of the P3 repeated-measures ANOVAs
P3 amplitude between the equal-reward to the selective-reward (Table 4) also resulted in a significant main effect of phase as well as a significant main effect of target type. In addition, the phase × target type interaction was also significant, suggesting that selectively increasing the value of one of the low-physical-salience targets also led to further increases in its P3 amplitude. This was confirmed through t tests revealing that, while no P3 amplitude differences were observed for the different target types in the equal-reward phase, the low-physical-salience, high-value targets in the selective-reward phase induced larger P3 than either the high-physical-salience, low-value (Δ 1.34 µV, t(22) = 2.57, p = 0.017) or the low-physical-salience, low-value targets (Δ 1.44 µV, t(22) = 3.06, p = 0.006). The second repeated-measures ANOVA on P3 latency generated a significant effect of phase, where the onset latency was shorter in the selective-reward phase (Table 4; Δ 8.58 ms, p = 0.018). There was also a significant main effect of target type, where the onset latency for the low-physical low-value target was significantly slower than both the high-physical target (Δ –13.04 ms, p < 0.001) and high-value target (Δ 8.26 ms, p = 0.005). There was no significant interaction term.
Discussion
In the present study, we tested whether physical salience and value-driven salience of a visual target modulate visual selection through different neural mechanisms, including examining what any such differences in these mechanisms might be. Behaviorally, when there was no reward, or when reward was equally assigned to all target types, responses to high-physical-salience targets were faster than those made to low-physical-salience targets. However, when high reward value was selectively associated with one of the low-physical-salience target colors, RTs for the high-value low-physical-salience targets became as fast as the RTs for those high-physical-salience targets. Given these similar changes in behavioral performance for both physical and value-driven salience, it is possible that they both might have accelerated visual selection through similar neural mechanisms. We tested this hypothesis by leveraging the high temporal resolution of ERPs, investigating several hallmark electrophysiological markers indexing the cognitive stages involved in the visual search task. If both forms of salience used the same neural mechanisms, then manipulating one form of salience while controlling for the other would still be expected to result in similar changes in each of these measures. Instead, we found clear evidence that physical and value-driven salience evokes clearly different changes specifically during attentional selection, indicating that these different forms of salience use different neural mechanisms to evoke similar behavioral outcomes, and providing insight as to what those differential mechanisms are.
Specifically, we first examined the bilateral sensory-evoked N1 ERP component, reflecting the sensory processing of the entire stimulus array. While we found no differences in the N1 between target types within any of the three phases, we did observe a significant increase in N1 amplitude across phases, particularly in going from the first to the second phase in which reward was introduced for all conditions. It seems likely that these effects were due to an overall increase in visual attention toward the stimulus array when the reward incentives were added to the task, reflecting increased overall motivation to perform well and obtain as much money as possible. This hypothesis is supported by recent evidence that the prospect of a reward can increase N1 amplitude (e.g., van den Berg et al., 2014). This effect may be mediated by the locus ceruleus-norepinephrine system, which modulates arousal (Berridge et al., 2012) and has been found to modulate the N1 and other ERPs in preparation for a task-relevant processes (Mückschel et al., 2017). However, this enhanced N1 amplitude was a general boost for all types of targets, with no selectivity effect found even in the selective-reward phase. Thus, this effect seems to reflect a global enhancement of the early sensory processing of the entire stimulus input, with no interaction with the factor of value-driven salience.
Next, we assessed the attentional-orienting-sensitive N2pc component. The N2pc can be taken as a temporal marker of the transition from the sensory coding of the whole stimulus display to the focal-attentional selection and analysis of the selected item (Woodman and Luck, 1999; Hickey et al., 2006). Previous studies have shown that the N2pc onset latency varies systematically as a function of physical salience. That is, the more a target physically differs from the surrounding stimulus items, the earlier the N2pc occurs, which cascades forward to yield faster RTs (Luck et al., 2006; Brisson et al., 2007; Zhao et al., 2011). In line with these previous findings, high-physical-salience targets elicited an earlier N2pc than low-physical-salience targets in all phases. This latency difference across the three phases was largely consistent and comparable with the RT differences between these two types of targets. For that reason, the impact of physical saliency on RT performance can be largely attributed to accelerating the speed of the attentional allocation to the targets, as reflected by the earlier N2pc component. In light of these findings, if the high-value targets in the selective-reward phase modulated attentional allocation through the same mechanisms as the physically salient targets, one would expect the N2pc latency for the high-value targets to become as early as for the high-physical-salience targets, paralleling the equivalent speeding up of the RTs. Instead, the N2pc latency for the high-value low-physical-salience targets was still significantly delayed compared with the high-physical-salience targets, indicating that, unlike physical salience, value-driven salience has little modulation on the speed of the attentional-shifting process reflected by the N2pc.
On the other hand, reward associations and value-driven salience did have a substantial impact on N2pc amplitude. The N2pc for all target types was significantly larger in the equal-reward phase relative to the baseline phase, similar to the nonspecific amplitude increases observed in the earlier-latency N1 when reward incentives were added. Of particular importance, however, was that, in the selective-reward phase, the high-value low-physical salience target substantially enhanced the N2pc amplitudes relative to both the other target types, including the physically salient ones, although its onset latency still remained significantly later than for those physical-salient targets. As this was the earliest point at which a specific difference for high-value salience was observed, it suggests that value-driven salience specifically creates a larger attentional selection response, but has relatively little effect on the speed at which it can occur. The selective effect on the latency of the attentional-orienting-sensitive N2pc was observed for physical salience, but the selective effect on N2pc amplitude by value-driven salience provides clear neural evidence that these two forms of salience modulate attentional selection via different neural mechanisms.
Lastly, we looked at the bilateral P3 component that occurs later in the processing cascade. There were no P3 amplitude differences between target types in the baseline phase, a global increase in P3 amplitude for all target types in the equal-reward phase, and further increase in the P3 amplitude selectively for high-value targets in the selective-reward phase. These findings are consistent with the notion that the P3 amplitude is sensitive to reward value (e.g., Yeung and Sanfey, 2004). Although the P3 latency results were not quite as robust, high-physical-salience targets tended to have an earlier P3 relative to the other target types in both the baseline and equal-reward phases, possibly reflecting the earlier differences in N2pc latencies. Additionally, the P3 onset latencies for the targets imbued with higher degrees of physical or value-driven salience in the selective-reward phase were earlier than the low-value low-physical-salience targets. This pattern of results is notable in that they mimic the RT differences in this phase. Together with the findings of the N2pc component, these P3 results suggest that the processing speed improvements by value-driven salience may begin during the attentional allocation stage but also continue to evolve later in the cognitive cascade.
On the whole, the current study indicates that the influence of physical saliency and value-driven saliency on visual attentional selection is accomplished via different neural mechanisms, although these different mechanisms can still lead to similar behavioral outcomes. While physical salience initiated attentional orienting more rapidly, as reflected by the earlier N2pc, increases in value-driven salience resulted in increases in N2pc amplitude but not its latency. This suggests that value-driven salience does not speed up the initial selection and orienting to the target stimulus, but instead enhances the strength of the attentional allocation. This greater attentional allocation is also coupled with changes in target processing, which eventually yield to improvements in RT. Thus, while physical salience and value-driven salience would appear to interact differently with attentional mechanisms, they can eventually still yield similar behavioral outcomes.
Top-down and bottom-up attentional influences are thought to be combined in a priority map that guides spatial attention (Bisley and Goldberg, 2010). Value-driven salience has also been theorized to compete with these influences in a similar manner (Awh et al., 2012; Chelazzi et al., 2013). This idea was recently elaborated on in a review article (Anderson, 2019) that suggested that value-driven salience was generated from the influence of the caudate nucleus onto the visual system. These reward-associated signals would continue to propagate until they reach the priority map in the lateral intraparietal sulcus; however, this review article also suggested that value-driven salience should exert similar influences as bottom-up influences derived from changes in physical salience (Anderson, 2019). While value-driven salience does clearly alter the spatial priority map and can result in similar behavioral outcomes, our work here shows that these attentional influences are not identical to those derived from changes in physical salience.
In conclusion, these results provide electrophysiological evidence that physical salience and value-driven salience use different mechanisms during attentional allocation, even as they lead to similar behavior effects, while providing insight into what these differential mechanisms appear to be. Thus, these results provide key neural evidence to support theoretical views that the attentional effects of value-driven salience are unique and dependent on a different set of neural mechanisms than those driving the effects of physical salience (Awh et al., 2012). The identification of value-driven salience's unique influence on the strength, rather than the speed, of attentional allocation may also help clarify some of the other unique properties of VDAC. This may be particularly useful in continuing to map out the rest of the neural mechanisms underlying VDAC, as this phenomenon may come about through more than one network of brain regions (Anderson, 2019). Thus, these findings provide important evidence regarding not only the theoretical nature of value-driven salience, but also the neural mechanisms that given rise to and govern this phenomenon.
Footnotes
The authors declare no competing financial interests.
This work was supported by National Institutes of Health R56-NS051048 to M.G.W.
- Correspondence should be addressed to Marty G. Woldorff at woldorff{at}duke.edu