Abstract
It is well known that top–down expectations affect perceptual processes. Yet, remarkably little is known about the relationship between expectations and conscious awareness. We address three crucial outstanding questions: (1) how do expectations affect the likelihood of conscious stimulus perception?; (2) does the brain register violations of expectations nonconsciously?; and (3) do expectations need to be conscious to influence perceptual decisions? Using human participants, we performed three experiments in which we manipulated stimulus predictability within the attentional blink paradigm, while combining visual psychophysics with electrophysiological recordings. We found that valid stimulus expectations increase the likelihood of conscious access of stimuli. Furthermore, our findings suggest a clear dissociation in the interaction between expectations and consciousness: conscious awareness seems crucial for the implementation of top–down expectations, but not for the generation of bottom-up stimulus-evoked prediction errors. These results constrain and update influential theories about the role of consciousness in the predictive brain.
SIGNIFICANCE STATEMENT While the relationship between expectations and conscious awareness plays a major role in many prediction-based theories of brain functioning, thus far few empirical studies have examined this relationship. Here, we address this gap in knowledge in a set of three experiments. Our results suggest that the effect of expectations on conscious awareness varies between different steps of the hierarchy of predictive processing. While the active use of top–down expectations for perceptual decisions requires conscious awareness, prediction errors can be triggered outside of conscious awareness. These results constrain and update influential theories about the role of consciousness in the predictive brain.
Introduction
A rapidly growing body of work indicates that sensory processing is strongly influenced by expectations we have about likely states of the world. Such expectations are shaped by the context in which we are operating, but also by learning, experience, and our genetic makeup (Friston, 2005; Bar, 2009; Summerfield and de Lange, 2014). Expectations are typically thought to originate from higher-level brain regions, such as the (pre)frontal cortex, which may guide information processing in lower-level sensory regions via top–down projections. In this framework, what we consciously see is proposed to be strongly influenced by the brain's expectations about, or its best guess of, the outside world (Gregory, 1980; Hohwy, 2012; Panichello et al., 2012). Initial studies support the idea that the brain uses information in the environment to build expectations of stimulus frequency or conditional probabilities to modify perceptual processing (Bar, 2004; Kok et al., 2012). These ideas have been formalized in theoretical models, such as predictive coding and sequential sampling models (Friston, 2005; Ratcliff and McKoon, 2008; Clark, 2013). Although these frameworks are attractive in their simplicity, how exactly expectations shape conscious perception and to what extent awareness guides the formation of expectations are still largely unknown.
At present, there are at least three issues that need to be resolved to further our understanding of the relationship between expectations and consciousness. The first issue relates to the effect that expectations may have on conscious awareness itself. It has been shown that valid expectations increase the speed of conscious access (Melloni et al., 2011; Pinto et al., 2015; Stein and Peelen, 2015; De Loof et al., 2016) and may help in selecting or facilitating stimulus interpretation when visual input is ambiguous or noisy (Bar et al., 2006; Denison et al., 2011; Panichello et al., 2012; Chang et al., 2015; Aru et al., 2016). It is yet an open question whether expectations can boost an otherwise unseen stimulus into conscious awareness, thereby enabling the switch from a nonconscious to a conscious stimulus representation, instead of merely facilitating its cognitive interpretation or making it appear sooner.
Second, it is an open question to what extent prediction errors, arising in a situation of invalid expectations, can be registered outside of conscious awareness. It has been shown that “oddball” stimuli (e.g., simple violations in auditory tone sequences) elicit early mismatch responses in electrophysiological signals: the mismatch negativity (MMN; Pöppel, 2004; Näätänen et al., 2007). Interestingly, MMNs can even be observed when attention is distracted from the tone sequences (Bekinschtein et al., 2009) or in several reduced states of consciousness, such as sleep (Ruby et al., 2008), anesthesia (Koelsch et al., 2006), and vegetative state (Bekinschtein et al., 2009). This suggests that the MMN reflects a preattentive nonconscious prediction error signal (Näätänen et al., 2001; Stefanics et al., 2011; Kimura and Takeda, 2015). However, it remains uncertain whether these signals originate in model-based comparisons of expectations to new input or merely reflect passive low-level sensory adaptation to repeated inputs (Garrido et al., 2009; Stefanics et al., 2016). The one study in which these mechanisms were dissociated in a nonconscious state showed adaptation remains operative during sleep, whereas prediction error detection disappears (Strauss et al., 2015), thus raising doubts about the notion that prediction errors may be registered nonconsciously.
The final issue concerns the role of awareness in implementing expectations. Many expectation-based models assume that expectations are implemented via top–down neural activation. Interestingly, influential theories of consciousness suggest that conscious access requires similar top–down interactions between higher-level (e.g., prefrontal) and lower-level (e.g., visual) brain regions, referred to as feedback or recurrent processing (Lamme and Roelfsema, 2000; Dehaene et al., 2006). Information that does not reach conscious access is thought to only trigger feedforward activity or local recurrent interactions between posterior brain regions. Therefore, it is unclear how nonconscious information, in the absence of feedback signals from higher-order cortical areas, could lead to the implementation of expectations.
Materials and Methods
Participants
We tested 26 participants in Experiment 1 (21 females; age, 19.5 ± 1.3 years), 85 participants in Experiment 2 (63 females; age, 22.0 ± 3.2 years), and 34 participants in Experiment 3 (27 females; age, 20.0 ± 1.1 years). All participants were right-handed and had normal or corrected-to-normal vision.
For all experiments, participants for whom the minimum number of observations was < 10 in any condition were excluded from analysis. Additionally, for Experiment 3 (EEG), we excluded two participants due to problems with the reference electrodes. In the end, this resulted in the inclusion of 25 participants for Experiment 1 (20 females; age, 19.5 ± 1.3 years), 67 participants for Experiment 2 (49 females; age, 21.9 ± 3.0 years), and 29 participants for Experiment 3 (22 females; age. 20.0 ± 1.1 years).
The studies were approved by the local ethics committee of the University of Amsterdam and written informed consent was obtained from all participants according to the Declaration of Helsinki. Compensation was €20 for Experiment 1, €25 for Experiment 2, and €30 for Experiment 3, or equivalents in course credit.
Materials
All stimuli were generated using the Psychophysics Toolbox (Brainard, 1997; RRID:SCR_002881) within a Matlab environment (MathWorks, RRID:SCR_001622). Stimuli were displayed on an ASUS LCD monitor (1920 × 1080 pixels, 120 Hz, 50.9 × 28.6 cm screen size, 46.3 pixels/°) on a “black” (RGB: [0 0 0], ±3 cd/m2) background while participants were seated in a dimly lit room, ∼70 cm away from the screen.
Procedure and stimuli
Participants performed an attentional blink (AB) task (Raymond et al., 1992), in which on every trial a rapid series of visual stimuli was presented consisting of a sequence of 17 uppercase letters drawn from the alphabet but excluding the letters I, L, O, Q, U, and V. No letter appeared more than once per trial. Letters were presented at fixation in a monospaced font (font size: 40 points; corresponding to a height of ∼1.2°) for 92 ms each.
Experiment 1.
Participants were instructed to detect target letters within the rapid serial visual presentation (RSVP). The first target (T1: G or H) was always presented at the fifth position of the RSVP. On most trials (80%) it was followed by a second target (T2: D or K) at Lag 2, Lag 4, or Lag 10 (respectively 183, 367, or 917 ms later). Each lag was equally likely. T1 was presented in green (RGB: [0 255 0]), while T2 and the distractor letters were white (RGB: [255 255 255]; ±320 cd/m2).
Crucially, there was a predictive relationship between the two targets (Fig. 1A). Namely, in the 80% of trials where a T2 was presented, the identity of T1 (e.g., G) predicted which T2 was likely (75%, e.g., D) or unlikely (25%, e.g., K) to appear. On the 20% remaining trials without a T2, a random distractor letter was presented at the T2 time point (every distractor letter was presented no more than once per trial.) The mapping of T1 and T2 was counterbalanced over participants, so that for half of the participants the most likely target combinations were G–D and H–K while for the other half G–K and H–D were most likely. To be able to distinguish different lags in the absence of a T2 stimulus, four gray squares (RGB: [200 200 200]; ±188 cd/m2; size: 0.35°; midpoint of each square centered at 1.30° from fixation) were always presented around the stimulus (T2 or distractor) at the T2 time point. Participants were instructed to use the timing information this cue provided when making decisions about the presence of a T2 (only the letters D or K; all other letters were distractors).
Task design and behavioral results of Experiments 1 and 2. A, The trial structure of the AB task used in Experiments 1 and 2. Each trial consisted of a stream of rapidly presented letters in which predefined target letters had to be detected and then reported at the end of the stream. The first target (T1: a green G or H) always appeared at the fifth position. The second target (T2: D or K) was presented at varying SOAs (lags) after the first one and was marked by placeholders. The identity of T1 predicted which of the T2 targets was most likely to appear, thereby introducing validly and invalidly predicted T2 targets. On 20% of the trials no second target was presented and a random distractor letter was presented instead. B, Percentage correct T2 target detection at each of the T1–T2 lags for valid expectations, invalid expectations, and T2-absent trials in Experiment 1. Validly predicted T2s were significantly more often perceived than invalidly predicted T2s. C, Percentage of T2 target detection at each of the T1–T2 lags after a valid or invalid expectation or on a T2-absent trial for Experiment 2. Again, validly predicted T2s were more often perceived, in particular at short lags. Error bars represent SEM.
Following a 150 ms blank period at the end of the RSVP, participants gave their responses. First, they indicated whether they had seen any T2 by pressing the left or right shift key on the keyboard. The mapping between the keys and the response options was randomized per trial to decouple participants' responses from the decision they had to make. Then they were asked to make a forced-choice judgment about the T2 letter (D or K) that was presented by typing in this letter. Finally, they made a similar response about the identity of T1 (G or H). We used long response timeout durations of 5 s and participants were instructed to value accuracy over response speed. The intertrial interval, as defined by the time between the last response and the onset of the stream, was 500–750 ms.
The experiment consisted of two 1 h sessions on separate days within 1 week. In the first session, participants received instructions about the task and subsequently performed the task for six blocks of 75 trials (total 300 trials). The goal of the training session was to familiarize participants with the task. Besides, since we did not instruct participants about the predictive relationship between T1 and T2, some practice on the task was required for them to (implicitly) learn this relationship. In the second session, participants first received a summary of the instructions, after which the actual experiment started. Participants performed six blocks of 90 trials (total of 540 trials) of the AB task. The first three participants performed six blocks of 105 trials (630 trials). In both sessions, participants received summary feedback about their performance at the end of each block, followed by a short break.
Experiment 2 (EEG).
The task in the EEG experiment was the same as in Experiment 1, except that in Experiment 2, we only asked participants to give one response by typing in the target letters they observed. In addition, we only used two different lags: Lag 3 (275 ms; two-thirds of trials) and Lag 10 (917 ms; one-third of trials). To further increase the number of trials, the intertrial interval range was reduced to 200–400 ms.
Again, the experiment consisted of two different sessions within 1 week. The first session (1 h) consisted of instructions followed by extensive training (720 trials over six blocks) on the task. Participants were not explicitly informed about the predictive relationship between the targets. In the second session (2 h), we first prepared the participant for the EEG measurements (see below) and gave brief instructions about the task. Then, participants performed 12 blocks of 120 trials (total, 1440 trials) of the AB task.
Experiment 3.
To investigate the importance of T1 detection for expectation effects on conscious access, we adjusted the task we used in Experiment 1 to decrease the visibility of T1 (see Fig. 4A). We now presented T1 in white instead of green to make it stand out less among the other stimuli. Furthermore, T1 duration was staircased per participant such that participants could report T1 on ∼75% of the trials. Starting in the second half of the training and continuing in the experimental session, T1 duration was decreased by one frame (8 ms) after each block if performance was >85% and increased by one frame if performance was <65%. To ensure T1 duration would not deviate too much from the duration of other stimuli, T1 duration was only allowed to be in the range of 42–142 ms (≤50 ms different from other stimuli). The median duration of T1 in the second session was 125 ms. On 20% of trials, no T1 was presented and a random distractor letter was presented instead. When both targets were present, T1 predicted which T2 was likely to follow with 75% accuracy.
We made a few changes to the task design to increase the efficiency of the design. The intertrial interval was reduced to values between 300 and 500 ms. In addition, we asked participants for only one response. They were asked to type in any target letter they had seen during the trial and refrain from typing in a T1 and/or T2 letter when they did not see any. The response was confirmed by pressing the space bar on the keyboard or when a timeout of 4 s had passed. To further increase the number of trials per condition, we decided to use only Lag 3 (two-thirds of trials) and Lag 10 (one-third of trials). Because T1 duration was staircased on an individual basis, the T1–T2 stimulus-onset asynchrony (SOA) differed between participants. On average, Lag 3 corresponded to an SOA of 308 ms while Lag 10 corresponded to an SOA of 950 ms.
Finally, we manipulated the instructions we gave to participants to see to what extent explicit knowledge of the relationship between T1 and T2 affected our results. As in Experiment 1, we tested participants during two separate sessions within 1 week. The first group of participants (N = 25) did not receive explicit instructions about the predictive relationships in either session. Thus, their instructions were similar to those given in Experiment 1. The second group of participants (N = 19) received explicit instructions about the T1–T2 relationship at the start of the second session, and a third group of participants (N = 23) received those instructions already at the start of their first session.
The first session (1 h) was used for instructions and training the participants on the task (10 × 75 trials). The experimental session in which participants performed the AB task lasted 1.5 h and contained 16 blocks of 75 trials (1200 trials).
Experimental design and statistical analysis: behavioral
Preparatory steps were done with in-house Matlab scripts. Statistical analyses (repeated-measures ANOVAs and paired t tests) were performed using JASP (Jeffreys's Amazing Statistics Program) software (Love et al., 2015; RRID:SCR_015823). In situations where a specifically tested hypothesis did not yield a significant result, we used a Bayesian equivalent of the same test to quantify the evidence for the null hypothesis (Rouder et al., 2012, 2017). In those cases, using JASP's default Cauchy prior, Bayes factors (BFs) were computed for each effect. To increase the interpretability in analyses with multiple factors, we used Bayesian model averaging to get a single BF for each effect in ANOVAs. This BF is the change from prior to posterior inclusion odds and can intuitively be understood as the amount of evidence the data gives for including an experimental factor in a model of the data. The BF will either converge to zero when the factor should not be included, or to infinity when it should be included in the model. Values close to one indicate that there is not enough evidence for either conclusion. We use the conventions from Jeffreys (1967) to interpret the effect sizes of our Bayesian analyses.
Experiment 1.
In our behavioral analyses, we looked at the T2-detection performance, given that T1 was correctly identified. A response was considered correct when (1) the participant indicated no T2 was present when no T2 was presented or (2) the participant correctly indicated a T2 was present and subsequently reported the correct target letter. Since expectation is only a meaningful concept when a T2 target was presented, the T2-absent trials, on which a distractor letter was presented instead of a T2, were not taken into consideration for the main statistical analyses. Trials where one of the responses was missing were deleted from all analyses. Percentage correct was used in a 3 × 2 repeated-measures ANOVA with the factors lag (Lag 2, Lag 4, Lag 10) and expectation (valid, invalid). In a control analysis, we repeated our analyses for Experiment 1 based on the T2-detection responses (ignoring the accuracy of the T2 identification) as dependent variable (see Results). Since the seen/miss response is orthogonal to the specific expectations about target letters, this analysis rules out simple response biases as a potential cause of our effects.
Experiment 2 (EEG).
The behavioral analyses for the EEG experiment were similar to those for Experiment 1. However, the factor lag had only two levels (Lag 3, Lag 10). Percentage correct T2 detection was computed as in Experiment 1 using only the trials on which T1 was correctly reported. A response was considered correct when the letter a participant entered was the letter that was presented or when a participant refrained from entering a letter when none was presented for the T2-absent trials. In addition, we computed a T2-detection measure to use in a control analysis: if a participant typed in any letter, we categorize the response as a “target seen” response, otherwise we call it a “target absent” response. This outcome measure was used in a control analysis.
Experiment 3.
In this experiment, participants gave only one response by typing in the target they had perceived. Trials on which no response was given or on which an impossible response was given (e.g., two T1 targets reported) were excluded from analyses. For T1 and T2 separately, we assessed the accuracy of the responses. The definition of correct and incorrect responses was the same as in Experiment 2 and we also used the same T2-detection measure.
Subsequently, T2 percentage-correct detection was used in a 2 × 2 × 2 × 3 mixed ANOVA with the within-subject factors lag (Lag 3, Lag 10), expectation (valid, invalid), and T1 visibility (T1 seen, T1 missed) and the between-subject factor instruction (none, Start Session 2, Start Session 1). As mentioned before, this between-subject factor was included to find out whether predictive effects would be modulated by explicit knowledge of the relation between T1 and T2. To investigate the effect of T1 visibility in more detail, we followed up the main analyses by other mixed ANOVAs in which we first split up the dataset based on T1 visibility. In situations where we found interactions with the factor instruction, we compared the effects of lag and expectation separately per instruction condition using repeated-measures ANOVAs and paired-sample t tests.
Finally, to test for an interaction between expectation validity and lag, we combined behavioral data from all experiments in a post hoc analysis. Only trials on which T1 was correctly identified were used. For Experiment 1 we averaged data for Lag 2 and Lag 4 to create an average “short lag” condition. Subsequently, these data were entered into a 2 × 2 × 3 mixed ANOVA with the within-subject factors lag (short, long) and expectation (valid, invalid) and the between-subject factor experiment (Experiment 1, Experiment 2, Experiment 3).
EEG measurements
EEG data were recorded with a BioSemi ActiveTwo system and sampled at 512 Hz (BioSemi). Potentials were measured from 64 scalp electrodes, along with two reference electrodes on the earlobes and four electrodes measuring horizontal and vertical eye movements. After data acquisition, EEG data were preprocessed with the FieldTrip toolbox for Matlab (Oostenveld et al., 2011; RRID:SCR_004849). First, data were re-referenced to the linked earlobes, high-pass filtered at 0.01 Hz, and epoched from −0.750 to 1 s surrounding the onset of T2. Data were visually inspected and trials and/or channels containing artifacts not related to eye blinks were manually removed, resulting in deletion of on average 9.1% (±3.9%) of trials and 2.0 (±1.7) channels. Independent component analysis was used to identify components related to eye blinks or other artifacts that could easily be distinguished from other EEG signals. After the independent component analysis, previously deleted channels were reconstructed based on a nearest neighbor approach. Trials were baseline corrected to the average amplitude before T1 onset (−0.750 to −0.275s). As a final step, we applied a 40 Hz low-pass filter to the trial data, after which event-related potentials (ERPs) were created separately for each condition of interest.
Experimental design and statistical analysis: electroencephalography
All EEG analyses are based exclusively on trials where T2 appeared at lag 3 and T1 was correctly identified. We used a combination of Fieldtrip (Oostenveld et al., 2011) and in-house Matlab scripts to perform our analyses. As a first step, we performed cluster-based permutation tests (Maris and Oostenveld, 2007) on the time-window 0–750 ms from stimulus onset to isolate significant ERP events relating to expectation validity (valid, invalid; regardless of T2 visibility) or T2 visibility (seen, missed; regardless of validity) or the interaction between these factors. Next, we used a Matlab script created in-house to isolate the significant events as clusters in time and space. For this purpose, we computed an average difference wave over all channels that were part of the cluster at any point in time. Subsequently, the onset and offset of a cluster were defined as the time period around the maximum difference where the difference did not drop below 50% of this maximum and where at least one channel showed a significant effect. We then selected the 10 channels that showed the largest effect in this time window. One of the observed events reflected a mixture of the traditionally observed P3a and P3b components (Sergent et al., 2005; Volpe et al., 2007). Therefore, we split the event into two clusters by manually selecting either the 32 most anterior or 32 most posterior EEG channels (from the central midline) before running the cluster-selection procedure.
As an alternative way to establish potential interactions between T2 detection and validity, we inspected in more detail the clusters isolated in the previous step. This may be a more powerful (but also less sensitive) way to detect small effects, because data are averaged over more time points and channels. Within each of the clusters, we performed a 2 × 2 repeated-measures ANOVA (and its Bayesian equivalent; see Behavioral analysis) with the factors T2 detection (seen, missed) and expectation validity (valid, invalid) on the cluster data averaged over channels and time. To prevent double dipping, in each analysis we only considered the effects orthogonal to the one used to define the cluster (e.g., not testing the effect of expectation in a cluster defined based on the expectation effect).
Results
Experiment 1: (how) do expectations affect conscious access?
In the first experiment we addressed the question of whether expectations about the likelihood of stimulus identity modulate the likelihood of conscious access and, if so, in what direction. To do so, we used the AB paradigm (Raymond et al., 1992). The AB is an impairment in the conscious perception of the second of two target stimuli presented in rapid succession when the initial target was correctly perceived. Here we modified the paradigm in such a way that the first target (T1: the letter G or H, in green) predicted which of the second targets would most be likely to appear in case a T2 target was presented (T2: the letter D or K; predicted, 75%; unpredicted, 25%; in white; Fig. 1A). On 20% of trials we presented a random distractor letter instead of a T2 target. At the end of each stream of letters, participants gave three responses. First, they indicated whether they had seen any of the two T2 targets (seen/unseen response). Second, they were prompted to make a forced-choice judgment about the identity of T2 (whether the letter D or K was presented). Third, participants had to make a similar forced-choice decision about the identity of T1 (whether the letter G or H was presented; see Materials and Methods). Participants were not explicitly instructed about the predictive relationship between T1 and T2.
In Figure 1 we plot the percentage of trials in which T2 was correctly detected and T1 discrimination was also correct (average T1 accuracy was 94.20%; SD = 5.77%) for the three different lags (Lags 2, 4, and 10). T2 was considered detected correctly when participants indicated they saw it (based on the first response) and correctly identified it (based on the second response). Overall, there was a clear AB, as reflected by reduced T2 detection when the time (i.e., lag) between T1 and T2 was shorter (Fig. 1B; main effect of lag: F(2,48) = 48.15, p < 0.001). Importantly, expectations modulated the T2-detection rate. T2 detection was significantly better when T1 validly predicted T2 (black lines) compared with when the expectation was invalid (gray lines, main effect of validity: F(1,24) = 7.10, p = 0.014; no significant interaction between lag and validity: F(2,48) = 1.30, p = 0.283). These results extend beyond findings of several previous studies (Melloni et al., 2011; Chang et al., 2015; Pinto et al., 2015; Stein and Peelen, 2015; Stein et al., 2015) by showing that conscious perception is partly determined by the transitional probability of the input the brain receives.
While these data support the notion that valid expectations trigger access to consciousness, it has been recognized that such findings may not be solely due to changes in perception, but perhaps are also due to changes in decision criteria or response biases (Gayet et al., 2014b; Yang et al., 2014; Attarha and Moore, 2015). To rule out the possibility that our effects could be explained by a response bias in which people simply report the target letter that they expected based on T1, regardless of whether they consciously perceived T2, we performed an analysis with T2 detection (instead of T2 discrimination; see Materials and Methods) as the dependent variable. This analysis takes into account only participants' first response (the seen/unseen response), regardless of whether the subsequent T2 letter identification was correct. Crucially, this analysis cannot be influenced by any decision/response biases because the response was orthogonal to the participants' expectation. Information about the most likely letter to appear cannot predispose participants to better determine whether a target letter was presented at all. Still, we observed a qualitatively similar pattern of results (main effect of validity: F(1,24) = 5.47, p = 0.028). This finding suggests that validity indeed boosted participants conscious access of T2, instead of merely eliciting a shift in response bias.
Experiment 2: EEG markers of conscious and nonconscious expectation violations
Subsequently, we tested whether expectation violations can be elicited by nonconsciously processed unpredicted stimuli or whether conscious perception of a stimulus is a prerequisite for it to trigger neural expectation error responses. To test this, we measured subjects' brain activity with EEG while they performed a similar task as in Experiment 1. First, we replicated the behavioral effects of Experiment 1 (Fig. 1C). Overall, T1 performance was high (mean = 93.61%; SD = 7.31%) and T2 detection was higher at Lag 10 than at Lag 3 (main effect of lag: F(1,28) = 128.72, p < 0.001), reflecting a robust AB. More importantly, validly predicted T2s were discriminated better than invalidly predicted T2s (main effect of validity: F(1,28) = 9.49, p = 0.005). The effects were similar in a control analysis where we considered the percentage of T2-seen responses (regardless of the exact letter participants entered), making it less likely that our effect can be explained by a response bias (main effect of validity: F(1,28) = 4.23, p = 0.049). In this experiment, the validity effect was significantly modulated by lag (validity × lag: F(1,28) = 5.86, p = 0.022), an effect that was numerically similar, but not significant in Experiment 1. Participants performed better on valid than on invalid trials at Lag 3, but there was no convincing evidence for an effect of expectations at Lag 10 (Lag 3 validity effect: t(28) = 3.40, p = 0.002; Lag 10 validity effect: t(28) = 0.98, p = 0.334). Thus, effects of expectations were larger in the time window in which T2 more often goes unperceived.
Next, we investigated potential differences in the neural processing of predicted and unpredicted stimuli, as a function of stimulus awareness. To this end, we contrasted invalidly and validly predicted T2s and tested this difference using cluster-based permutation testing, correcting for multiple comparisons across both time (0–750 ms) and (electrode) space (Fig. 2; see Materials and Methods; Maris and Oostenveld, 2007). We found one significant difference over frontocentral electrode channels, which reflected greater T2-elicited negativity for invalid compared with valid trials between 174 and 314 ms (p = 0.015; Fig. 2B), therefore potentially reflecting some type of mismatch response. We then further analyzed this event to test whether the difference was modulated by, or dependent on, conscious perception of T2. Crucially, the size of this frontocentral mismatch component was independent of T2 awareness (F(1,28) = 0.04; p = 0.850; BF = 0.254; Fig. 2C), indicating that both seen and unseen T2s generated a frontocentral mismatch response.
ERP effects related to T2 prediction validity. A, Topographic maps of the difference between validly and invalidly predicted T2s over time (0 corresponds to T2 onset). Cluster-based permutation tests were used to isolate the significant events, while correcting for multiple comparisons across time and (electrode) space. On each head map, channels with a significant effect for ≥50% of its time window are highlighted. B, The average ERP time course of the 10 channels shown on the head map on the left, shown separately for each validity condition. The significant time-window is marked by a black line above the x-axis. Invalidly predicted T2s were associated with greater frontocentral negativity than validly predicted T2s. C, Bar graphs showing the average amplitude of the four conditions (visibility × prediction) for the significant neural event shown in B. In all plots error bars represent SEM.
Additionally, analyses of T2 visibility effects (regardless of expectation validity) replicated previously reported findings (Kranczioch et al., 2003; Sergent et al., 2005; Harris et al., 2013). In these analyses, we examined the difference in ERPs following seen and missed T2s using a cluster-based permutation test (Fig. 3), revealing two significant events. First, a significant negative difference could be observed over (left) posterior electrodes from 170 to 355 ms after T2 onset (p = 0.010; Fig. 3A). This event was followed by a significant long-lasting positive event (p < 0.001), reflecting a mixture of the P3a and P3b components, extending over frontal and central electrodes.
ERP effects related to T2 visibility analyses. A, Topographic maps showing the difference between seen and missed T2s over time (0 corresponds to T2 onset). Cluster-based permutation tests were used to isolate the significant events while correcting for multiple comparisons across time and (electrode) space. On each head map, channels showing a significant difference for ≥50% of its time window are highlighted. Three events were isolated based on the permutation tests. B, D, F, For each of the events individually, the average ERP time course of the 10 channels shown on the head map on the left, separately for T2-seen and T2-missed conditions is shown. The significant time-window is marked by a black line above the x-axis. C, E, G, Bar graphs showing the average amplitude of the four conditions (visibility × prediction) for the significant neural events shown in B, D, F. In all plots error bars represent SEM.
Subsequently, we had a closer look at the interactions between conscious access and expectation validity. Therefore, we analyzed in more detail the ERP events isolated in the previous step (Fig. 3B–G). For this analysis we first isolated the traditionally observed AB-related P3a and P3b ERP components from the long-lasting positive ERP event that differentiated between seen and missed T2s (Sergent et al., 2005). Doing so resulted in an early positive P3a cluster (Fig. 3D) over frontocentral channels that was significant between 395 and 586 ms and a somewhat later positive P3b cluster (Fig. 3F) over more posterior parietal channels, which was significant between 445 and 611 ms. Within each of these clusters we performed repeated-measures ANOVAs with the factors validity and T2 detection. We found no evidence in any of the events that the T2 detection effect was modulated by expectation validity (early left-posterior event: F(1,28) = 0.29; p = 0.597; BF = 0.260; P3a: F(1,28) = 1.56; p = 0.222; BF = 0.230; P3b: F(1,28) = 2.10; p = 0.159; BF = 0.296), though the BF values suggest that the evidence for the absence of such interactions is moderate at best. This is somewhat surprising, because especially the late positive events have previously been related to conscious access (Sergent et al., 2005; Rutiku et al., 2015) and metacognition (Desender et al., 2016). However, recent investigations show they may reflect cognitive processing at even later stages, merely arising as a consequence of becoming consciously aware of information (Pitts et al., 2014; Silverstein et al., 2015). We did not find evidence that the amplitude of these ERP events was modulated by expectation validity, which may suggest that once a stimulus has been perceived consciously, it is irrelevant whether or not the expectation was valid.
Finally, we directly tested for an interaction between conscious access and expectation by comparing the validity ERP effect (invalid–valid) for T2 seen and T2 missed trials in a cluster-based permutation test (this analysis takes into account the entire scalp topography). Again, no significant interactions between these factors were observed (all cluster p's > 0.10).
Experiment 3: the role of conscious awareness in implementing top–down expectations
In our final experiment, we addressed the question of whether expectation formation itself can unfold in the absence of awareness and subsequently influence conscious access (Fig. 4). To address this question, we changed the color of T1 from green to white and for each subject staircased T1 duration in such a way that T1 was correctly identified on ∼75% of the trials (actual T1 identification performance: mean = 76.03%; SD = 8.65%). T1 duration did not differ between trials where T2 was seen and trials where T2 was missed (T2 detection: t(66) = 0.31, p = 0.752; T2 seen: mean = 117.42 ms; T2 missed: mean = 117.46 ms), which indicates that T1 visibility was not determined by stimulus duration. Likely, internal fluctuations in the system (e.g., variability in attention) must be causing participants to sometimes see T1 and sometimes miss it. Moreover, on 20% of trials no T1 was presented (but replaced by a distractor). Further, to test to what extent explicit knowledge of the predictive relationships between stimuli would increase the validity effects, we varied the moment at which explicit information about the predictive relations between T1 and T2 was provided. The experiment consisted of a training session and a test session on separate days. A first group of subjects received no explicit instructions about the predictive relations in either session and had to learn them implicitly through experience with the task; the second group received explicit instructions about the T1–T2 relations in the test session only, but not in the first training session; and the third group received explicit instructions already from the start of the experiment.
Task design and behavioral results of Experiment 3. A, Trial structure of the task used in Experiment 3. T1 visibility was staircased at ∼75% correct by manipulating its duration (on 20% of trials no T1 was presented). B, Percentage of correct T2 target detection at each of the T1–T2 lags after a valid or invalid expectation and on a T2-absent trials for trials where T1 was correctly reported (T1 seen). As in Experiments 1 and 2, when T1 was seen, validly predicted T2s were more often detected than invalidly predicted T2s. C, Solid lines show percentage of T2 target detection at each of the T1–T2 lags after a valid or invalid expectation and on T2-absent trials for trials where T1 was presented but missed. In contrast to T1-seen trials (B), when T1 was not seen, validity did not enhance T2 detection. However, a missed T1 still triggered a significant AB, compared with trials on which no T1 was presented (dotted line). Error bars represent SEM.
T1 visibility strongly affected T2 detection. When T1 was seen, T2 detection was markedly lower than when T1 was missed (main effect of T1 awareness: F(1,64) = 4.62, p = 0.035), in particular at short lags (T1 awareness × lag: F(1,64) = 72.95, p < 0.001). Validly predicted targets were detected more often (main effect of validity: F(1,64) = 33.39, p < 0.001). The effect of expectation validity on T2 detection varied as a function of T1 awareness and instructions (T1 awareness × validity: F(1,64) = 40.55, p < 0.001; validity × instruction: F(1,64) = 5.91, p = 0.004; T1 awareness × validity × instruction: F(2,64) = 11.33, p < 0.001). When T1 was seen (Fig. 4B), a clear AB was observed (main effect of lag: F(1,64) = 170.01, p < 0.001) and validly predicted targets were more often detected than invalidly predicted targets (main effect of validity: F(1,64) = 64.97, p < 0.001; as in Exps. 1 and 2). Like in the previous experiments, a control analysis considering only the percentage of T2-seen responses (regardless of the exact letter participants entered) also revealed a significant effect of validity (main effect of validity: F(1,64) = 65.83, p < 0.001), making it unlikely that response biases are causing the effect. Interestingly, we also observed a significant AB for missed T1s, reflecting a nonconsciously elicited AB (main effect of lag: F(1,64) = 74.42, p < 0.001). This AB effect cannot be explained by an overall T2-detection performance benefit for targets presented later in the trial because the AB was larger for trials on which T1 was presented but missed compared with trials on which no T1 was presented in the trial (lag × T1 presence: F(1,66) = 24.19, p < 0.001). However, although missed T1s triggered an AB, expectation validity did not affect T2-detection performance for missed T1s (main effect of validity: F(1,64) = 0.35, p = 0.554), regardless of the type of instruction participants received about the predictive relation between T1 and T2 (validity × instruction: F(2,64) = 0.64, p = 0.533). A Bayesian equivalent of the repeated-measures analysis strongly suggested validity should not be included in a model of the data (BF = 0.084).
The above results highlight that only when T1 was seen, valid expectations facilitated T2 detection. A post hoc analysis on T1-seen trials only revealed that this effect was modulated by how explicitly we instructed participants about the predictive relationship between T1 and T2 (validity × instruction: F(2,64) = 14.83, p < 0.001). The validity effect, as defined by the difference between valid and invalid trials, averaged across the two lags, increased with more explicit instructions (Group 1: 1.87%; Group 2: 19.53%; Group 3: 26.27%). These results indicate that, not only does the visibility of T1 define the predictive impact on T2 detection, but also the extent to which these predictive relations are explicitly known affects the impact of expectations on conscious access. This may also explain why the validity effect appeared more pronounced in Experiment 3 than in Experiments 1 and 2: in Experiments 1 and 2 subjects were not explicitly instructed about the predictive relations between T1 and T2.
Finally, in contrast to Experiment 2, on T1-seen trials the validity effect was independent of lag (validity × lag: F(1,64) = 1.750, p = 0.191). Since we anticipated stronger expectation effects at short lags, behavioral data from all three experiments was combined in a post hoc analysis. Only trials on which T1 was correctly identified were used and for Experiment 1 we averaged data for Lag 2 and Lag 4 to create an average “short lag” condition. A significant interaction between validity and lag showed that across all experiments, the expectation effect was stronger at short lags compared with the long lags (validity × lag: F(1,118) = 5.73, p = 0.018; no validity × lag × experiment interaction: F(2,118) = 0.065, p = 0.937).
Discussion
In this report we investigated three important questions regarding the intricate relationship between top–down expectations and conscious awareness. The first question that we addressed was how prior information about the identity of an upcoming stimulus influences the likelihood of that stimulus entering conscious awareness. Using a novel AB paradigm in which the identity of T1 cued the likelihood of the identity of T2, we showed that stimuli that confirm our expectation have a higher likelihood of gaining access to conscious awareness than stimuli that violate our expectations, especially at short lags. The expectation effect was qualitatively similar across all three experiments, regardless of subtle experimental differences in task design and overall performance between those experiments. Furthermore, it could not be explained by simple shifts in the response criterion, because it was also present for a dependent measure orthogonal to the expectation manipulation. Together, this suggests that valid expectations amplify the perceptual strength of a stimulus and therefore increase the chance of conscious access, possibly due to the sharpening of its neural representations (Kok et al., 2012). This interpretation is supported by previous experiments that have shown varying effects of expectations on (subjective) perception, such as studies showing that prior knowledge increases the speed (Melloni et al., 2011; Chang et al., 2015; Pinto et al., 2015; De Loof et al., 2016) and accuracy (Stein and Peelen, 2015) of stimulus detection. Furthermore, our findings complement recent studies showing that the AB can be reduced when there is knowledge about temporal statistics of the task (Lasaponara et al., 2015; Visser et al., 2015) or when the latency of T2 targets is explicitly cued (Martens and Johnson, 2005; Nieuwenstein et al., 2005). In addition, two recent reports have shown that when attention is diverted, in some subjects expecting the presentation of a stimulus can elicit an illusory stimulus percept even though no stimulus is presented. (Mack et al., 2016; Aru and Bachmann, 2017). Future experimentation is required to shed light on the generalizability of our effect to simpler tasks. Such experiments may also consider using other measures of subjective perception (e.g., perceptual awareness scale; Overgaard et al., 2006).
The second question we addressed was related to the extent to which nonconscious stimuli can trigger prediction error responses, as measured with EEG. Over the last 20 years, we and others have shown that nonconscious information processing is rather sophisticated (Dehaene and Changeux, 2011; van Gaal and Lamme, 2012), and that a diverse range of high-level cognitive processes can unfold nonconsciously (Dehaene et al., 2001; Custers and Aarts, 2005; Lau and Passingham, 2007; Pessiglione et al., 2007, 2008, van Gaal et al., 2010, 2012, 2014). Interestingly, in Experiment 2 we found that expectations violated by a nonconscious stimulus trigger a stronger negative frontocentral ERP component than expectations that are confirmed. This neural event was similar for trials on which T2 was seen and on trials where T2 was missed, highlighting that conscious awareness of a stimulus is not a prerequisite for it to trigger a prediction error response (Mathews et al., 2014; Malekshahi et al., 2016). This effect may reflect a mismatch signal, similar to the MMN (Näätänen et al., 2007), which is a negative deflection following oddball stimuli that develops 100–200 ms after stimulus onset. Sometimes this effect lasts longer, in some experiments until ∼400 ms, depending on the specifics of the task and stimulus material (Pöppel, 2004; Stefanics et al., 2011; Kimura and Takeda, 2015). While in terms of interpretation this effect is similar to a mismatch effect, its topography is slightly different than a typical visually evoked MMN, which generally peaks more posteriorly, although considerable variation in its topography has been reported (Pöppel, 2004). Alternatively, it is possible that the higher activation for valid compared with invalid trials corresponds to the frontal selection positivity, which is a well known marker of nonspatial attentional processes (Kenemans et al., 1993). In our paradigm, this could be explained as improved attentional selection when expectations are confirmed. Although the exact nature of the observed component deserves future experimentation, the key finding is that the effect was independent of T2 perception and purely depends on the validity of the expectation. This is in line with studies that have shown context influences on nonconscious information processing (Nakamura et al., 2007; Van Opstal et al., 2011; Gayet et al., 2014a; Rohaut et al., 2016), studies that have shown that the MMN can be observed when the expectation violations are unattended (Bekinschtein et al., 2009; Stefanics et al., 2011; King et al., 2013; Dykstra and Gutschalk, 2015; Kimura and Takeda, 2015), and more generally evidence for relatively high-level processing of nonconscious stimuli (Luck et al., 1996; van Gaal and Lamme, 2012; Silverstein et al., 2015). Nevertheless, the absence of interactions in the ERP is also somewhat surprising (but see Rutiku et al., 2016), because as noted earlier such interactions between expectation validity and conscious T2 detection were present in behavior. A neural basis for this effect should exist, but may be very subtle. Recently, a study by Aru et al. (2016) found early (<100 ms) differences in signal amplitude over posterior channels that predicted the behavioral benefit of prior knowledge on the detection of stimuli presented at the threshold of perception. Another potentially interesting signature to investigate could be the onset of components related to conscious perception (Melloni et al., 2011) and how they relate to expectations. Moreover, it is possible that instead of signal strength, it is the signal-to-noise ratio or sharpness of the representation that is improved (Kok et al., 2012). Possibly, valid expectations do not modulate the amplitude of the neural response, but instead increase the specificity of the neural representation.
In the final experiment, we showed that conscious perception of T1, initiating the expectation, is a prerequisite for influences on conscious access to occur. In the subset of trials where subjects did not see T1, there was no expectation effect on T2-detection performance. This result contrasts with findings from a recent study that suggested that some priors may operate nonconsciously (Chang et al., 2016). Chang and colleagues presented participants with masked gray-scale natural scene images and found that the nonconscious processing of these images improved subsequent recognition of their degraded counterparts, so-called “Mooney images,” presented seconds later. One explanation for this difference is that the priors on which the effects of Chang et al. relied may be more automatic and hard-wired than the relatively arbitrary relationships that people must learn and actively use in our experiments. It is possible that lower-level, automatic expectations are more easily processed outside of awareness compared with the more active ones studied here.
Further, it is also possible that with more training we would find nonconscious expectation effects. However, since subjects were already trained on the task on a separate day before performing the experimental session, this possibility seems unlikely. We did observe greater validity effects when subjects were made explicitly aware of the predictive nature of T1, suggesting that explicit knowledge of stimulus associations can facilitate the effects of stimulus-induced expectations. Finally, it should be noted that we did not test the full range of timing intervals between T1 and T2. It has been shown and proposed that the processing of nonconscious stimuli is relatively fleeting (Greenwald et al., 1996; Dehaene et al., 2006; but see King et al., 2016), so it is conceivable that the T1–T2 lags that we have used here may have been too long to observe expectation effects triggered by unseen T1s. Nevertheless a significant AB was observed on trials on which T1 was missed, indicating that attention was still captured by a missed T1 at the T1–T2 lags used here. This latter result is in line with evidence showing that nonconscious stimuli are able to trigger attentional capture (Ansorge et al., 2009; Mulckhuyse and Theeuwes, 2010; Hsieh et al., 2011) and with a study showing lower T2 detection for T1s that were missed compared with trials without a T1 [in that experiment this effect was independent of lag (Nieuwenstein et al., 2009)].
In summary, three main conclusions can be drawn from the present series of studies. First, expectation confirmation, compared with violation, increases the likelihood of conscious awareness, suggesting that valid expectations amplify the perceptual strength of a stimulus. Second, nonconscious violations of conscious expectations are registered in the human brain. Third, expectations need to be implemented consciously to subsequently modulate conscious access. These results suggest a differential role of conscious awareness in the hierarchy of predictive processing, in which the active implementation of top–down expectations requires conscious awareness, whereas a conscious expectation and a nonconscious stimulus can interact to generate prediction errors. How these nonconscious prediction errors are used for updating future behavior and shaping trial-by-trial learning is a matter for future experimentation.
Footnotes
The authors declare no competing financial interests.
This work was supported by the Netherlands Organization for Scientific Research (NWO VENI 451-11-007 awarded to S.v.G.; NWO VIDI 452-13-016 awarded to F.P.d.L.), the European Research Council (ERC-2015-STG_679399 awarded to H.A.S.) and the James S. McDonnell Foundation (Understanding Human Cognition, 220020373, awarded to F.d.L.). We thank Doris Dijksterhuis, Sjoerd Manger, and Thomas Dolman for their valuable assistance with data acquisition. We thank Timo Stein and Josipa Alilovic for valuable comments on a previous draft of this manuscript.
- Correspondence should be addressed to Simon van Gaal, University of Amsterdam, Department of Psychology, 1001 NK Amsterdam, the Netherlands. simonvangaal{at}gmail.com
