Figures
Abstract
There is ample evidence to show that many types of visual information, including emotional information, could be processed in the absence of visual awareness. For example, it has been shown that masked subliminal facial expressions can induce priming and adaptation effects. However, stimulus made invisible in different ways could be processed to different extent and have differential effects. In this study, we adopted a flanker type behavioral method to investigate whether a flanker rendered invisible through Continuous Flash Suppression (CFS) could induce a congruency effect on the discrimination of a visible target. Specifically, during the experiment, participants judged the expression (either happy or fearful) of a visible face in the presence of a nearby invisible face (with happy or fearful expression). Results show that participants were slower and less accurate in discriminating the expression of the visible face when the expression of the invisible flanker face was incongruent. Thus, facial expression information rendered invisible with CFS and presented a different spatial location could enhance or interfere with consciously processed facial expression information.
Citation: Ye X, He S, Hu Y, Yu YQ, Wang K (2014) Interference between Conscious and Unconscious Facial Expression Information. PLoS ONE 9(8): e105156. https://doi.org/10.1371/journal.pone.0105156
Editor: Marcello Costantini, University G. d'Annunzio, Italy
Received: December 5, 2013; Accepted: July 21, 2014; Published: August 27, 2014
Copyright: © 2014 Ye et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The research reported in this manuscript was supported by National Natural Science Foundation of China (Grant 91232717; 81171273; 81123002), Chinese Academy of Sciences grant XDB02050001. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Over the past several decades, experimental psychologists have demonstrated that a remarkably wide range of visual information processing can occur outside of awareness, which in turn significantly influence various aspects of human behavior [1]–[3]. In particular, it has been shown that emotional information (e.g., fearful, disgust, or happy expressions) from the face or the body can be processed in the absence of visual awareness [4]. Neuroimaging studies using backward masking [5], [6], binocular rivalry [7]–[9] or Continuous Flash Suppression [10], [11] have demonstrated that the haemodynamic responses of the amygdala, superior temporal sulcus face sensitive area, and anterior cingulate can be elicited by emotional stimuli rendered invisible.
Additionally, emotional stimuli can modulate ongoing cognitive processes [12], [13]. Subliminal affective responses to stimuli can be elicited and implicitly affect our perception of certain context, influencing our choices in approach-avoidance behaviors [14]–[17]. A number of studies using subliminal affective priming stimuli have shown the influence by subliminal presented prime on response toward supraliminal target including pleasant and unpleasant words [18], [19], happy and sad/angry facial expression [14], [16], [17], [20] and positive/ negative pictures [21], [22]. Also, exposure to subliminal prime with words related to thirst increased participants' beverages consumption [15].
Binocular rivalry, the alternating percept between two eyes' images when different images are presented dichoptically, can be used to study unconscious visual information processing. Previous fMRI studies demonstrate that during binocular rivalry suppressed emotional faces can be processed and generate stronger amygdala activity compared with neutral stimuli [8], [9]. A relatively new procedure modified from binocular rivalry called Continuous Flash Suppression (CFS) is a more controlled way to render visually presented stimuli invisible [23], [24]. By exposing one eye to continuously flashing contour-rich and high-contrast random patterns, CFS makes a visual stimulus invisible for a relatively long period of time [23], and has become a very effective technique for investigating the mechanisms of unconscious visual information processing. With faces suppressed by CFS, responses to suppressed fearful faces remain robust in the amygdala and the superior temporal sulcus [10]. In behavioral studies, when initially suppressed, upright faces broke interocular suppression faster than inverted faces [25], and fearful faces faster than neutral faces [26]–[28]. A number of recent studies indicate that the exposure to an emotional stimulus rendered invisible under CFS can evoke adaptation effects and influence the consequent reaction [29], [30]. However, some have questioned whether these effects could be attributed to the emotional level analysis or simply due to the image features that make stimuli more like faces [31].
Previous studies have shown that unconsciously processed expression information could modulate the processing of subsequently presented images [29], [30]. The study by Jiang et al. shows that a back-ward masked emotional face could prime and interfere with affective judgment of a subsequently presented visible face, demonstrating that the emotion conflict could be trigged by backward-masked emotional information [32]. Given that back-ward masking and CFS could have different consequences [33], in this study, we investigated the interaction between consciously perceived and unconsciously processed emotional information, using CFS to render images invisible. The current study focused on the potential influence of an invisible facial expression on the expression judgment of a spatially non-overlapping visible emotional face. If the unconscious expression information processing could influence the judgment of consciously perceived facial expression, then we would expect to see a congruency effect - the reaction time would be shorter and accuracy higher when the expressions of the visible and invisible faces are incongruent than when they are congruent. Thus we investigated this issue by presenting a target face visibly on one side of the fixation and presenting another face on the other side of the fixation, but rendered invisible using CFS. The results of our study show a robust congruency effect, suggesting that unconsciously processed information could enhance or interfere with consciously processed information, even when they are spatially non-overlapping.
Methods
Participants
Twenty-eight participants, (22.3±1.82 years old, 12 male) right-handed, healthy native Chinese-speaking students at Anhui Medical University, participated in this experiment. All participants were naive to the purpose of the experiment. All had normal or corrected to normal visual acuity, and no history of neurological trauma or psychiatric disorders. All participants were paid for their participation and gave written informed consent in accordance with the procedures and protocols approved by the Human Participants Review Committee of Anhui Medical University.
Stimuli
Photographs of four male and four female faces taken from 8 Chinese actors were assembled for use in the present experiment. Each face was photographed against a gray background in full frontal view (no glasses, jewelry, or paraphernalia). Each of these 8 faces was photographed twice, once with happiness expression and once with fear expression, for a total of 16 photographs. The individuals reported in this manuscript have given written informed consent (as outlined in PLOS consent form) to publish these case details.
Stimuli were generated with MATLAB (www.mathworks.com) and presented on a Lenovo 19-inch LCD monitor (set at 1,440*900 pixel resolution and a refresh rate of 60 Hz) using the psychophysical toolbox (www.psychtoolbox.org; Brainard, 1997). The left and right eye's images were displayed side-by-side on the monitor and viewed through a mirror stereoscope mounted in front of a chinrest. A frame (12.4° *12.4°) that extended beyond the outer border of the stimulus and fixation point was presented to facilitate stable convergence of the two eyes' images. The viewing distance was 43.5 cm.
Procedure
Stimuli were displayed against a gray background. During the whole experiment two black-and-white frames (12.4° *12.4°) were presented side by side on the screen, such that one frame was visible to each eye. In the center of each frame a red central fixation cross (0.66° *0.66°) was displayed. Participants were asked to maintain stable fixation throughout each experiment block. In all experiments, face stimuli were 8 happy face photographs (four female, subtended 3.3° * 4.0° of visual angle) and 8 fearful face photographs (four female, with the same visual angle as happy faces).
The main experiment was divided into two phases: a block of the training phase followed by three blocks of the test phase. Each trial started with a 1.5 s presentation of the fixation cross and the black-and-white frame only. Then a pair of colored, high-contrast, Mondrian-like CFS masks (similar to those used by Jiang et al. [25] and Sterzer et al. [34]) measuring 3.3°*4.6° were flashed to the participant's dominant eye (determined as the eye with longer dominance time based on a quick binocular rivalry test using gratings) at a frequency of 15 Hz, while a happy or fearful face (distractor stimulus) subtending 3.2° * 4.0° of visual angle was introduced to the nondominant eye (see Figure 1, The horizontal distance between the centers of this pair of masks was 6.6°). The face was presented either to the left or to the right of the fixation cross, at a random location within the area corresponding to the location of the CFS masks. The contrast of the suppressed face stimulus was ramped up linearly from 0 to 40% of its original contrast within a period of 600 ms from the beginning of the trial and then remained constant until the participant had made a response. At the time when the suppressed face reached the stable contrast (following 600 ms of contrast ramping), a target face image (happy or fearful) was presented on the other side of the fixation cross, binocularly, thus visibly. The target face was presented for 200 ms before it disappeared. The participants were asked to decide whether the face was happy or fearful as quickly and accurately as possible by pressing one of two buttons. Reaction time was calculated as the time from the onset of the target face to the time of the button press. Participants were asked to press another key in each trial if they saw any part of the suppressed face (failure of suppression), and such trials were excluded from further analysis.
The suppressed face image was presented to one eye with its contrast gradually ramped up, and was expected to be suppressed by the dynamic noise. The contrast of the suppressed face image was ramped up linearly from 0 to 40% of its original contrast within a period of 600 ms from the beginning of the trial and then remained constant until the participant had made a response. When the suppressed face reached a stable contrast, the target face image (happy or fearful) was presented on the other side of the fixation cross to both eyes, thus visibly for 200 ms before it disappeared. Participants were asked to respond, as quickly as possible, to the target face image, by pressing one of two keys to indicate whether it was happy or fearful.
The order of the faces was pseudo-randomized so that no more than four faces with the same affective valence were presented consecutively. The three test blocks each consisted of 80 trials. Participants took 3 min rest before starting the next block.
Results
The expressions (happy/fearful) depicted in the subliminally presented stimulus and in the visible target were either congruent or incongruent. We analyzed the data based on the congruency of the expressions. Results showed that a vast majority of the 28 participants had shorter response time for the congruent than the incongruent condition. Reaction times were calculated from the correct trials only. The average response time in the congruent condition was significantly shorter than that in the incongruent condition (427 ms vs. 448 ms), t(27) = 3.13, p<.005 (Fig. 2A). In addition, participants were more accurate judging the expression of the target face when the invisible distractor had a congruent than incongruent expression (71.31% vs. 63.73%), t(27) = 2.32, p<.005 (Fig.2B). In addition to the independent objective measure of the suppression effectiveness (described below), we also had an indication of the effective suppression from participants' reports during the main experiment, since the flanker face broke suppression in only 1.5% of trails. These results demonstrate that unconsciously presented emotional information at one location influenced the judgment of consciously processed expression information at a different spatial location.
A. mean reaction times in the congruent and incongruent conditions, showing that the reaction time is significantly slower in the incongruent than that in the congruent condition (448 ms vs. 427 ms, t(27) = 3.13, p<.005); B. mean accuracy for the congruent and incongruent conditions, showing that participants responded more accurately in the congruent than in the incongruent conditions (71.31% vs. 63.73%, t(27) = 2.32, p<.005).
Objective Measures of the Suppression Effectiveness
Because of the importance of keeping the suppressed image invisible during the test phase, in addition to asking the participants to report during the congruency experiment if the suppressed image became visible (including partially visible), we also performed an objective measure of suppression effectiveness following the main experiment. Sixteen of the participants (8 male, 8 female) completed a two-alternative forced choice (2AFC) task. The stimulus parameters in this 2AFC experiment were the same as those in the main experiment. Because in the main experiment the suppressed stimuli disappeared when the participant made a judgment about the target and pressed the key, we set the presenting duration for the suppressed stimuli in the 2AFC experiment according to the mean response time from the main experiment. During each trial, participants were asked to make a forced choice response indicating on which side of the fixation the face image was presented. The results showed that none of the participants performed significantly different from chance as demonstrated by binominal tests (p>0.05 for all 16 participants), with a mean correct percentage of 0.496±0.010 (mean ±SEM, t(15) = 0.3454, p>0.7). Thus the 2AFC objective test provided additional assurance that participants were not aware of the suppressed stimuli (see Figure 3).
Proportions of correct responses for 16 participants in a two-alternative forced choice task in determining on which side a face image was presented while the face image was suppressed in the same way as in the main experiment. None of the participants performed significantly different from chance, as demonstrated by binominal tests (p>0.05 for all 16 participants). Error bars in the mean performance are 95% confidence intervals.
The above control experiment indicated that participants were not aware of which side the suppressed face was presented. In another set of 16 participants, we further assessed whether participants were able to make judgments about the valence of the suppressed face. During each trial, participants were asked to make a forced choice response indicating on whether the face was happy or fearful by pressing one of two buttons. The results showed that none of the participants performed significantly different from chance, as demonstrated by binominal tests (p>0.05 for all 16 participants) with a mean correct percentage of 0.4858±0.0079(mean±SEM, t(15) = 1.793, p>0.05). Thus results from the two control experiments give us high confidence that participants did not perceive the suppressed face nor did they have an explicit representation of the valence of the suppressed face.
Since the objective awareness experiment (location judgment) was run on 16 out of the participants in the main experiment, and there is the possibility that the congruency effect observed in the main effect was primarily due to the trials in which the suppressed face was presented for longer durations than the average duration, we further checked the congruency effect in the main experiment only using data from these 16 participants and only using trials with duration that are equal to or shorter than the trials in the awareness check. With this subset of data meeting the above criteria, the average response time in the congruent condition was still significantly shorter than that in the incongruent condition (381 ms vs. 396 ms, t(15) = 2.70, p<0.05); and the accuracy in the congruent condition also remained higher than that in the incongruent condition (67.46% vs. 60.02%, t(15) = 2.58, p<0.05).
General Discussion
The present study was designed to investigate whether facial expression from an invisibly presented flanker could influence the recognition of expression of a visible target face. The flanker stimuli were rendered invisible using continuous flash suppression [23], [24]. Results showed that the exposure to subliminal facial expressions presented at different spatial location indeed influenced the expression recognition of the visible face, as seen in the congruency effect: congruent facial expressions facilitated participants' responses both in terms of the faster reaction time and improved accuracy. This observed congruency effect is consistent with the many published reports of unconscious processing of emotional information [15], [17], [29], [32]. Since the happy and fearful faces have systematic image differences (e.g., sizes of eyes and mouths, etc), we acknowledge the possibility that the observed congruency effect could be attributed to the congruency between the different images.
Early studies have shown that invisible affective information could be processed in the brain [35], [36]. Invisible affective stimuli can generate a subliminal affective priming effect that influences subsequent emotional processing [17]. Results from the present study using CFS to render faces invisible are consistent with and extends finding using back-masked faces [32] in showing that subliminal affective facial expression could induce emotional conflict on a visibly presented spatially non-overlapping face.
Several imaging studies investigating the emotional information processing under CFS have shown that the subliminal emotional stimuli during binocular rivalry suppression also could be processed especially in the amygdala [8]–[10]. One possibility is that the suppressed emotional information could engage amygdala via the subcortical pathway through the superior colliculus and the pulvinar [9], [37], [38]. However, there is still a debate on the role of amygdala in the detection of fear signal [8], [26]–[28], [37], in particular with one study showing that the advantage of fear detection still exists even when bilateral amygdala were damaged [28].
The congruency effect observed here in expression judgment reflects a conflict resolution process. Congruency effects between facial expressions (fearful/happy) with emotional words (e.g., “happy” or “fear”) written on faces have been reported before [39], but what is different here is that the conflicting expression information is not between two consciously perceived inputs, but between a visible target and an invisible flanker. The interaction between conscious and unconscious emotional information could potentially occur in the amygdala, or at high-level cortical sites. Further neuroimaging research will be needed to investigate the specific neural mechanisms responsible for the integration of conscious and unconscious input information.
Conclusions
The present study demonstrated that unconsciously processed facial expression information can influence the recognition of consciously processed facial expression information, across different spatial locations. Our results add to the growing evidence that invisible information can be processed rapidly and can directly influence conscious behavior.
Author Contributions
Conceived and designed the experiments: XY SH YQY KW. Performed the experiments: XY SH YH. Analyzed the data: XY SH YQY KW. Wrote the paper: XY SH YQY KW.
References
- 1. Naccache L, Gaillard R, Adam C, Hasboun D, Clemenceau S, et al. (2005) A direct intracranial record of emotions evoked by subliminal words. Proc Natl Acad Sci U S A 102: 7713–7717.
- 2. Dehaene S, Naccache L, Le Clec HG, Koechlin E, Mueller M, et al. (1998) Imaging unconscious semantic priming. Nature 395: 597–600.
- 3. Wong PS, Bernat E, Bunce S, Shevrin H (1997) Brain indices of nonconscious associative learning. Conscious Cogn 6: 519–544.
- 4. Tamietto M, de Gelder B (2010) Neural bases of the non-conscious perception of emotional signals. Nat Rev Neurosci 11: 697–709.
- 5. Whalen PJ, Rauch SL, Etcoff NL, McInerney SC, Lee MB, et al. (1998) Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. J Neurosci 18: 411–418.
- 6. Morris JS, Ohman A, Dolan RJ (1998) Conscious and unconscious emotional learning in the human amygdala. Nature 393: 467–470.
- 7. Yoon KL, Hong SW, Joormann J, Kang P (2009) Perception of facial expressions of emotion during binocular rivalry. Emotion 9: 172–182.
- 8. Williams MA, Morris AP, McGlone F, Abbott DF, Mattingley JB (2004) Amygdala responses to fearful and happy facial expressions under conditions of binocular suppression. J Neurosci 24: 2898–2904.
- 9. Pasley BN, Mayes LC, Schultz RT (2004) Subcortical discrimination of unperceived objects during binocular rivalry. Neuron 42: 163–172.
- 10. Jiang Y, He S (2006) Cortical responses to invisible faces: dissociating subsystems for facial-information processing. Curr Biol 16: 2023–2029.
- 11. Jiang Y, Shannon RW, Vizueta N, Bernat EM, Patrick CJ, et al. (2009) Dynamics of processing invisible faces in the brain: automatic neural encoding of facial expression information. Neuroimage 44: 1171–1177.
- 12. Vuilleumier P, Armony JL, Driver J, Dolan RJ (2001) Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron 30: 829–841.
- 13. Carretié L, Hinojosa JA, Albert J, López-Martín S, De La Gándara BS, et al. (2008) Modulation of ongoing cognitive processes by emotionally intense words. Psychophysiology 45: 188–196.
- 14. Dannlowski U, Suslow T (2006) Test-retest reliability of subliminal facial affective priming. Psychol Rep 98: 153–158.
- 15. Winkielman P, Berridge KC, Wilbarger JL (2005) Unconscious affective reactions to masked happy versus angry faces influence consumption behavior and judgments of value. Pers Soc Psychol Bull 31: 121–135.
- 16. Rotteveel M, de Groot P, Geutskens A, Phaf RH (2001) Stronger suboptimal than optimal affective priming? Emotion 1: 348–364.
- 17. Murphy ST, Zajonc RB (1993) Affect, cognition, and awareness: affective priming with optimal and suboptimal stimulus exposures. J Pers Soc Psychol 64: 723–739.
- 18. Otten S, Wentura D (1999) About the impact of automaticity in the minimal group paradigm: evidence from affective priming tasks. European Journal of Social Psychology 29: 1049–1071.
- 19. Draine SC, Greenwald AG (1998) Replicable unconscious semantic priming. J Exp Psychol Gen 127: 286–303.
- 20. Winkielman P, Zajonc, Norbert Schwarz RB (1997) Subliminal affective priming resists attributional interventions. Cognition & Emotion 11: 433–465.
- 21. Hermans D, Spruyt A, De Houwer J, Eelen P (2003) Affective priming with subliminally presented pictures. Can J Exp Psychol 57: 97–114.
- 22. Banse R (2001) Affective priming with liked and disliked persons: Prime visibility determines congruency and incongruency effects. Cognition & Emotion 15: 501–520.
- 23. Tsuchiya N, Koch C (2005) Continuous flash suppression reduces negative afterimages. Nat Neurosci 8: 1096–1101.
- 24. Fang F, He S (2005) Cortical responses to invisible objects in the human dorsal and ventral pathways. Nat Neurosci 8: 1380–1385.
- 25. Jiang Y, Costello P, He S (2007) Processing of invisible stimuli: advantage of upright faces and recognizable words in overcoming interocular suppression. Psychol Sci 18: 349–355.
- 26. Yang E, Zald DH, Blake R (2007) Fearful expressions gain preferential access to awareness during continuous flash suppression. Emotion 7: 882–886.
- 27. Stein T, Seymour K, Hebart MN, Sterzer P (2014) Rapid fear detection relies on high spatial frequencies. Psychol Sci 25: 566–574.
- 28. Tsuchiya N, Moradi F, Felsen C, Yamazaki M, Adolphs R (2009) Intact rapid detection of fearful faces in the absence of the amygdala. Nat Neurosci 12: 1224–1225.
- 29. Almeida J, Pajtas PE, Mahon BZ, Nakayama K, Caramazza A (2013) Affect of the unconscious: Visually suppressed angry faces modulate our decisions. Cognitive, Affective, & Behavioral Neuroscience 13: 94–101.
- 30. Adams WJ, Gray KL, Garner M, Graf EW (2010) High-level face adaptation without awareness. Psychological Science 21: 205–210.
- 31. Gray KL, Adams WJ, Hedger N, Newton KE, Garner M (2013) Faces and awareness: low-level, not emotional factors determine perceptual dominance. Emotion 13: 537–544.
- 32. Jiang J, Bailey K, Chen A, Cui Q, Zhang Q (2013) Unconsciously triggered emotional conflict by emotional facial expressions. PLoS One 8: e55907.
- 33. Almeida J, Mahon BZ, Nakayama K, Caramazza A (2008) Unconscious processing dissociates along categorical lines. Proc Natl Acad Sci U S A 105: 15214–15218.
- 34.
Sterzer P, Haynes JD, Rees G (2008) Fine-scale activity patterns in high-level visual areas encode the category of invisible objects. J Vis 8 : 10 11–12.
- 35. Dijksterhuis A, Aarts H (2003) On wildebeests and humans: the preferential detection of negative stimuli. Psychol Sci 14: 14–18.
- 36. Ohman A, Soares JJ (1994) “Unconscious anxiety”: phobic responses to masked stimuli. J Abnorm Psychol 103: 231–240.
- 37. Morris JS, Ohman A, Dolan RJ (1999) A subcortical pathway to the right amygdala mediating “unseen” fear. Proc Natl Acad Sci U S A 96: 1680–1685.
- 38. Troiani V, Hunyadi E, Riley M, Herrington J, Schultz R (2010) Cortical and subcortical correlates of nonconscious face processing. Journal of Vision 10: 608–608.
- 39. Etkin A, Egner T, Peraza DM, Kandel ER, Hirsch J (2006) Resolving emotional conflict: a role for the rostral anterior cingulate cortex in modulating activity in the amygdala. Neuron 51: 871–882.