Abstract
In this study, we examined whether integration of visual and auditory information about emotions requires limited attentional resources. Subjects judged whether a voice expressed happiness or fear, while trying to ignore a concurrently presented static facial expression. As an additional task, the subjects had to add two numbers together rapidly (Experiment 1), count the occurrences of a target digit in a rapid serial visual presentation (Experiment 2), or judge the pitch of a tone as high or low (Experiment 3). The visible face had an impact on judgments of the emotion of the heard voice in all the experiments. This cross-modal effect was independent of whether or not the subjects performed a demanding additional task. This suggests that integration of visual and auditory information about emotions may be a mandatory process, unconstrained by attentional resources.
Article PDF
Similar content being viewed by others
References
Bertelson, P., Pavani, F., Ladavas, E., Vroomen, J., & de Gelder, B. (2000). Ventriloquism in patients with unilateral visual neglect. Neuropsychologia, 38, 1634–1642.
Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory—visual spatial discordance. Perception & Psychophysics, 29, 578–584.
Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321–332.
Charpentier, F., & Moulines, E. (1989). Pitch-synchronous waveform processing techniques for text-to-speech synthesis. Proceedings of Eurospeech, 89, 13–19.
de Gelder, B., Pourtois, G., van Raamsdonk, M., Vroomen, J., & Weiskrantz, L. (2001). Unseen stimuli modulate conscious visual experience: Evidence from inter-hemispheric summation. NeuroReport, 12, 385–391.
de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14, 289–311.
de Gelder, B., Vroomen, J., Pourtois, G., & Weiskrantz, L. (1999). Non-conscious recognition of affect in the absence of striate cortex. NeuroReport, 10, 3759–3763.
de Gelder, B., Vroomen, J., & Teunisse, J.-P. (1995, November). Hearing smiles and seeing cries: The bimodal perception of emotion. Paper presented at the 36th Annual Meeting of the Psychonomic Society, Los Angeles.
Dolan, R., Morris, J., & de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences, 98, 10006–10010.
Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lipreading. Nature, 381, 66–68.
Driver, J., & Vuilleumier, P. (2001). Unconscious processing in neglect and extinction. In B. de Gelder, E. de Haan, & C. A. Heywood (Eds.), Out of mind: Varieties of unconscious processes (pp. 107–139). Oxford: Oxford University Press.
Joseph, J. S., Chun, M. M., & Nakayama, K. (1996). Attentional requirements in a “preattentive” feature search task. Nature, 379, 805–807.
Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice-Hall.
Lavie, N. (1995). Perceptual load as a necessary condition for selective attention. Journal of Experimental Psychology: Human Perception & Performance, 21, 451–468.
Lavie, N. (2000). Selective attention and cognitive control. In S. Monsell & J. Driver (Eds.), Attention & performance XVIII: Control of cognitive processes (pp. 175–194). Cambridge, MA: MIT Press.
Ledoux, J. E. (1996). The emotional brain: The mysterious underpinnings of emotional life. New York: Simon & Schuster.
Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
Massaro, D. W., & Egan, P. B. (1996). Perceiving affect from the voice and the face. Psychonomic Bulletin & Review, 3, 215–221.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.
Morris, J. S., de Gelder, B., Weiskrantz, L., & Dolan, R. J. (2001). Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind field. Brain, 124, 1241–1251.
Morris, J. S., Oehman, A., & Dolan, R. J. (1999). A subcortical pathway to the right amygdala mediating “unseen” fear. Proceedings of the National Academy of Sciences, 96, 1680–1685.
Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelink, M. (2000). The time course of intermodal binding between seeing and hearing affective information. NeuroReport, 11, 1329–1333.
Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: 2. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127–190.
Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception & Psychophysics, 63, 651–659.
Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception & Performance, 26, 1583–1590.
Whalen, P. J., Rauch, S. L., Etcoff, N. L., McInerney, S. C., Lee, M.B., & Jenike, M.A. (1998). Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge. Journal of Neuroscience, 18, 411–418.
Wickens, D. D. (1984). Processing resources in attention. In R. Parasuraman & D. R. Davies (Eds.), Varieties of attention (pp. 63–102). Orlando, FL: Academic Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Vroomen, J., Driver, J. & Gelder, B.D. Is cross-modal integration of emotional expressions independent of attentional resources?. Cognitive, Affective, & Behavioral Neuroscience 1, 382–387 (2001). https://doi.org/10.3758/CABN.1.4.382
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/CABN.1.4.382