Brief articleAudio–visual speech perception is special
Section snippets
Subjects
Ten students of the Helsinki University of Technology were studied. All reported normal hearing and normal or corrected-to-normal vision. None of the subjects had earlier experience with SWS stimuli. Two subjects were excluded from the subject pool because they reported perceiving the SWS stimuli as speech before being instructed about their speech-like nature.
Stimuli
Four auditory stimuli (natural /omso/ and /onso/ and their sine wave replicas) and digitized video clips of a male face articulating
Experiment 2
In Experiment 1, the different tasks were always performed in the same order, so that the non-speech mode always preceded speech mode for the SWS stimuli. The reason for this was that once the subject “enters speech mode” it is impossible to hear the SWS stimuli as non-speech. However, this procedure might have created a learning effect so that subjects might have become more used to SWS stimuli. Then at least part of the large integration effect observed with the incongruent stimuli could have
Discussion
Our results demonstrate that acoustic and visual speech were integrated strongly only when the perceiver interpreted the acoustic stimuli as speech. If the SWS stimuli had always been processed in the same way, the influence of visual speech should have been the same in both speech and non-speech modes. This result does not depend on the amount of practise with listening to SWS stimuli as confirmed by the results obtained in Experiment 2.
We suggest that when SWS stimuli were perceived as
Acknowledgements
The research of T.S.A. was supported by the European Union Research Training Network “Multi-modal Human–Computer Interaction”. Financial support from the Academy of Finland to the Research Centre for Computational Science and Engineering and to MS is also acknowledged. We thank Ms Reetta Korhonen for help in data collection and Riitta Hari (Low Temperature Lab, HUT) for valuable comments on the manuscript.
References (27)
- et al.
‘Putting the face to the voice’: matching identity across modality
Current Biology
(2003) - et al.
The motor theory of speech perception revised
Cognition
(1985) Establishing and maintaining perceptual coherence: unimodal and multimodal evidence
Journal of Phonetics
(2003)- et al.
Multimodal perceptual organization of speech: evidence from tone analogs of spoken utterances
Speech Communication
(1998) - et al.
A feature-integration theory of attention
Cognitive Psychology
(1980) - Boersma, P., & Weenink, D., (1992–2002). Praat a system doing phonetics by computer, v. 4.0.13....
Cross-modal processing in the human brain: insights from functional neuroimaging studies
Cerebral Cortex
(2001)Cooperating brain systems in selective perception and action
Listeners do hear sounds, not tongues
Journal of the Acoustical Society of America
(1996)Echoes of echoes? An episodic theory of lexical access
Psychological Review
(1998)