Short CommunicationNeural correlates of bimodal speech and gesture comprehension☆
Introduction
People of all ages, cultures, and backgrounds gesture when they speak. What function do these hand movements serve? Although there is consensus that gesture plays an important role during language production (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001; Hadar, Wenkert-Olenik, Krauss, & Soroker, 1998; Iverson & Goldin-Meadow, 1998; Kita & Özyürek, 2003; Krauss, 1998; McNeill, 1992; Morrel-Samuels & Krauss, 1992; Rauscher, Krauss, & Chen, 1996), there is considerable debate on whether gesture also serves a communicative function during language comprehension (Clark, 1996; Kelly, Barr, Church, & Lynch, 1999; Krauss, 1998; Krauss, Morrel-Samuels, & Colasante, 1991; McNeill, 1992).
On one side, researchers who take a “gesture as communication” stance contend that gesture and speech are tightly integrated, with gesture influencing speech processing even at the earliest stages of comprehension (Cassell, McNeill, & McCullough, 1999; Kelly et al., 1999; McNeill, 1992). The standard paradigm used to support this claim has been to present people with verbal and gestural communication and to demonstrate that gesture is closely and unconsciously integrated with speech at comprehension.
On the other side, researchers who take a “gesture as non-communication” stance argue that gesture and speech are independent systems and that gesture does not influence language comprehension in a significant way (Krauss, 1998; Krauss et al., 1991). These researchers argue that the “gesture as communication” studies have demonstrated, at best, a trivial relationship between speech and gesture at comprehension. For example, gesture may be used as “add-on” information to comprehend a communicator’s meaning only after the speech has been processed. In other words, in rare cases when gesture does influence comprehension, the attention to gesture is post hoc and does not impact early stages of speech processing.
At the core of this debate is the fact that all of the previous studies have relied on indirect behavioral measures that do not provide access to the underlying neurocognitive processing of speech and gesture. The present experiment addresses this issue by using a more direct measure of the neurocognitive processing of speech and gesture: event-related potentials (ERPs). Because ERPs provide excellent temporal information—they indicate when neurocognitive processes occur—the ERP technique is perfectly suited for investigating the relationship of gesture and speech in language comprehension.
The experiment addresses two specific questions. The first question asks whether gesture influences ERPs to speech? A strong version of the “gesture as non-communication” view predicts that because gesture is completely independent from language, gesture should not have any impact on the neural processing of the accompanying speech. In contrast, the “gesture as communication” view predicts that because gesture and speech are fundamentally integrated, the ERP data will support past behavioral research demonstrating that gestures do influence speech processing.
The second question asks if gesture does impact speech processing, what is the time-course of this influence? A weak form of the “gesture as non-communication” view predicts that a gesture influence could occur only after the brain had already processed the semantic content of the speech. For example, this view would predict a late ERP effect resembling an N400 (Kutas & Hillyard, 1984). Previous researchers have argued that this type of effect reflects post-semantic processing of speech (Holcomb, 1993; Osterhout & Holcomb, 1995). In contrast, the “gesture as communication” view predicts that gesture would also affect pre-semantic processing of the speech. For example, this view predicts that gesture would influence early portions of the brainwave—such as, sensory, P1–N1, and P2 components—that reflect low-level, sensory/phonological processing of speech (Rugg & Coles, 1995).
Section snippets
Participants
Fifteen right-handed, Caucasian college undergraduates (6 males, 9 females; mean age: 20) participated for course credit.
Materials
Participants watched digitized videos (created with a Sony DV100 digital camcorder and edited with iMovie Macintosh software) of a person producing audiovisual speech and gesture. The audiovisual segments contained a male actor (face and hands toward the camera) sitting at a table behind a tall, thin glass and a short, wide dish. Note that the actor’s mouth, face, and hands
Results and discussion
The PCA identified five factors that accounted for 87% of the total variance. Factor 1 accounted for 34% of the variance and ranged from 324 to 648 ms; factor 2 (25% of variance) ranged from 568 to 900 ms; factor 3 (13% of variance) ranged from 148 to 352 ms; factor 4 (8% of variance) ranged from 72 to 168 ms.; and factor 5 (7% of variance) ranged from 0 to 92 ms.
The weighted and Varimax-rotated PCA factors were submitted to a three-way repeated measures ANOVA with Gesture (no gesture, matching,
General discussion
The results have provided answers to the two questions posed in this experiment, in addition to raising some interesting methodological considerations.
Conclusion
Language naturally occurs in a rich communicative and audiovisual context. The results from the present study demonstrate that one aspect of this context—hand gesture—significantly impacts the comprehension of accompanying speech at multiple stages of language comprehension. This finding not only pertains to theories of gesture–speech integration, but also to any account of how the brain comprehends language in natural discourse.
References (29)
The electrophysiology of incidental and intentional retrieval: ERP old/new effects in lexical decision and recognition memory
Neuropsychologia
(1999)- et al.
Crossmodal links in endogenous and exogenous spatial attention: Evidence from event-related brain potential studies
Neuroscience and Biobehavioral Reviews
(2001) - et al.
Gesture and the processing of speech: Neuropsychological evidence
Brain & Language
(1998) - et al.
Human neural systems for face recognition and social communication
Biological Psychiatry
(2002) - et al.
Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory
Journal of Memory and Language
(1999) - et al.
Right hemisphere activation during indirect semantic priming: Evidence from event-related potentials
Brain and Language
(1998) - et al.
What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking
Journal of Memory and Language
(2003) - et al.
The lateral distribution of event-related potentials during sentence processing
Neuropsychologia
(1982) - et al.
Principal component analysis of event-related potentials: Simulation studies demonstrate misallocation of variance across components
Electroencephalography Clinical Neurophysiology
(1984) - et al.
Misallocation of variance in event-related potentials: Simulation studies of the effects of test power, topography, and baseline-to-peak versus principal component quantifications
Journal of Neuroscience Methods
(2003)
Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information
Pragmatics and Cognition
Using language
Auditory evoked potential correlates of speech sound discrimination
Perception and Psychophysics
Explaining math: Gesture lightens the load
Psychological Science
Cited by (0)
- ☆
The authors thank Dennis Molfese, Susan Goldin-Meadow, and Howard Nusbaum for making comments on earlier versions of this paper. We extend special gratitude to Ron Crans for programming the stimulus presentation software.