Neural correlates of bimodal speech and gesture comprehension

Abstract

The present study examined the neural correlates of speech and hand gesture comprehension in a naturalistic context. Fifteen participants watched audiovisual segments of speech and gesture while event-related potentials (ERPs) were recorded to the speech. Gesture influenced the ERPs to the speech. Specifically, there was a right-lateralized N400 effect—reflecting semantic integration—when gestures mismatched versus matched the speech. In addition, early sensory components in bilateral occipital and frontal sites differentiated speech accompanied by matching versus non-matching gestures. These results suggest that hand gestures may be integrated with speech at early and late stages of language processing.

Introduction

People of all ages, cultures, and backgrounds gesture when they speak. What function do these hand movements serve? Although there is consensus that gesture plays an important role during language production (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001; Hadar, Wenkert-Olenik, Krauss, & Soroker, 1998; Iverson & Goldin-Meadow, 1998; Kita & Özyürek, 2003; Krauss, 1998; McNeill, 1992; Morrel-Samuels & Krauss, 1992; Rauscher, Krauss, & Chen, 1996), there is considerable debate on whether gesture also serves a communicative function during language comprehension (Clark, 1996; Kelly, Barr, Church, & Lynch, 1999; Krauss, 1998; Krauss, Morrel-Samuels, & Colasante, 1991; McNeill, 1992).

On one side, researchers who take a “gesture as communication” stance contend that gesture and speech are tightly integrated, with gesture influencing speech processing even at the earliest stages of comprehension (Cassell, McNeill, & McCullough, 1999; Kelly et al., 1999; McNeill, 1992). The standard paradigm used to support this claim has been to present people with verbal and gestural communication and to demonstrate that gesture is closely and unconsciously integrated with speech at comprehension.

On the other side, researchers who take a “gesture as non-communication” stance argue that gesture and speech are independent systems and that gesture does not influence language comprehension in a significant way (Krauss, 1998; Krauss et al., 1991). These researchers argue that the “gesture as communication” studies have demonstrated, at best, a trivial relationship between speech and gesture at comprehension. For example, gesture may be used as “add-on” information to comprehend a communicator’s meaning only after the speech has been processed. In other words, in rare cases when gesture does influence comprehension, the attention to gesture is post hoc and does not impact early stages of speech processing.

At the core of this debate is the fact that all of the previous studies have relied on indirect behavioral measures that do not provide access to the underlying neurocognitive processing of speech and gesture. The present experiment addresses this issue by using a more direct measure of the neurocognitive processing of speech and gesture: event-related potentials (ERPs). Because ERPs provide excellent temporal information—they indicate when neurocognitive processes occur—the ERP technique is perfectly suited for investigating the relationship of gesture and speech in language comprehension.

The experiment addresses two specific questions. The first question asks whether gesture influences ERPs to speech? A strong version of the “gesture as non-communication” view predicts that because gesture is completely independent from language, gesture should not have any impact on the neural processing of the accompanying speech. In contrast, the “gesture as communication” view predicts that because gesture and speech are fundamentally integrated, the ERP data will support past behavioral research demonstrating that gestures do influence speech processing.

The second question asks if gesture does impact speech processing, what is the time-course of this influence? A weak form of the “gesture as non-communication” view predicts that a gesture influence could occur only after the brain had already processed the semantic content of the speech. For example, this view would predict a late ERP effect resembling an N400 (Kutas & Hillyard, 1984). Previous researchers have argued that this type of effect reflects post-semantic processing of speech (Holcomb, 1993; Osterhout & Holcomb, 1995). In contrast, the “gesture as communication” view predicts that gesture would also affect pre-semantic processing of the speech. For example, this view predicts that gesture would influence early portions of the brainwave—such as, sensory, P1–N1, and P2 components—that reflect low-level, sensory/phonological processing of speech (Rugg & Coles, 1995).