Abstract
Simulation models of expression recognition contend that to understand another's facial expressions, individuals map the perceived expression onto the same sensorimotor representations that are active during the experience of the perceived emotion. To investigate this view, the present study examines facial expression and identity recognition abilities in a rare group of participants who show facilitated sensorimotor simulation (mirror-touch synesthetes). Mirror-touch synesthetes experience touch on their own body when observing touch to another person. These experiences have been linked to heightened sensorimotor simulation in the shared-touch network (brain regions active during the passive observation and experience of touch). Mirror-touch synesthetes outperformed nonsynesthetic participants on measures of facial expression recognition, but not on control measures of face memory or facial identity perception. These findings imply a role for sensorimotor simulation processes in the recognition of facial affect, but not facial identity.
Introduction
Simulation accounts of expression recognition contend that to understand the emotion associated with another's facial expression, the observer simulates the sensorimotor response associated with generating the perceived expression (Adolphs, 2002; Gallese et al., 2004; Goldman and Sripada, 2005; Keysers and Gazzola, 2006; Bastiaansen et al., 2009; Keysers and Gazzola, 2009). This view is supported by evidence that responses in expression-relevant facial muscles are increased during subliminal exposure to emotional expressions (Dimberg et al., 2000), that preventing the activation of expression-relevant muscles impairs expression recognition (Oberman et al., 2007), and that perceiving another's expressions and producing one's own recruits similar premotor and somatosensory regions (Hennenlotter et al., 2005; van der Gaag et al., 2007). Further, neuropsychological findings indicate that damage to right somatosensory cortices is associated with expression-recognition deficits (Adolphs et al., 2000), and transcranial magnetic stimulation findings have demonstrated the involvement of the right somatosensory cortex for facial expression recognition, but not face identity recognition, in healthy adults (Pitcher et al., 2008). These findings imply that purely visual face-processing mechanisms interact with sensorimotor representations to facilitate expression recognition. This may differ from facial identity recognition, as there is no clear indication of how one could simulate another's identity (Calder and Young, 2005).
A complimentary approach is to consider whether facilitation of sensorimotor mechanisms promotes expression recognition. One example of facilitated sensorimotor simulation is the case of mirror-touch synesthesia, in which simply observing touch to others elicits tactile sensations on the synesthete's own body (Blakemore et al., 2005; Banissy et al., 2009b). Functional brain imaging indicates that this variant of synesthesia is linked to heightened neural activity in a network of brain regions also activated in nonsynesthetic subjects when observing touch to others. This mirror-touch system is comprised of brain areas active during both the observation and passive experience of touch (including primary and secondary somatosensory cortices and premotor cortex) (Keysers et al., 2004; Blakemore et al., 2005; Ebisch et al., 2008). It has been suggested that brain systems involved in mirroring the experiences of others may be crucial for social perception because they provide a plausible neural mechanism to facilitate sensorimotor simulation of another's perceived state (Gallese et al., 2004; Keysers and Gazzola, 2006). In this sense, mirror-touch synesthesia can be viewed as a case of heightened sensorimotor simulation, which may be able to inform us about the role of sensorimotor simulation mechanisms in social cognition. For example, Banissy and Ward (2007) reported that mirror-touch synesthetes, but not other synesthetes, show heightened emotional empathy compared with control participants.
Here we examined whether mirror-touch synesthetes differed in another aspect of social perception; namely, facial expression recognition. We compared mirror-touch synesthetes and nonsynesthetic controls on facial expression recognition, identity recognition, and identity perception tasks. Based on the hypothesis that mirror-touch synesthetes have heightened sensorimotor simulation mechanisms, we predicted that synesthetes would show superior performance on expression recognition tasks but not on the facial identity control tasks that are less dependent on simulation.
Materials and Method
Participants
Eight mirror-touch synesthetes (six female and two male; mean age ± SD = 45.6 ± 11.7 years) and 20 nonsynesthetic control participants (15 female and five male; mean age ± SD = 35.6 ± 13.6 years) took part in the study. All cases of mirror-touch synesthesia were confirmed using a previously developed visual–tactile congruity paradigm designed to provide evidence for the authenticity of the condition (Banissy and Ward, 2007) (supplemental Table 1, available at www.jneurosci.org as supplemental material).
Materials and procedure
Participants completed four tasks in a counterbalanced order. These tasks are detailed below.
Films facial expression recognition.
This task investigated participants' ability to recognize the emotional expressions of others. In each trial, participants were presented with an adjective describing an emotional state followed by three images (each image shown for 500 ms) of the same actor or actress displaying different facial expressions. Participants were asked to indicate which of the three images best portrayed the target emotional adjective.
To portray subtle and realistic facial expressions, expression stimuli were captured from films (Fig. 1a). Fifty-eight target images (preceded by three practice trials) from 15 films were used. All films were from a non-English speaking country to decrease the probability that participants had seen them or were familiar with the actors. Target and distractor stimuli were selected based on four pilot studies (Garrido et al., 2009). Each stimulus was shown once during the test, and trials were presented in a fixed order over two blocks (29 trials per block).
Cambridge face memory test.
To test face recognition, we compared performance of synesthetes and nonsynesthetes on the Cambridge face memory test long form (CFMT+) (Russell et al., 2009). The task is an adapted version of the Cambridge face memory test (CFMT) (Duchaine and Nakayama, 2006) (Fig. 1b) and was designed to distinguish normal from supernormal ability to recognize faces (Russell et al., 2009). During the task, participants were asked to learn to recognize six unfamiliar male faces from three different views and were then tested on their ability to recognize these faces in a three-alternative forced-choice task.
The test was comprised of four sections, each more difficult than the previous. The first three sections were taken from the original CFMT (Duchaine and Nakayama, 2006) and the addition of the final section forms the longer CFMT+ (Russell et al., 2009). The test began by testing recognition with the same images that were used during training. This relatively easy introduction was followed by a section using novel images that show the target faces from previously unseen perspectives and different lighting conditions, and a third section consisting of novel images with visual noise added. The final section contained 30 very difficult trials in which distractor images repeated much more frequently, targets and distractors contained more visual noise than the images in the third section, cropped (i.e., only showing internal features) and uncropped images (i.e., showing hair, ears, and necks, which had not been shown in the previous sections) were used, and images showing the targets and distractors making emotional expressions were included. The percentage of correct responses for each section and overall were measured.
Cambridge face perception test.
To investigate facial identity perception, we administered the Cambridge face perception test (Duchaine et al., 2007). This test assesses the ability to perceive differences between facial identities. Memory demands are minimal because faces are presented simultaneously. During the task, participants were shown a target face (from a 3/4 viewpoint) and six faces (from a frontal view) morphed between the target and another face in varying proportions so that they varied systematically in their similarity to the target face (Fig. 1c). Participants were asked to sort the six faces by similarity to the target face and were given 1 min to do so. The task involved eight upright and eight inverted trials that alternated in a fixed pseudorandom order. Performance was measured by an error score. This was calculated by summing the deviations from the correct position for each face, with one error reflecting each position that a face must be moved to be in the correct location. For example, if a face was one position from the correct location the error score was one. If it was three positions away from the correct location, this was an error score of three.
Same–different expression and identity matching task.
This task investigated participants' abilities to match another's facial identity or facial expressions (Pitcher et al., 2008).
In the expression matching task, participants were presented with a sample face (250 ms), followed by a fixation cross (1000 ms), and then a target face (250 ms). Participants were asked to indicate whether the target facial expression matched or was different from the sample facial expression. On half of the trials, the target and sample face expressed the same emotion and half the sample–target pairs showed different emotions (Fig. 1d). A total of 72 trials (split between two blocks) were completed. Each image showed one of six female models making one of six basic facial expressions (anger, disgust, fear, happiness, sadness, or surprise). Each stimulus was a grayscale image taken from the Ekman and Friesen's (1976) facial affect series. The hair and neck of stimuli were removed using Adobe Photoshop. In the expression task, identity always changed between sample–target pairs and each expression was presented an equal number of times.
In the identity matching task, the same stimuli and procedure were used. Participants were asked to indicate whether the sample and target face were the same or a different person. Half of the trials showed pairs with the same identity and half with a different identity. Expression always changed between the sample and target face, and the six models were presented an equal number of times.
Results
Participant age was used a covariate on all analyses because of a slight trend for synesthetes to differ from controls on age (t(26) = 1.84, p = 0.078).
Films facial expression recognition
Accuracy and reaction time performance were compared separately using a one-way between-subjects ANCOVA. One control participant was withdrawn from analysis due to difficulties in understanding the meaning of expression adjectives and performing more than 3 SDs below the control group mean on accuracy and reaction time.
Synesthetes showed superior abilities at recognizing the emotional expressions of others (Fig. 2). Analysis of accuracy performance revealed that mirror-touch synesthetes outperformed control participants on expression recognition (F(1,24) = 16.38, p < 0.001) (Fig. 2a). This difference was not due to a speed–accuracy trade-off, as no significant effect of group (synesthete or control) was found for reaction time performance (F(1,24) =.0.962, p = 0.336) (supplemental Fig. 1, available at www.jneurosci.org as supplemental material). These findings suggest that mirror-touch synesthetes show superior facial expression recognition, which may be due to heightened sensorimotor simulation mechanisms.
Cambridge face memory test
No significant differences were observed between synesthetes and controls on either the CFMT (F(1,25) = 0.023, p = 0.880) (supplemental Fig. 2, available at www.jneurosci.org as supplemental material) or the CFMT+ (F(1,25) = 0.095, p = 0.761) (Fig. 2b). Therefore, unlike facial expression recognition, synesthetes and controls did not differ in their ability to memorize facial identity.
Cambridge face perception test
Error scores on the eight upright and eight inverted trials were summed to determine the total number of upright and inverted errors. A 2 (group) × 2 (trial type) ANCOVA revealed a significant effect of trial type (F(1,25) = 5.81, p < 0.05), which was due to an inversion effect, whereby overall participants were less accurate on inverted (mean ± SEM = 70 ± 3) compared with upright trials (mean ± SEM = 41.5 ± 3.21). Importantly, this effect did not interact with group (F(1,25) = 0.37, p = 0.549) and no main effect of group was found (F(1,25) = 0.253, p = 0.619) (Fig. 2c,d). Therefore, unlike expression recognition, synesthetes and controls did not significantly differ in their abilities to match facial identities.
Same–different expression and identity matching task
A 2 (group) × 2 (task) mixed ANCOVA was conducted. No main effect of task or group was found. There was, however, a significant interaction between task and group (F(1,25) = 4.507, p < 0.05). Controls were more accurate on the identity matching task relative to the emotion matching task (F(1,18) = 5.10, p < 0.05). Synesthetes did not show this pattern; analysis of within-subject effects revealed no significant difference between the two tasks for the synesthetic group (F(1,6) = 0.759, p = 0.417). There was also a nonsignificant trend for synesthetes to outperform controls on the expression matching task (Fig. 2e), but for controls to outperform synesthetes on the identity matching task (Fig. 2f).
Discussion
This study investigated expression and identity face processing in mirror-touch synesthetes and nonsynesthete control participants. We predicted that heightened sensorimotor simulation mechanisms would result in superior expression recognition, but would not affect the identity recognition abilities of mirror-touch synesthetes. Consistent with these predictions, mirror-touch synesthetes were superior when recognizing facial expressions but not facial identities. Moreover, we found that synesthetes significantly outperformed control participants in their ability to accurately categorize facial expressions, but did not differ from nonsynesthetes in their ability to memorize or perceive facial identity. We also found that controls performed better at matching another's identity than their expressions, but the reverse trend was shown by synesthetes, a significant group × condition interaction. These findings are therefore consistent with simulation accounts of expression recognition, which suggest that individuals understand others' emotional expressions by simulating the sensorimotor response associated with generating the perceived facial expression (Adolphs, 2002; Gallese et al., 2004; Goldman and Sripada, 2005; Keysers and Gazzola, 2006, 2009).
A variety of sources indicate that recognizing another's identity and expressions relies upon multiple stages of representation, including purely visual, multimodal, expression-general, and expression-specific mechanisms (Adolphs et al., 2000; Anderson et al., 2000; Calder et al., 2001, 2004; Lawrence et al., 2002; Keane et al., 2002; Pitcher et al., 2008). Simulation accounts of expression recognition contend that one mechanism involved in expression, but not identity recognition, is an internal sensorimotor reenactment of the perceived expression (Adolphs, 2002; Gallese et al., 2004; Goldman and Sripada, 2005; Keysers and Gazzola, 2006). Functional brain imaging (Hennenlotter et al., 2005; van der Gaag et al., 2007) and neuropsychological (Adolphs et al., 2000) and transcranial magnetic stimulation studies (Pitcher et al., 2008) suggest a key role for somatosensory resources in expression recognition. Our findings that individuals who show increased levels of somatosensory simulation (mirror-touch synesthetes) demonstrate superior facial expression, but not facial identity perception, are consistent with this view. More specifically, they suggest that the level of vicarious activations in the somatosensory system contributes to the recognition of others' facial expressions.
One may note, however, that based solely on the interaction observed in our identity–expression matching task, it is possible that our findings may be more in line with reduced levels of identity perception rather than superior expression perception in mirror-touch synesthesia. There are a number of reasons why this is not the case. The identity–expression matching task demonstrates a significant group × task interaction, suggesting a different profile of strengths across groups, but does not allow us to establish definitively its locus. Therefore, it would be premature to conclude that synesthetes have identity recognition problems from this task alone and we must consider the wider picture of the other tasks. The differences on the Cambridge face perception test and the Cambridge face memory test are not significant, and therefore imply normal facial identity processing. Synesthetes do, however, show superior performance on the expression recognition measure (films task), suggesting superior expression recognition in mirror-touch synesthesia.
Although our findings are focused on facial affect processing, it is also of note that somatosensation may provide a more general component to our understanding of other people's actions and emotions (Keysers et al., 2010). For example, in both humans and monkeys, neural activity in the primary and secondary somatosensory cortices is evoked when performing and when passively observing hand actions (Raos et al., 2004; Evangeliou et al., 2009; Gazzola and Keysers, 2009; Keysers et al., 2010) and the level of activity in this system has been shown to correlate with self-reported empathy in nonsynesthetes (Gazzola et al., 2006). We have previously observed that mirror-touch synesthetes show heightened emotional reactive empathy compared with controls, but do not differ on other components of empathy (Banissy and Ward, 2007). The findings from the current investigation therefore imply that mirror-touch synesthesia may be linked to more general enhancements in emotion processing (e.g., emotional empathy, expression perception) and implicate the somatosensory system in this process.
It also remains to be established whether heightened emotion sensitivity displayed by mirror-touch synesthetes is a cause or consequence of this type of synesthetic experience. Although we assume that mirror-touch synesthetes form part of the synesthetic population, and are therefore a unique group of participants, the principles that bias what type of synesthesia will or will not be developed are a matter of debate (Grossenbacher and Lovelace, 2001; Hubbard and Ramachandran, 2005; Sagiv and Ward, 2006; Bargary and Mitchell, 2008; Cohen Kadosh and Walsh, 2008; Cohen Kadosh et al., 2009). Conceptually there are at least two possibilities: (1) mirror-touch synesthetes reflect the top end of a spectrum along which emotion sensitivity ranges (e.g., the normal architecture for multisensory interactions) and this biases them toward interpersonal synesthetic experience, or (2) mirror-touch synesthetes are a unique population whose extra sensory experiences predispose them to superior emotion sensitivity. Although this is relatively difficult to disentangle, it is of note that mirror-touch synesthesia is not the only variant of synesthesia linked with enhanced perceptual and cognitive processing. For example, synesthetes who experience color as evoked sensations show superior memory for colors (Yaro and Ward, 2007) and enhanced perceptual processing of color (Yaro and Ward, 2007; Banissy et al., 2009a) relative to nonsynesthetes; visual–sound synesthetes (synesthetes for whom seeing visual motion triggers auditory perception) demonstrate an advantage in perceiving visually presented rhythmic patterns compared with nonsynesthetes (Saenz and Koch, 2008); and synesthetes who experience a conscious mapping between time and space (e.g., they consciously perceive months and years in particular spatial layouts) show superior visuospatial abilities compared with nonsynesthetes, but do not show superior performance on tasks that do not make use of their synesthesia (Mann et al., 2009; Simner et al., 2009). Thus, a general feature of synesthesia may be facilitated perceptual and cognitive processing related to the modality of synesthetic experience.
In summary, this study demonstrates that mirror-touch synesthesia is associated with superior facial expression recognition abilities. The observed superiority in face processing is restricted to expression recognition. Mirror-touch synesthetes showed enhanced emotional expression abilities, but did not differ from controls on identity processing measures. Given that mirror-touch synesthesia has been linked to heightened somatosensory simulation, these findings are consistent with simulation-based accounts of expression recognition and indicate that somatosensory resources are an important facet in our ability to recognize the emotions of others.
Footnotes
- Received November 11, 2009.
- Revision received November 24, 2010.
- Accepted December 3, 2010.
-
This work was supported by a Medical Research Council grant to V.W. and an Economic and Social Research Council (ESRC) Fellowship to M.J.B. M.J.B. is supported by a British Academy Postdoctoral Fellowship. J.W. is supported by an ESRC grant. We thank Abigail Birnbaum for help with data collection and Richard Russell for assistance with the research.
- Correspondence should be addressed to Michael Banissy, Institute of Cognitive Neuroscience, University College London, London WC1N 3AR, United Kingdom. m.banissy{at}ucl.ac.uk
- Copyright © 2011 the authors 0270-6474/11/311820-05$15.00/0