Abstract
Learning new identities is crucial for effective social interaction. A critical aspect of this process is the integration of different images from the same face into a view-invariant representation that can be used for recognition. The representation of symmetrical viewpoints has been proposed to be a key computational step in achieving view-invariance. The aim of this study was to determine whether the representation of symmetrical viewpoints in face-selective regions is directly linked to the perception and recognition of face identity. In Experiment 1, we measured fMRI responses while male and female human participants viewed images of real faces from different viewpoints (−90, −45, 0, 45, and 90° from full-face view). Within the face regions, patterns of neural response to symmetrical views (−45 and 45° or −90 and 90°) were more similar than responses to nonsymmetrical views in the fusiform face area and superior temporal sulcus, but not in the occipital face area. In Experiment 2, participants made perceptual similarity judgements to pairs of face images. Images with symmetrical viewpoints were reported as being more similar than nonsymmetric views. In Experiment 3, we asked whether symmetrical views also convey an advantage when learning new faces. We found that recognition was best when participants were tested with novel face images that were symmetrical to the learning viewpoint. Critically, the pattern of perceptual similarity and recognition across different viewpoints predicted the pattern of neural response in face-selective regions. Together, our results provide support for the functional value of symmetry as an intermediate step in generating view-invariant representations.
SIGNIFICANCE STATEMENT The recognition of identity from faces is crucial for successful social interactions. A critical step in this process is the integration of different views into a unified, view-invariant representation. The representation of symmetrical views (e.g., left profile and right profile) has been proposed as an important intermediate step in computing view-invariant representations. We found view symmetric representations were specific to some face-selective regions, but not others. We also show that these neural representations influence the perception of faces. Symmetric views were perceived to be more similar and were recognized more accurately than nonsymmetric views. Moreover, the perception and recognition of faces at different viewpoints predicted patterns of response in those face regions with view symmetric representations.
Introduction
Faces are seen from many different angles in everyday life and differences in viewpoint play an important role in social perception. For example, different orientations provide useful information about internal mental states, such as the focus of attention, and they directly affect social attributions (Sutherland et al., 2017). However, changes in viewpoint make the process of face recognition more difficult, because so many different views can be generated from the same identity. Despite this challenge, we can recognize familiar faces from different viewpoints with relative ease (Hancock et al., 2000), raising the critical theoretical question of how this viewpoint-invariance for recognizing familiar faces is achieved (Young and Burton, 2017; Young, 2018). Cognitive models of face processing have suggested that the recognition of facial identity is based on a view-invariant representation that receives convergent input from relatively viewpoint-specific representations (Bruce and Young, 1986; but see Tarr and Bulthoff, 1998; Burton et al., 1999). Understanding how the brain generates this viewpoint invariant representation is central to understanding how we recognize faces.
Neurophysiological studies have shown that neurons in the temporal lobe can be selective for different facial viewpoints (Perrett et al., 1991). This led to the idea that recognition is initially based on multiple viewpoint-specific representations that are a precursor to viewpoint-invariant representations of identity (Perrett et al., 1998). However, these studies also reported a subpopulation of neurons that showed bimodal responses in which there was selectivity to two different viewpoints, typically symmetrical viewpoints. More recently, Freiwald and Tsao (2010) and Dubois et al. (2015) used fMRI in combination with single-neuron recording in different face regions of the monkey temporal lobe. They found that in the most posterior face regions (ML/MF), neurons were selective for the viewpoint of the face. However, a more anterior face patch (AL) contained neurons that showed mirror-symmetric tuning for viewpoint. The most anterior region (AM) contained view-invariant neurons.
Neuroimaging studies have also found evidence for the representation of viewpoint symmetry in face-selective regions of the human brain. Early studies found viewpoint-selective responses to unfamiliar faces in face-selective regions (Grill-Spector et al., 1999; Andrews and Ewbank, 2004; Fang et al., 2007; Carlin et al., 2011; Guntupalli et al., 2017; Weibert et al., 2018), with partial view-invariance (20–30°) for familiar faces (Eger et al., 2005; Pourtois et al., 2005; Ewbank and Andrews, 2008). More recently, a number of studies have found selectivity to mirror-symmetric viewpoints in face-selective regions (Axelrod and Yovel, 2012; Kietzmann et al., 2012, 2015, 2017; Guntupalli et al., 2017). These studies found that the pattern of response in face regions was more similar for symmetrical views of the face compared with nonsymmetrical views. However, the existence of mirror-symmetric representations in face regions has been challenged by reports maintaining that patterns of response can be better explained by view-dependent representations (Ramirez et al., 2014, Ramírez, 2018).
The existence and location of mirror-symmetric representations of faces is important, because they are often thought to form a key computational step in the generation of viewpoint-invariant representations (Freiwald and Tsao, 2010; Axelrod and Yovel, 2012; Kietzmann et al., 2012). The aim of this study was to determine where mirror-symmetric representations exist and whether there is a direct link with the perception and recognition of faces. Previous behavioral studies using face matching tasks have found better performance on face image pairs showing symmetrical compared with nonsymmetrical viewpoints (Troje and Bülthoff, 1998; Busey and Zaki, 2004). Here, we used perceptual matching and face-learning paradigms to ask whether mirror-symmetric representations in face-selective regions can be predicted by performance on such behavioral tasks. A key feature of our study is the use of real human faces, as opposed to computer generated faces. Human faces are not perfectly mirror symmetrical, so it is important to use real human faces to determine whether the brain represents symmetry in the real world and whether these representations are important for the perception and recognition of identity.
Materials and Methods
Participants
All participants had normal or corrected-to-normal vision with no history of neurological conditions. Twenty right-handed participants (10 female, mean age 25.3 ± 3.1) took part in Experiment 1. Twenty participants took part in Experiment 2 (12 female, mean age 24.2 ± 3.6). Forty-eight participants took part in Experiment 3 (37 female, mean age 23.2 ± 5.2). Written consent was obtained from all participants and the studies were approved by the York Neuroimaging Centre Research Ethics Committee (Experiment 1), the Department of Psychology Ethics Committee at the University of York (Experiment 2) and Durham University (Experiment 3). All experiments conformed to the principles of the Declaration of Helsinki.
Experiment 1
Experimental design.
Face images were taken from the Radboud Faces Database (Langner et al., 2010). There were five stimulus conditions, presented in a block design: (1) right profile (−90°), (2) right ¾ profile (−45°), (3) front view (0°), (4) left ¾ profile (45°), and (5) left profile (90°). These viewpoints were shown across five different identities (Fig. 1). Images were placed onto a 1/f amplitude mask to ensure that all images stimulated the same amount of the visual field despite changes in viewpoint.
Examples of stimuli from Experiment 1. Each column shows the sequence of images in a representative stimulus block for the different conditions. Within each block the viewpoint remained the same, with the identity varying across images.
Images from each viewpoint condition were presented in a blocked design. Each block contained five images (Fig. 1, columns), with each image presented for 1 s followed by a 200 ms gray screen. There was a 9 s inter-block period during which a gray fixation screen was presented. Each viewpoint condition was repeated six times, giving a total of 42 blocks. The order of identities in each block was randomized across blocks. To ensure participants were paying attention during the scan, participants were required to monitor the images and indicate using a response box when they saw a red dot superimposed onto one of the images. Accuracy on this task was very high (99.9 ± 0.5%).
Imaging parameters.
Data for Experiment 1 were collected using a GE 3 Tesla HD Excite MRI system with an eight channel phased array head coil tuned to 127.4 MHz. A gradient echo planar imaging (EPI) sequence was used to acquire the data. The acquisition parameters were as follows: 38 contiguous axial slices, repetition time: 3 s, echo time: 32.5 ms, flip angle: 90°, field-of-view: 28.8 × 28.8 cm, matrix: 128 × 128, slice-thickness: 3 mm, voxel size: 2.25 × 2.25 × 3 mm. To improve registration, the EPI image was coregistered with a T1-weighted image taken in the same plane, before being registered to the high-resolution main structural scan (T1-weighted, 1.13 × 1.13 × 1 mm) of each participant. This was then coregistered to the standard MNI 152 brain.
fMRI analysis.
Our main analysis focused on face-selective regions [fusiform face area (FFA), occipital face area (OFA), superior temporal sulcus (STS), inferior frontal gyrus (IFG); amygdala (AMG)]. There were two important principles underlying the way in which we defined the face-selective regions-of-interest (ROIs). The first principle was that ROIs should be based on independent data. Given that we were investigating the reliability of patterns of response across individuals, it was essential that these came from independent participants. The second principle was that ROIs must be of the same size (number of voxels), to allow the MVPA analyses to have comparable potential power to detect underlying patterns of response in each region.
An independent localizer scan was therefore used to define group level ROIs using different participants (n = 83). Responses to faces that varied in identity and viewpoint were compared with the response to scrambled faces. ROIs comprised of the 500 most significant voxels in the OFA, FFA, and STS (Sormaz et al., 2016). Our analysis was supplemented by using ROIs based on probabilistic visual field maps developed by Wang et al. (2015). Our rationale for using these masks was to determine how the representation of face viewpoint changes from early to higher levels of the visual system. The size of each region in our analysis is shown in Table 1-2.
Pattern analyses were performed using the PyMVPA toolbox (http://www.pymvpa.org/; Hanke et al., 2009). Parameter estimates from a univariate analysis of the main experiment were first normalized by subtracting the average response across the five viewpoint conditions (−90, −45, 0, 45, 90°). The reliabilities of the neural patterns of response were then determined using a modified form of the correlation-based MVPA method devised by Haxby et al. (2001), whereby patterns of response from each participant were compared with the patterns resulting from the group analysis with that participant left out. This leave one participant out (LOPO) method allowed us to determine the consistency of the patterns of response across participants by measuring how similar each participant's responses were to those for the rest of the group. This method has been successfully used in several recent studies from our research group (Rice et al., 2014; Watson et al., 2014; Coggan et al., 2016; Weibert et al., 2018). The group pattern was derived by entering all but one of the participants' data into a higher-level group analysis (mixed effects; FLAME, http://www.fmrib.ox.ac.uk/fsl). This group pattern of response for each condition was then correlated with the pattern from the participant who was omitted from the group. For each unique pair of conditions, the LOPO method was repeated 20 times, with a different participant being omitted from the rest of the group each time. A Fisher's Z-transformation was then applied to the correlations before statistical analysis.
To assess whether there were distinct patterns of response to individual viewpoint directions, paired t tests were used to test the difference between the average within-condition (e.g., −90 vs −90°, −45 vs −45°) and the corresponding between-condition correlations. If a viewpoint evoked a distinct pattern of response, then the within-condition correlations for the individual participant and rest of the group data should be higher than the between-condition correlations in the given region.
Next, a representational similarity analysis (Kriegeskorte et al., 2008) was performed to determine how information about viewpoint was represented. We compared the fit achieved by three models derived from different theoretical perspectives: (1) Viewpoint, (2) Direction, and (3) Symmetry. In the Viewpoint model the value of each cell was proportional to the degree of difference in rotation between viewpoints. In the Direction model, cells involving combinations of viewpoints with the same direction (both left-facing or both right-facing) were given the value 1, whereas all other cells were coded 0. In the Symmetry model, cells showing symmetrical viewpoints were given a value of 1 and nonsymmetrical viewpoints were given a value 0. To prevent differences in the overall magnitude of within-condition and between-condition correlations artificially inflating differences in correlations between matrices, our analysis was only performed on the between-cluster comparisons. All models were normalized using a Z-transform (mean = 0, SD = 1) and then used in a linear regression analysis, with the outcomes defined as the correlation matrices obtained from the MVPA concatenated across LOPO iterations. For each model, elements within the matrix were extracted and flattened to a vector. These vectors were then repeated and tiled to match the number of participants. For each participant, correlation matrices were extracted and flattened to a vector. These vectors were then concatenated and entered into the model as the outcome variable. This analysis yielded a regression coefficient and an error that reflected variance across participants. All regression analyses included a constant term. From this analysis, it was possible to determine the relative fit to each model in each ROI.
Experiment 2
Stimuli and experimental design.
To determine whether symmetrical viewpoints were seen as being more similar than nonsymmetrical viewpoints, we conducted a behavioral study in which participants rated the perceptual similarity of pairs of images which varied in viewpoint. Stimuli consisted of the same greyscale images used in Experiment 1. Images were presented in pairs, with the identity across the two images remaining the same, but the viewpoint changing. Images were presented sequentially, with the first image being presented for 1 s, a 200 ms ISI, and then the second image. Each viewpoint was presented with every other viewpoint, in both the first and second position. For each identity, there were 2 trials for each of the 10 viewpoint combinations. This was repeated for each of the 5 identities, giving a total of 10 trials for each of the 10 viewpoint combinations. The order of trials was randomized for each individual participant. Participants were required to respond with a button press indicating how similar they perceived the images to be, on a scale of 1–7 (1 being less similar and 7 being more similar). Participants were given an unlimited time to respond. The perceptual similarity between symmetric and nonsymmetric responses was compared using a paired t test. The perceptual similarity between different viewpoints was then used as a model in a regression analysis of the fMRI data from different regions.
Experiment 3
Stimuli and experimental design.
Experiment 3 used a face-identity learning paradigm (see Longmore, Liu, and Young, 2008) to determine whether a face learnt in one viewpoint conveyed an advantage in the recognition of the symmetrical viewpoint. Faces from the Radboud database were again used in this experiment. There were 20 male identities each posing a neutral facial expression at the following viewpoints: −90°, −45°, 0°, 45°, 90°. The 20 identities were split into two sets. Participants were randomly assigned to Set 1 or Set 2. Within each set, each of the 10 identities was assigned to one of the five viewpoints. The assignment of identities to viewpoints was randomized for each participant. This generated 10 face images: 2 images for each viewpoint.
In the learning phase, participants were presented with these 10 face images. The faces were presented sequentially, with each face being presented for 5 s, with a 500 ms interstimulus interval (ISI) between each trial. Underneath each face was a first name. Names were randomly assigned to faces for each participant from a set of 10 names. These names were chosen to be short and common in the United Kingdom, consisting of one or two syllables and three or four letters, e.g., Paul, Tim. Participants were instructed to remember the face and its corresponding name.
In the training phase, the 10 faces were split into two blocks of 5 faces. Participants were presented with the first block of five faces. These faces were presented individually, and for an unlimited time. Participants were asked to pick the name that they believed belonged to the face. The five name options were displayed below the face and participants had to use the mouse to click on the name they thought matched the face. The order of the names was random for each participant. Once a response had been recorded, participants were given feedback to indicate whether they were correct or not. If they were incorrect, they were told what the correct name was for the target face. This feedback was provided to aid and reinforce the learning in this training phase. To move to the next block of five faces, the participants had to correctly name each face twice in a row. For example, if a given face was named correctly once, and then incorrectly the next time it was presented, the correct count for this face was returned to 0 and the participant had to complete two more trials correctly in a row, to continue. Once the participant had correctly named the face twice in a row, it was removed from the block. This process was repeated with the remaining block of five faces. Next, the entire set of 10 faces was presented. In this final block of the training phase, participants had to correctly name all 10 faces twice in a row (in the same way as described above) to complete the phase.
In the final test phase of the experiment, participants were presented with all images from the set. This included the 10 images used for training and the remaining 40 images that were not used during training. Images were presented twice, giving 100 trials. The task was to match the name to the face from the 10 names displayed underneath the face. Feedback was not given in this phase of the experiment. For each identity, the aim was to determine whether the (untrained) face images that were symmetrical to the trained view were identified correctly more often than the (untrained) face images that were not symmetrical to the trained view. If participants had correctly learned an image trained in Phases 1 and 2, they should then be able to correctly recognize the same image at this final stage (e.g., when learned in 45° and tested in 45°). For this reason, only identities that were correctly recognized 100% of the time in the test stage when tested in the same viewpoint as they were learned were retained for analysis. The recognition of symmetric and nonsymmetric responses was compared using a paired t test. The recognition rate between different viewpoints was then used as a model in a regression analysis of the fMRI data in different regions.
Results
Experiment 1
Figure 2 shows the results of the MVPA analysis demonstrating the similarity in the patterns of response to different viewpoint directions in the (A) face and (B) visual field regions. To determine whether there were distinct patterns of response to individual viewpoint directions, within-viewpoint (e.g., −90, −90°) correlations were compared with between-viewpoint (e.g., −90, −45°) correlations.
Correlation matrices showing the similarity in the patterns of response across viewpoints in (A) face-selective and (B) visual field regions. C, Distinct patterns of response were demonstrated by higher within-viewpoint correlations compared with between-viewpoint correlations. ***p < 0.001, **p < 0.01, *p < 0.05.
There was a significant difference between within-viewpoint compared with between-viewpoint correlations in all face regions except the AMG and IFG (Fig. 2C; Table 1). To determine whether the size of the face regions was important, we repeated the analysis with smaller (200 voxel) masks and found a similar pattern of results (Table 1-1). There was also significant difference between within-viewpoint compared with between-viewpoint correlations across many visual field areas. This overall pattern demonstrates that there are distinct representations of particular face viewpoints across visual cortex.
Within-viewpoint and between-viewpoint correlations and associated paired t test across all ROIs
Table 1-1
Table 1-2
Next, we asked how similar the pattern of response to viewpoint was across all the ROIs by comparing the neural correlation matrices in Figure 2A,B. Figure 3A shows the similarity in the representation of viewpoint across all regions. To determine the pattern of similarity in the representation across regions, a hierarchical clustering analysis was performed using an unweighted average distance method for computing the distance between clusters and the “1 − correlation” values as the distance metric (Fig. 3B). The distinct clusters shown by the output of the clustering show that the way viewpoint is represented differs between regions.
A, Representational similarity matrix showing the similarity in the neural representations across regions. B, Hierarchical clustering analysis showing regions with similar patterns of response to face viewpoint.
To determine how viewpoint is represented in different regions, our next analysis investigated how three different models of viewpoint representation were able to predict patterns of response. Figure 4 and Table 2 show the models for each representation and the corresponding regression coefficient for each region. To determine whether the size of the face regions was important, we repeated the analysis with smaller (200 voxel) masks and found a similar pattern of results (Table 2-1). The analysis was also repeated across all regions with multiple regression (Table 2-2) and using a permutation test for statistical significance (Table 2-3).
Regression analysis of fMRI data showing how different models predict patterns of response to viewpoint in different regions. A, The Viewpoint and (B) Direction models predict the representational similarity in low-level visual areas. C, In contrast, the symmetry model predicted patterns in high-level regions including the FFA and STS. ***p < 0.001, *p < 0.05.
Regression coefficients for the viewpoint representation models across all ROIs
Table 2-1
Table 2-2
Table 2-3
The Viewpoint and Direction models (Fig. 4A,B) showed a similar pattern with high coefficients in the early visual field regions (V1–V4) and in some of the ventral temporal visual field regions (VO1–VO2). However, the coefficient values were not significant in the lateral occipital visual field regions (LO1, LO2) and the face-selective regions. The only exception was the OFA, which had a significant regression coefficient for both Viewpoint and Direction. The Symmetry model (Fig. 4C) showed an opposite pattern of results. We found significant but negative coefficients in the early visual field regions (V1–V4) and in some of the ventral visual field regions (VO1–VO2). In contrast, there were significant positive coefficients in other ventral visual field regions (PH1–PH2) and in the FFA and STS. The OFA did not show a significant effect for symmetry. The AMG and IFG did not show significant coefficients for any of the three models.
We also analyzed our data to ask whether low-level differences can account for the pattern of data. To investigate the effects of low-level image properties on patterns of neural response in face-selective regions, the image statistics of each object were computed using the GIST descriptor (Oliva and Torralba, 2001). For each image, a vector of 2048 values was obtained by passing the image through a series of 32 Gabor filters (8 orientations at 4 spatial frequencies), and windowing the filtered images along a 8 × 8 grid or 64 spatial locations. Each vector represents the image in terms of the output of each Gabor filter at each position across the image (Rice et al., 2014; Watson et al., 2014, 2016). Image similarities between conditions were measured by correlating the GIST descriptors for all combinations of images. The similarity matrix of the correlation values for the GIST descriptor across all pairwise combinations of conditions was then used as a regressor in a regression analysis with the fMRI data. Table 3 shows that, consistent with previous studies (Rice et al., 2014; Watson et al., 2016; Weibert et al., 2018), low-level stimulus properties can account for some of the variance in the patterns of response in early visual field areas but also in higher visual areas such as the core face regions (OFA, FFA, STS).
Regression coefficients for a model of low-level image properties across ROIs
Experiment 2
Experiment 2 aimed to assess the perceptual experience of facial viewpoint symmetry. Participants were presented with pairs of faces which showed different viewpoints and were asked to rate how similar they believed the images were on a scale of 1–7 (1, less similar; 7, more similar). A similarity matrix of each of the viewpoint combinations can be seen in Figure 5A. To assess whether participants rate symmetrical directions more similar than nonsymmetrical directions, data were averaged across symmetrical and nonsymmetrical pairings. Participants rated symmetrical viewpoints as more similar than nonsymmetrical viewpoint conditions (t(19) = 6.37, p < 0.001).
A, Perceptual similarity ratings between viewpoint directions (left). A regression analysis using the perceptual similarity ratings as a model showed an increase in the coefficients from low-level to high-level visual regions, with the highest values in the FFA and STS. B, Recognition rates for different combinations of viewpoint during the training and test phases of the recognition experiment (left). A regression analysis using the recognition values as a model showed an increase in the coefficients from low-level to high-level regions, with the highest values in the FFA and STS. ***p < 0.001, **p < 0.01, *p < 0.05.
Next, we asked whether the pattern of response in the perceptual similarity task could predict the patterns of response in different regions of the brain (Fig. 5A, right). Using a regression analysis with perceptual similarity as the model, we found that responses from V1 were not predicted by the pattern of perceptual similarity. However, the correlation coefficients showed a progressive increase along the visual hierarchy with the highest regression coefficients in the FFA and STS (Table 3). This shows a clear link between the perceptual similarity of different viewpoints and the pattern of response in some face regions.
Experiment 3
The aim of Experiment 3 was to determine whether learning a face at a particular viewpoint conferred an advantage in the recognition of the symmetric viewpoint. Figure 5B shows the percentage recognition rates for different combinations of viewpoint from training and test phases of the experiment. To compare performance across symmetrical and nonsymmetrical conditions, data were averaged across symmetrical and nonsymmetrical viewpoint combinations. Participants performed significantly better when tested with a viewpoint that was symmetrical to the one which they had learned (t(47) = 2.63, p = 0.012).
Next, we asked whether accuracy in the recognition task could predict the patterns of response in different regions of the brain (Fig. 5B, right). Using a regression analysis with recognition accuracy as the model, we found that early visual areas were not significant. In contrast, only LO2, FFA, and STS showed positive regression coefficients (Table 3). This demonstrates a link between behavioral performance on a face learning task and patterns of response in the face-selective regions, such as the FFA.
To determine whether the size of the face regions was important, we repeated the analysis of Experiment 2 and Experiment 3 with smaller (200 voxel) masks and found a similar pattern of results (Table 4 and Table 4-1). The analysis was also repeated across all regions with multiple regression (Table 4-2). A multiple regression found significant effects of Similarity across many visual areas, but was largest in the FFA and STS. However, we did not find any additional benefit of the Recognition model. This is likely explained by the similarity in the models which have a correlation of r = 0.53. Finally, we reanalyzed the data using a permutation test for statistical significance (Table 4-3).
Regression coefficients demonstrating the ability of the behavioral data from Experiment 2 (Similarity ratings) and Experiment 3 (Recognition) in predicting the neural responses across all ROIs
Table 4-1
Table 4-2
Table 4-3
Discussion
The aim of this study was to investigate and understand responses to symmetric views of real human faces in face-selective regions of the human brain. A viewpoint-symmetric representation was found in the FFA and STS, but not in the OFA (Experiment 1). To determine whether a symmetrical representation of viewpoint can convey an advantage in the perception and recognition of faces, participants performed a perceptual similarity and a face recognition task. We found that symmetric viewpoints were perceived to be more similar than nonsymmetric viewpoints (Experiment 2) and that identities learnt at a particular viewpoint were more accurately recognized at the symmetrical viewpoint compared with nonsymmetrical viewpoints (Experiment 3). Critically, these behavioral judgements of symmetry and recognition across different views were able to predict patterns of response in face-selective brain regions.
Previous neurophysiological studies have found a large number of neurons with maximal responses to specific viewpoints (Perrett et al., 1991, 1998; Freiwald and Tsao, 2010; Dubois et al., 2015). We also found distinct patterns of response to viewpoint throughout visual cortex. Our findings are therefore consistent with previous neuroimaging studies that have also found distinct patterns of response to specific viewpoints (Carlin et al., 2011; Axelrod and Yovel, 2012; Kietzmann et al., 2012; Ramirez et al., 2014; Dubois et al., 2015; Guntupalli et al., 2017). These neural responses to viewpoint are also consistent with behavioral studies that have shown the importance of viewpoint-selective representations in the perception and recognition of unfamiliar faces (Bruce, 1982; Hill and Bruce, 1996; Fang and He, 2005; Longmore et al., 2008).
This widespread evidence of viewpoint selectivity to face images across the visual cortex provides a challenge to understanding how invariant representations of facial identity are achieved (Perrett et al., 1998). One possibility is that different viewpoint-selective units converge to generate invariant units of facial identity, similar to the face recognition units suggested by cognitive models of face recognition (Bruce and Young, 1986; Burton et al., 1999). However, the discovery of neurons that are tuned to symmetric viewpoints of the face has led to the idea that these may provide an important intermediate computational step before full invariance is achieved (Freiwald and Tsao, 2010), or at least facilitate the process. Evidence that these viewpoint symmetric representations are found in the human brain comes from neuroimaging studies, which have found that the patterns of response in face regions to symmetrical viewpoints are more similar than to nonsymmetrical viewpoints (Axelrod and Yovel, 2012; Kietzmann et al., 2012, 2017; Guntupalli et al., 2017). However, there has been some inconsistency in the literature about which regions show a symmetrical representation of faces. Kietzmann et al. (2012, 2017) found viewpoint symmetry represented in the OFA and FFA. However, other studies report symmetrical representations in the FFA and STS, but not in the OFA, leading them to conclude that there is a hierarchical processing of face viewpoint in face regions (Axelrod and Yovel, 2012; Guntupalli et al., 2017). Finally, Ramirez et al. (2014, 2018) have argued that symmetry responses in the FFA could be better explained by a view-dependent mechanism. In contrast, to the current and previous studies, they presented faces in the periphery to test the invariance of viewpoint symmetry. So, it is possible that a lack of position invariance could explain the difference in results across studies (Kietzmann et al., 2017). Another possible reason for the difference in results could be the methodological choices in MVPA. However, view symmetry in faces has been shown across different studies that have used a range of MVPA approaches (Axelrod and Yovel, 2012; Kietzmann et al., 2012, 2017; Guntupalli et al., 2017).
To determine where viewpoint symmetry is represented, we compared how three different models (Viewpoint, Direction, and Symmetry) predicted patterns of neural response in different regions of visual cortex. The predictions of the Viewpoint model were exclusively based on the angular separation between the different viewpoints, the Direction model coded whether combinations of viewpoints were both left-facing or right-facing, and the Symmetry model explicitly allowed different symmetric orientations (e.g., −45 and +45, or −90 and +90) to be represented as similar to each other regardless of the angular separations themselves (which are 180° for −90 and +90 images and only 90° for −45 and +45 images).
We found that the Viewpoint and Direction models best predicted responses in early visual cortex (V1–V4) and the OFA, but showed a gradual decline in high-level regions and were not able to explain the patterns in the FFA and STS. In contrast, the patterns of response in the FFA and STS (but not the OFA) were best predicted by the Symmetry model. These findings are consistent with a hierarchical organization of viewpoint responsiveness across visual regions in which more posterior regions have view-dependent representations, but more anterior regions (including classic face-selective regions) are sensitive to viewpoint symmetry (Freiwald and Tsao, 2010; Axelrod and Yovel, 2012; Kietzmann et al., 2012, 2017; Guntupalli et al., 2017).
Although our results provide further support for the representation of viewpoint symmetry in face-selective neural regions, such as the FFA, it has not been clear whether these representations are important for the perception and recognition of faces. That is, the link between neural and behavioral measures has not been investigated directly. To address this issue, we compared symmetrical and nonsymmetrical views in a perceptual similarity task and a recognition task. We found that symmetrical views of the face are perceived to be more similar than nonsymmetrical views. Similarly, we found that novel face images that were symmetric to a learned face view were recognized better than nonsymmetric face views. These results are consistent with previous studies that have shown a behavioral advantage for symmetric compared with nonsymmetric viewpoints (Troje and Bülthoff, 1998; Busey and Zaki, 2004). However, to investigate the link between symmetric and nonsymmetric viewpoints and neural responses, we used the data from our behavioral results to predict patterns of response across visual cortex. The data from both the perceptual similarity and recognition experiments showed a progressive increase in their ability to predict neural responses from low-level to high-level regions. Patterns of response in face regions, such as the FFA and STS, were predicted best by performance on both the perceptual similarity and recognition tasks. This provides the first evidence for a close link between symmetrical representations in the brain and a behavioral advantage in the recognition of faces.
A crucial difference between this study and many previous studies investigating symmetry is the use of real faces. Many previous studies have used computer generated faces that are often themselves mirror symmetric. However, human faces are not actually completely symmetrical (Fig. 1). So, if the brain uses symmetry to assist reaching viewpoint invariance, it needs to be able to allow for these deviations from symmetry found in real human faces. We have been able to show that neural and behavioral responses are able to compensate for this lack of full mirror symmetry.
Although this study has focused on face processing, a preference for bilateral symmetry is a more general property of the visual system (Bertamini et al., 2018; Keefe et al., 2018). The bias in neural processing is also evident in perceptual judgements showing that we are adept at discriminating small deviations in bilateral symmetry (Corballis and Beale, 1976; Rhodes et al., 2005) and find bilaterally symmetrical images more esthetically pleasing than nonsymmetric images (Jacobsen et al., 2006; Makin et al., 2012). Our study differs from these studies as bilateral symmetry is not evident in any one image itself. Rather, we have shown that symmetry enhances the integration over time of two images. Nevertheless, we did find that sensitivity to viewpoint symmetry was evident in regions such as PH1 and PH2. Future studies will be necessary to determine the extent to which similar neural and perceptual mechanisms are involved in both processes.
An important feature of our findings is that the spatial patterns of response to viewpoint generalized across participants. This observation complements other neuroimaging studies using univariate methods that have already shown that the locations of face-selective regions in the ventral visual pathway are broadly consistent across individuals (Kanwisher et al., 1997; Davies-Thompson and Andrews, 2012). This implies that common principles may well underpin the organization of these regions. In our analysis, we used multivariate methods to compare the spatial pattern of response in each individual with the spatial pattern from the rest of the group of participants (Poldrack et al., 2009; Rice et al., 2014; Watson et al., 2014; Coggan et al., 2016; Weibert et al., 2018). The success of this approach implies that much of the topographic pattern of response to facial viewpoint is consistent across individuals. Of course, it is possible that a finer-grained within-participant analysis could reveal more information. However, it is unclear how this could lead to a completely different pattern of response. Indeed, our results are consistent with many previous studies using within-participant analyses (Axelrod and Yovel, 2012; Kietzmann et al., 2012; Guntupalli et al., 2017). These observations are significant in that they suggest that our findings reflect the operation of large-scale organizing principles that are consistent across different individuals.
In conclusion, our results demonstrate that the spatial patterns of responses to facial viewpoint in the FFA and STS are sensitive to symmetry. A model that explicitly represented image symmetry was better able to predict patterns of response in these face regions than models based exclusively on image viewpoint or direction. We also found that symmetrical viewpoints are perceived to be more similar and are more easily recognized than nonsymmetrical viewpoints in purely behavioral tasks. Finally, we were able to establish a direct link between the neuroimaging and behavioral findings by showing that these behavioral data could predict patterns of response in face-selective regions, such as the FFA and STS. Together, these results support the idea that symmetrical representations are an important computational step in the generation of view-invariant representations of faces that are essential to familiar face recognition.
Footnotes
T.R.F. was supported by a studentship from the University of York. We thank the staff at the York Neuroimaging Centre for their help throughout this project, and David Watson and David Coggan for help with the analysis.
The authors declare no competing financial interests.
- Correspondence should be addressed to Timothy J. Andrews at timothy.andrews{at}york.ac.uk
