Intracranial Electroencephalography and Deep Neural Networks Reveal Shared Substrates for Representations of Face Identity and Expressions

Stimuli.

Subjects viewed face images from the Karolinska Directed Emotional Faces (KDEF) dataset (Lundqvist et al., 1998). The KDEF dataset consists of 4900 images depicting 70 individuals (50% female) showing seven different expressions from five different angles. The following expression categories were included in the experiment: happy, sad, afraid, angry, and neutral. Each combination of a face identity and a facial expression was shown in different viewpoints, including 0° (frontal view), 45° (left and right views), and 90° (profile; left and right views).

Experimental paradigm.

Before completing the main task, participants completed a functional localizer task (Li et al., 2019; Boring et al., 2021). Subjects were shown images of faces, houses, bodies, words, hammers, and phase-scrambled faces. More details about the design of the functional localizer can be found in the studies by Li et al. (2019) and Boring et al. (2021). The data from the functional localizer was used to identify electrodes that respond selectively to faces. An electrode was deemed face selective using the criteria described in the Electrode localization section.

Two different sets of participants completed two different versions of the experiment (Li et al., 2019; Boring et al., 2021), which we will refer to as A and B. In both experiments, each trial began with a face image presented for 1000 ms. This was followed by a 500 ms intertrial interval, during which a fixation cross was presented at the center of the screen. Subjects were instructed to press a button to identify whether the presented face was male or female. Subjects were asked to respond as quickly and as accurately as possible. A set of 10 practice trials was executed before the start of the experiment.

In experiment A, each subject performed one session containing 600 trials. Subjects viewed a set of stimuli that contained eight identities, five expressions, and five viewpoint angles (left/right profile, left/right 45°, and frontal). Each stimulus was presented three times within a session. In experiment B, subjects performed at least two sessions and viewed a different subset of KDEF stimuli. Subjects viewed a set of stimuli that contained 40 identities, five expressions, and three viewpoint angles (profile, 45°, and frontal). Each stimulus was shown only once per session.

Data preprocessing.

Data were preprocessed at the University of Pittsburgh. Further details can be found in the studies by Li et al. (2019) and Boring et al. (2021). The data analyzed here contain 14 depth electrodes and 11 surface electrodes. Depth electrodes and surface electrodes were used to record local field potentials at 1000 Hz. Reference and ground electrodes were distantly placed from the recording electrodes subdurally and having contacts oriented toward the dura. Surface area of the recording site was similar across grid and strip electrode contacts. In this manuscript, “electrode contacts” will be referred to as “electrodes.” There were no consistent differences in neural responses observed between the grid and depth electrodes. To extract single-trial potential signals, the raw data were bandpass filtered, preserving the frequencies from 0.2 to 115 Hz. This step was implemented using a fourth-order Butterworth filter. After removing slow and linear drift as well as high-frequency noise, a 60 Hz line noise was also removed with 55–65 Hz as the stop band. Single-trial potentials were time locked to the stimulus onset for the trial with the signal sampled at 1000 Hz.

Raw data were also inspected to identify and reduce artifacts. There were no ictal events detected. The mean maximum amplitude across all trials was computed, and any trials with a maximum amplitude 5 SDs above the mean were discarded. Trials that had a difference of ≥25 µV between back-to-back sampling instances were discarded as well. This resulted in <1% of trials being removed.

Electrode localization.

The location of the electrodes (Fig. 2A) was determined using an automated method that was used to coregister grid electrodes and electrode strips (Hermes et al., 2010). Patient high-resolution postoperative CT scans were coregistered with anatomic MRI scans to section electrode contacts before patients underwent surgery and implantation of the electrodes. Preoperative and postoperative imaging scans were also used to localize stereo EEG electrodes. Face-selective electrodes were identified by analyzing data from a functional localizer, during which participants were shown images of faces, bodies, words, hammers, houses, and scrambled faces. An electrode was defined as face selective if its temporal response patterns could be used to decode faces from other object categories significantly above chance (Li et al., 2019;Boring et al., 2021).

Deep convolutional neural network models.

DCNNs were implemented to model the neural data. Each network was trained to perform one task only, either identity recognition or expression recognition. Therefore, identity-trained models will be referred to as identity DCNNs, and the expression-trained models as expression DCNNs. For both the identity and expression DCNNs, we used a densely connected architecture (DenseNet; Huang et al., 2017; Fig. 1A) as well as a residual neural network (ResNet-18) architecture.

The identity DCNNs were trained to label identities using the CelebA dataset (http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). CelebA consists of >300,000 images. To match the size of the dataset used for the two networks, a subset of CelebA was used. The subset contained 28,709 images for training and an additional 3589 images labeled for testing, containing a total of 1503 identities. These identities were randomly chosen, ensuring that at least 20 images were available for each identity. All images were sized 48 × 48 pixels and grayscale.

The expression DCNNs were trained to label expressions using face images from the Facial Expression Recognition 2013 (FER2013) dataset (Goodfellow et al., 2013). The dataset is split to contain 28,709 images specified for training and 3589 images labeled as “public test” for validation. All images were sized 48 × 48 pixels and grayscale.

Once trained, the DCNNs were tested on their ability to perform identity and expression recognition using the KDEF dataset (Lundqvist et al., 1998). This was done by freezing the weights of DCNNs and extracting the activations of units in the last convolutional layer of each network for each of the images. Activations for the different images were then used as the inputs to a simple readout layer. To test for identity labeling, the readout layer was trained on all KDEF images except from one expression category (85.7% train, 14.3% test). The left-out expression category was then used to test the ability of the network to label identity. Cross-validation was performed so that each expression category could be left out for training (seven testing sets) and performances were averaged. To test for expression labeling, images from seven identities were held out of the training set for the readout layer (90% train, 10% test). Cross-validation was performed so that each set of 7 identities could be left out for training (10 testing sets), and performances were averaged.

The DenseNet trained to recognize identity achieved an accuracy of 26.5% on a left-out subset of CelebA, and the DenseNet trained to recognize expression achieved an accuracy of 63.5% on a left-out subset of FER2013 (Schwartz et al., 2023) Using the DenseNet, each network was able to transfer to KDEF for the task it was trained to perform (identity DenseNet on identity recognition: accuracy = 95.2%, chance level = 1.42%; expression DenseNet on expression recognition: accuracy = 81.9%, chance level = 14.2%). The identity DenseNet was able to label facial expression on the KDEF dataset with an accuracy of 77.7%. The expression DenseNet was able to label face identity on the KDEF dataset with an accuracy of 89.7%.

To facilitate comparison with previous studies, we additionally trained an identity and an expression DCNN based on the ResNet architecture (ResNet-18; He et al., 2016). The ResNet-18 networks were trained using the same datasets that were used for the DenseNets. The ResNet-18 trained to recognize identity achieved an accuracy of 28.0% on a left-out subset of CelebA, and 91.5% on KDEF (chance level = 1.42%). The ResNet-18 trained to recognize expressions achieved an accuracy of 61.3% on a left-out subset of FER2013, and 66.4% on KDEF (chance level = 14.2%). When transferring to the different tasks, the identity ResNet-18-labeled facial expression and the expression ResNet-18-labeled identity with accuracies of 55.7% and 80.1% on KDEF, respectively. Therefore, both DCNNs performed better than chance on left-out images from the datasets that they were trained on, as well as on images from the KDEF dataset. However, they did not transfer to KDEF as well as the DenseNets. A ResNet-18 pretrained on ImageNet to perform object recognition (henceforth referred to as the object ResNet-18) was implemented as an additional model comparison. Details on the training can be found in the study by He et al. (2016). The object ResNet-18 was trained using images in RGB mode. Since the identity and expression DCNNs were trained using grayscale images, we modified the weights of the conv1 layer here by summing over the dimension of the input channels. The object ResNet-18 was able to label identity and expression on KDEF images with accuracies of 96.2% (chance level = 1.42%) and 61.4% (chance level = 14.2%), respectively. A randomly initialized DenseNet and Resnet-18 (same architectures as trained DCNNs) were also used as additional control analyses.

Training and testing datasets comparisons.

Since we could not access a sufficiently large dataset including both identity and expression labels, the identity DCNNs and expression DCNNs were trained using two different datasets. It is possible that the testing dataset (KDEF) might be more similar to one of the two training datasets (either CelebA or FER2013). If this is the case, the networks for which the training and testing datasets are more similar might perform better. To test this possibility, both training datasets were compared with the testing dataset by evaluating the similarity between image representations using features from the object ResNet-18 (see “Deep neural network models”). The object-trained ResNet-18 was used to extract feature representations from different layers for images in the identity and expression training datasets, and for images in the testing dataset. For each layer, Pearson r correlation coefficients were computed between the features of image pairs where one image is taken from the testing dataset (KDEF) and one from either the identity or expression training dataset (this analysis was performed separately for each training dataset). Correlations were computed for one channel at a time and averaged across channels. This was done for 100 different randomly chosen image pairs, and the correlations were averaged across the pairs. For each layer, this procedure yielded a measure of the similarity between the training and testing datasets based on the features in that particular layer. To estimate the robustness of the results, the analysis was conducted 10 times, each time selecting a different randomly chosen set of image pairs.

When comparing images from CelebA and KDEF, this analysis yielded mean values of 0.1254, 0.3606, 0.1894, 0.2430, and 0.1244 for conv1, and hidden layers 1−4 respectively. When comparing images from FER2013 and KDEF, this analysis yielded mean values of 0.1708, 0.4263, 0.2141, 0.2739, and 0.0848 for conv1, and hidden layers 1–4, respectively.

The similarity between the training datasets and the testing dataset is comparable. In addition, neither of the training datasets is more similar to the testing dataset for all layers of the object ResNet-18. If anything, the FER2013 dataset shows greater similarity to KDEF for most layers. Therefore, if the CelebA-trained networks were to better account for neural responses, it would be unlikely that this is because of CelebA being more similar to the testing dataset (KDEF).

Representational similarity analysis: comparison between DCNNs.

Before comparing the representations in DCNNs to neural responses, we sought to quantify the differences in the representations learned by the identity DCNNs and by the expression DCNNs. Transfer-learning tests conducted in a previous study demonstrate that these DCNNs learn representations that can be used to perform the other task with above-chance accuracy (Schwartz et al., 2023). For example, representations in layers of the expression of DenseNet could be used to read out the identity of faces (Schwartz et al., 2023). However, this does not imply that the identity and expression DenseNets have the same representations.

To test the similarity of the representations in the two DCNNs, we used representational similarity analysis (RSA). We analyzed the representations in multiple hidden layers of the neural networks. Specifically, features from either four or five hidden layers were extracted: the first convolutional layer, and the last layer in each of the three dense blocks (after shrinkage) or the last layer in each of the four residual blocks. For each of these layers, we calculated RDMs using a three-step procedure. First, we extracted feature vectors for all KDEF images used in the experiment. Next, we mean centered the feature vectors by calculating and subtracting the mean feature vector across all KDEF images. Finally, for all pairs of images, we calculated the correlation distance between their mean-centered feature vectors (correlation distance is 1 − r, where r is Pearson's correlation). In experiment B, information about viewpoint only included the viewpoint angle, without distinguishing between left and right viewpoints, therefore the feature vectors for the left and right viewpoints were averaged (i.e., left and right profile views averaged, left and right half views averaged).

This procedure produced RDMs of size 200 × 200 for experiment A, and RDMs of size 600 × 600 for experiment B (Fig. 1B,C). Note that, as described in the Experimental paradigm subsection, the sizes of the RDMs are different in the two experiments because different subsets of the KDEF images were used in experiment A and experiment B. In the end, the Kendall τ rank correlation coefficient (τ_B)was used to compute the similarity between the RDMs from different layers in the two different DCNNs. A 4 × 4 cross-network similarity matrix for the trained DenseNets is shown in Figure 1D.

Representational similarity analysis of neural data.

To retain as much data as possible, we initially performed an analysis on all of the face-selective electrodes, including those from participants who were shown each stimulus once. In this analysis, we computed separate RDMs for each of three temporal windows (125–175, 175–225, and 225–275 ms). This specific temporal range was chosen based on previous studies on the temporal dynamics of visual face perception (Barbeau et al., 2008). As discussed in more detail later, it remains possible that some face information might be encoded in later time windows as well (Ghuman et al., 2014; Li et al., 2019; Boring et al., 2021; see also the Temporal localizer subsection). For each temporal window per each electrode, we extracted a 50-dimensional vector, such that the value for each dimension reflects the amount of measured response in the corresponding millisecond of the 50 ms window. RDMs were obtained by following the same procedure used for the DCNN RDMs, using correlation distance to determine the dissimilarity between the response patterns for each pair of stimuli. As in the RSA for the DCNNs, the average response over all stimuli was subtracted from each stimulus response to remove any baseline that is stimulus independent.

In addition to this, we performed an RSA restricted to highly reliable responses from electrodes located in the fusiform gyrus (n = 7). Highly reliable electrodes and time windows were identified following the procedure described below. We then extracted patterns of response from each reliable electrode and time window, and performed the RSA following the same approach described in the previous paragraph, comparing neural RDMs to RDMs extracted from the DenseNet models.

Temporal localizer.

We sought to identify time windows during which face-selective electrodes show the most reliable responses. The data time series was segmented using disjoint, successive time windows of 50 ms. The first window was centered at 0 ms post-stimulus onset, and the last at 500 ms. Therefore, the windows included time points starting from 25 ms before stimulus onset to 525 ms after onset. Disjoint windows were used to reduce the number of multiple comparisons. To identify which of the time windows contained relatively less noise compared with the amount of base signal, all presentations of the neural response of a stimulus were correlated within a specific time window. An average correlation over all time windows for that stimulus was obtained as well. The average correlations across all time windows were then subtracted from the time window-specific correlations. A paired t test was performed between the correlation of the stimulus responses for a given time window and the average correlation of the mean response averaged over all time windows for that stimulus (p < 0.05). This determined which of the time windows contained response patterns whose test–retest reliability was significantly higher than average. To correct for multiple comparisons, all t tests were Bonferroni corrected. One electrode was excluded from the RDM analysis because of not containing any time windows with reliable responses (p > 0.05).

Representational similarity analysis: comparison between neural activity and DCNNs.

We next aimed to compare the neural representations in specific time windows to the representations in the DCNNs. In particular, we evaluated the extent to which RDMs computed using the identity DCNNs and using the expression DCNNs correlate with RDMs based on the iEEG measurements (Kriegeskorte and Kievit, 2013; Khaligh-Razavi and Kriegeskorte, 2014). To calculate the concordance between the RDMs of the DCNN and the neural RDMs, we performed two types of analysis. In the first type of analysis, we compared the RDMs extracted from neural data to RDMs extracted from individual layers of the identity DCNNs and of the expression DCNNs, calculating the Kendall τ_B. Since negative variance-explained values are uninterpretable, any negative τ_B correlations were set to 0 (Fang et al., 2022). The smallest of the negative values was −0.003. This affected a total of 37 of 360 τ values. This procedure was repeated using RDMs from the object ResNet-18 and untrained DCNNs as well. However, since this analysis compares neural representations to the representations in one DCNN layer at a time, one limitation of this analysis is that it does not capture the overall correspondence between neural data and the representations across all layers of a DCNN jointly.

Comparing neural representations to DCNN representations one layer at a time does not reveal to what extent different layers of the DCNN encode redundant information or unique information. To address this question, we introduced a new type of analysis using semipartial Kendall τ rank correlation (Kim, 2015) to evaluate the overall correspondence between the RDMs extracted from the neural data and each of the identity and expression DCNNs when considering jointly the representations in all layers of the DCNNs.

Semipartial correlations measure the strength of the relationship between two variables (i.e., between the neural RDM and the first hidden block RDM) while controlling for the effects of other variables (i.e., the initial convolutional RDM). Within each DCNN model, the semipartial τ_B was calculated for each layer, controlling for the effect of the previous layers. Then, the semipartial τ_B values were summed to obtain a cumulative τ_B value. This allows one to control for redundancy between the layers, evaluating the overall similarity between the models and the data without inflating the τ_B values.

After calculating the semipartial τ_B values between the face-selective electrodes and the identity and expression DCNNs, we performed model comparison using Bayes factor to potentially establish evidence for the absence of differences between the ability of DCNN to account for neural responses (Keysers et al., 2020). This was done to evaluate the statistical evidence for the possibility that there is no difference between the identity DCNN's representational similarity to the neural representations and the expression DCNN's representational similarity to the neural representations (and more precisely, that they come from a same distribution. The analysis with Bayes factor was performed using the set of all face-selective electrodes to maximize statistical power.

Relative contribution of identity and expression.

Next, we set out to test whether different sets of electrodes were more strongly correlated with one DCNN over the other. The dataset included electrodes located in the ventral stream as well as electrodes located in lateral temporal regions. If ventral regions are specialized for identity recognition and lateral regions are specialized for expression recognition, ventral electrodes might have a greater cumulative τ_B with the identity DCNN, while lateral electrodes might have a greater cumulative τ_B with the expression DCNN. Alternatively, electrodes in ventral and lateral regions might be similar in terms of their relative correspondence to the identity DCNN and to the expression DCNN.

To compare the relative similarity of neural RDMs in individual electrodes to the RDMs of the identity and expression DCNNs, each electrode at each time window was plotted as a point in a 2D space, where the coordinate along the x-axis was determined by the cumulative Kendall τ_B between the electrode RDM and the identity DCNN RDM, and the coordinate along the y-axis was determined by the cumulative Kendall τ_B between the RDM of the electrode and the expression of DCNN RDM. If ventral electrodes have comparatively higher Kendall τ_B values with the identity DCNN, and lateral electrodes have comparatively higher Kendall τ_B values with the expression DCNN, the two sets of electrodes should fall on lines with different slopes, where the slopes correspond to the ratio between the cumulative τ_B for the identity model and the cumulative τ_B values for the expression model. In particular, electrodes in the ventral stream that are comparatively better explained by the identity DCNN should fall on a line that is closer to the identity axis, while electrodes in the lateral stream should fall on a line that is closer to the expression axis (despite electrodes varying in how well they are explained overall). This would demonstrate the presence of an interaction between DCNN model type and brain region (in line with the classical view). By contrast, if all the electrodes fall on the same line, it means that the relative performance of the identity and expression DCNN models at explaining neural responses is similar for the two streams (in contrast with the classical view), demonstrating the absence of an interaction between DCNN model type and brain region.

Frequentist tests are designed to test for the presence of significant interactions, but a lack of significant effects does not demonstrate no interaction. This makes it challenging to test for the absence of an interaction. However, Bayesian tests are built in such a way that they can evaluate the strength of evidence for the absence of an effect. Thus, a Bayesian approach is implemented to evaluate the relative support for a model in which all the electrodes fall on the same line compared with a model in which the electrodes can fall on two separate lines, one for each stream.

To statistically test whether ventral and lateral electrodes fall on lines with different slopes, we fit the data with two competing linear regression models: one model with two separate slopes for the ventral and lateral electrodes, and one model with a single slope. We then performed model selection with the Bayesian information criterion (BIC) to determine which linear regression model provides a better account for the data. A lower BIC score signifies the better model. The difference between BIC scores, δ = BIC_separate − BIC_combined, determines the size of the effect: a difference >10 denotes strong evidence for the better model (Raftery, 1995).

To further examine the ratio between identity and expression model performance for the DenseNet models, we calculated an index ranging from −∞ to ∞, where negative values indicate that the neural representations correlate more with representations in the expression DCNN, and positive values indicate that they correlate more with the identity DCNN. To accomplish this, for each electrode and time window, we calculated the index log ratio (LR) = log(τ_id/τ_exp). This was then plotted as a histogram where LRs between −∞ and 0 represent expression-preferring electrode/time window combinations and LRs between 0 and +∞ represent identity-preferring electrode/time window combinations. When conducting comparisons with the DenseNets, three electrodes had cumulative τ_B values ≤0 in one time window; therefore, the log ratio could not be calculated, and they were not included in the log ratio histogram (Fig. 3B).