Article Figures & Data
Figures
- Figure 1.
Stimulus sets used in the experiments. A, One example of each image from the eight categories in the FOB stimulus set. Each category contained 16 images, for a total of 128 images in this stimulus set. B, Average HMAX S1 and C2 image similarity (Pearson's correlation) between all images in each of the eight categories in the FOB dataset, averaged over all pairs of the 16 images in each category. HMAX S1 features are responses to Gabor filters and thus mimic properties of V1, while C2 features are modeled on higher level visual responses (Serre et al., 2007; Mutch et al., 2010). Error bars show the SD in the HMAX feature correlations across pairs of the 16 images. As can be seen, images in the face category were most similar to each other while images in the scrambles image category were most different from each other. C, An example of the eight head orientations for 1 of the 25 people in the FV stimulus set.
- Figure 2.
Individual image decoding accuracies and average firing rates in the three face patches to each image in the face, body, and object stimulus set. A, The sorted decoding accuracies for each image in the FOB set. Red bars indicate decoding accuracies of face images. The three nonface images with the highest decoding accuracies for each patch are shown as insets at the top of each subplot. The decoding accuracies for all face images are in the top 50% for ML/MF and AL, and all 16 face images had the highest decoding accuracies in AM. B, The average population firing rate for each image (in the same order as A). Total population firing rates for all face images (apart from one image in AL) were higher than for nonface images. C, Individual image decoding accuracies plotted as a function of individual image firing rates. There is a positive correlation between decoding accuracies and firing rates for both face and nonface images (significant correlations are denoted in green with an asterisk).
- Figure 3.
Decoding accuracies for discriminating between the 16 objects within each category on the face, body, and object stimulus set. The within-category decoding accuracy for faces is similar to the accuracy levels seen in several other object categories in ML/MF and AL (right, center subplots) while the within-category decoding accuracy for faces is much higher than the decoding accuracy of other object categories in AM (right subplot). These results show that for most categories, images within the same category elicit different neural responses. The black bars under the plots indicate the time when the stimulus was present, the orange bars under the plot indicate the length of the decoding time window, with the orange circle indicating that the decoding accuracies are aligned to the center of this bin. Chance decoding is 1/16.
- Figure 4.
Face versus nonface, and category1 versus category2 decoding accuracies. A, Decoding accuracies for determining whether an image was a face versus a nonface (face detection) using data from ML/MF (blue), AL (green), and AM (red). The decoding analysis was run separately seven times for face versus each of the nonface categories, and the decoding accuracies were averaged over the seven results. B, A comparison of category 1 versus category 2 decoding for the nonface categories (colors for the different brain regions are the same as in A). The decoding analysis was run for all 21 pairs of nonface category 1 versus nonface category 2, and the plotted results are the average of each of these 21 decoding analyses. The black bars under the plots indicate the time when the stimulus was present, and the orange bars indicate the length of the decoding time window. C, All 28 results for discriminating between all pairs of categories using 300 ms of data (i.e., all pairs of decoding results that went into the averages in A and B). Chance on these binary decoding tasks is 50%.
- Figure 5.
Pose-specific and pose-invariant face identity information. A classifier was trained to discriminate between individuals using data from one head orientation and then tested with data from another head orientation. The left column shows results from ML/MF, middle column from AL, and right column from AM. First row is for training on the left profile, middle row for training on the straight pose, and bottom row for training on the right profile face. Each bar in the histogram shows decoding accuracy as the result of testing at a specific location (L, left profile; 3/4 L, left 3/4 profile; S, straight; 3/4 R, right 3/4 profile; R, right profile; U, head tiled up; D, head tilted down; B, back of the head). Cyan indicates statistical significant results (p < 0.005 permutation test), and dark blue indicates failure to reach significance. We see higher decoding accuracy in AL and AM compared with ML/MF. Additionally, we see an increase in pose-invariant generalization from training on one head orientation and testing on a different head orientation going from ML/MF to AL to AM.
- Figure 6.
Average pose-specific and pose-invariant identity decoding results. Results from decoding the identity of a face when training and testing a classifier with images from the same head orientation (pose-specific decoding, blue trace) or with images from different head orientations (pose-invariant decoding, red trace). Pose-specific decoding results are averaged over all seven conditions for training and testing on the same head orientation, while the pose-invariant decoding is averaged over all 36 pose-invariant decoding permutations (back of head conditions were not included in this analysis). Error bars indicate the SDs over all these decoding results. The results are shown for ML/MF (left), AL (middle), and AM (right). AL and AM have higher pose-specific decoding accuracies than ML/MF and there is an increase in pose-invariant information from ML/MF to AL to AM. The black bars under the plots indicate the time when the stimulus was present, and the orange bars under the plot indicate the length of the decoding time window.
- Figure 7.
Head pose decoding results. Results from decoding the pose of the head generalizing over facial identity. A, Decoding results for the three face patches plotted as a function of time. Head pose is better represented in ML/MF and AL than in AM. The black bars under the plots indicate the time when the stimulus was present, and the orange bars under the plot indicate the length of the decoding time window. B, Decoding results plotted as a function of the number of selective neurons used (i.e., results are plotted using the 2 most selective neurons for decoding, the 3 most selective neurons, etc.). The selectivity of neurons was based on ANOVA p values using data from the training set. The results show that AM has lower decoding performance than ML/MF and AL even when only a small number of neurons is used indicated that even the most pose-selective neurons in AM are less pose selective than the most pose-selective neurons in ML/MF and AL.
- Figure 8.
Classifier comparison to assess how much information is in the overall firing rate level and in patterns across neurons. Each subplot compares the results from four different classifiers (Maximum Correlation Coefficient classifier in green, Minimum Angle classifier in cyan, Poisson Naive Bayes classifier in purple, and Total Activity classifier in red). The columns contain data from the three different face patches. A, Results from decoding whether an image was a face or is in another object category on the FOB dataset (i.e., same decoding task as in Fig. 4). B, Within-category discrimination for monkey whole bodies on the FOB dataset (i.e., same decoding task as in Fig. 3). C, Pose-specific individual face discrimination on the FV dataset (i.e., same as the blue traces on Fig. 6). For face versus nonface results, all classifiers performed similarly. For all other analyses, there was much more information in classifiers that extracted patterns of information across neurons (PNB, Maximum Correlation Coefficient, Minimum Angle) than the classifier that used the total activity (TAct). The black bars under the plots indicate the time when the stimulus was present, and the orange bars under the plot indicate the length of the decoding time window.
Tables
- Table 1.
A table listing the classification decision functions for the four different classifiers used in this paper
A. Poisson Naive Bayes (PNB) 
B. Total Activity (TAct) C. Maximum Correlation Coefficient (MCC) D. Minimum Angle (Min Ang) wc is a vector that is the mean of the training data from class c, w̄c is a scalar that is the mean of wc, x is the test vector to be classified, x̄ is a scalar that is the mean of x, and n is the number of neurons; thus training the classifier consists of learning wc and w̄c and testing the classifier consists of determining which class x belongs to. As can be seen, all these classifiers are rather similar and mainly differ in how they normalize the data and, consequently, whether they take the overall level of population activity into account (A and B) or whether they only examine relative differences in the firing rate activity between neurons (C and D). TAct, Total Activity; MCC, Maximum Correlation Coefficient; Min Ang, Minimum Angle.
- Table 2.
Correlation between individual image firing rates and decoding accuracies for the three face patches show that higher population firing rates were correlated with higher decoding accuracies
ML/MF AL AM Human faces −0.34 0.29 0.50* Human bodies 0.10 0.62* 0.47 Fruits and vegetables 0.42 0.80* 0.23 Gadgets 0.79* 0.34 0.78* Human hands 0.60* 0.66* −0.32 Scrambled patterns −0.21 −0.01 0.11 Monkey body parts 0.55* 0.89* 0.34 Monkey whole bodies 0.33 0.47* 0.37 Each correlation was calculated separately for the 16 images in each category. Asterisks indicate statistically significant positive correlations (p < 0.05) using a permutation test. There were no statistically significant negative correlations.
