Heterogeneous Single-Unit Selectivity in an fMRI-Defined Body-Selective Patch

Category selectivity of the midSTS body patch

We measured single units and LFPs, simultaneously, for the 100 images of the main stimulus set that were presented randomly interleaved for 200 ms each during passive fixation (see Materials and Methods). These images were half of the stimuli used in the fMRI study of Popivanov et al. (2012). There were 10 images in each of the 10 stimulus classes: monkey faces, human faces, headless monkey bodies, headless human bodies, mammals, birds, body-like sculptures, fruits/vegetables, and two sets of control objects matched in low-level stimulus properties to the monkey bodies (objects M) and the human bodies (objects H), respectively (Fig. 1A, examples). Responsive neurons were searched when presenting the 100 images, centered at the fovea. In those cases in which responses were weak, we mapped the receptive field with the image that elicited the strongest response foveally, and then retested the neuron by presenting the 100 images at the center of the receptive field. Of the 134 responsive neurons (for definition, see Materials and Methods) recorded at the category-selective body patch of monkey E, 35 (26%) neurons were tested at peripheral locations (average eccentricity, 5.5°). In monkey B, only 10% (8/81) were tested at peripheral locations (average eccentricity, 3.7°). Below, we will report only data on responsive neurons for the optimal location or for the foveal location when no mapping was obtained, ensuring that each neuron is contributing only once to the sample. Results were similar when restricting the sample only to the foveal presentations. All responsive neurons showed an excitatory response for at least one image and often showed inhibitory responses to some images, which is typical for inferior temporal cortex.

The single-unit responses, averaged across the images of a class, differed significantly across classes (repeated-measures ANOVA on normalized responses per neuron; p < 0.0001 in each animal). For each monkey, the average response was greater for the four classes that contained bodies (monkey and human bodies, mammals, and birds) compared with the other classes, including the body like sculptures (Fig. 1D). Thus, we defined the body category as consisting of images of these four classes. The nonbody category included all the other classes: the monkey and human faces, objects M, objects H, fruits/vegetables, and sculptures. In some of the analyses, explicitly mentioned below, we excluded the sculptures from the nonbody category, because of their body-like appearance. In both monkeys, the average normalized response to the body category was significantly larger than the average response to the nonbody images (paired t test; p < 0.00001 in each animal). This preference for the body category was present early on in the response of monkey E but it was more pronounced in the later phase of the response in monkey B. Responses were stronger for bodies compared with either human or monkey faces. This difference was highly significant for each of the four body classes in monkey E (post hoc Bonferroni t tests; each body class, each face class; p < 0.00001); however, it reached significance only for the monkey bodies (post hoc Bonferroni t tests; p < 0.01) but not for the other body classes (all p > 0.48) in monkey B. Interestingly, the monkey bodies elicited a larger response than the human bodies in each animal, but this difference failed to reach significance. The monkey bodies produced a significantly larger response compared with the objects M class (post hoc Bonferroni t tests; p < 0.00001 in each animal), in agreement with the fMRI contrast that was used to define the recording location.

The single-unit data represent a relatively small sample of the population of neurons in the targeted body-patch regions. Therefore, we also simultaneously measured LFPs (using the same electrode) and computed the power as a function of peristimulus time and spectral frequency. It has been suggested that the power for frequencies >50 Hz can be used as a proxy for the spiking activity of the population of neurons close to the electrode (Ray and Maunsell, 2011). As shown in Figure 3, the LFP power for those frequencies was strongly selective for stimulus class in both monkeys, with greater power for the four body classes, which aligns perfectly with the single-unit data. We quantified the mean body category selectivity of the LFP signal by comparing the average normalized power for each of five spectral frequency bands (see Materials and Methods and Fig. 3 for definitions of frequency bands) for bodies and nonbodies (excluding sculptures). In each animal, the mean normalized power was significantly larger (paired t test) for bodies compared with nonbodies for each of the gamma bands (Table 1). The same trend was present for the beta band, but the body category selectivity became much weaker than for the gamma bands and reached significance in one animal only. The alpha bands showed a stronger mean response to nonbodies compared with bodies, but as for the beta bands, the difference between the two categories was relatively small (Table 1). For comparison, Table 1 also shows the mean normalized spiking activity of single units recorded at the same sites as the LFPs.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Body part stimulus set. A, The three exemplars of each of the seven body-part classes are shown in rows per class. B, Five in-plane orientations of one particular exemplar (step size 45°). The body parts measured 4° on a side and were shown on uniform gray background.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Category selectivity of the LFP power. Time-frequency plots representing the power change normalized to baseline for monkey E (A) and monkey B (B), averaged across sites (monkey E, N = 133; monkey B, N = 66) for each of the 10 stimulus classes. The boundaries of the frequency bands (alpha, α; beta, β; low gamma, γ_L; middle gamma, γ_M; high gamma, γ_H) are shown on the first plot in both panels. Stimulus onset and offset are marked by white vertical dotted lines.

Figure 4 directly compares the normalized single unit spiking activity with the LFP power in different spectral bands for each of the 10 stimulus classes for the population of the same recording sites. In both monkeys, the Pearson correlations between the mean spiking activity and mean power in the gamma bands were all >0.92 (p < 0.0002; N = 10 classes), except for a correlation of 0.82 (p < 0.005) between the low-gamma band power and spiking activity in monkey B. However, no significant correlations between spiking activity and power were present for the beta and alpha bands (Fig. 4, right), despite a significant, yet completely different, class-specific modulation of the LFP power in these lower frequency bands (ANOVA; p < 0.001 for each animal and band). This pattern of the correlations between spiking activity and the LFP power across frequency bands (Fig. 4) fits the presence of significant body category selectivity for both spiking activity and gamma power and the weaker or even reversed selectivity for the lower-frequency bands (Table 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Correlation between the category selectivity of spiking activity and LFP power in different frequency bands. Left, Normalized spiking activity (spikes) averaged per class of those units for which valid LFP measurements were obtained and normalized LFP power averaged for the same sites. The mean LFP power is averaged within each of five frequency bands. Colors indicate the normalized response strength (see bottom color bar). Each class is indicated by an example image. Right, Pearson correlation coefficient r between the mean spiking response and the mean power for each frequency band. The red line depicts the statistical threshold above which the correlations are significantly different from 0 (p < 0.05; N = 10 classes).

Category selectivity of single units

The above data indicate that the mean neuronal activity in the fMRI defined body patch is greater for bodies compared with other stimulus classes, including faces. However, based on these neuronal population analyses one cannot conclude that there is body category selectivity at the single unit level. In other words, does each of the neurons within the body patch prefer body images above images of other classes? Or is there a small pool of highly selective body cells embedded within a pool of noncategory selective cells? Alternatively, are there many weakly selective body cells that drive the population response?

To assess this, we computed for each single neuron a BSI that contrasts the mean net responses to body and nonbody images (Materials and Methods; Fig. 5A). A BSI larger than zero shows a preference for bodies with a BSI of 0.33 corresponding to a twofold greater net response to bodies compared nonbodies. In a first conservative analysis, we excluded the body-like sculptures from the nonbody category. The median BSI with only monkey and human faces, fruits/vegetables, objects M, and objects H as nonbody classes was 0.47 and 0.25 for monkeys E and B, respectively; values significantly >0 (Wilcoxon test; p < 0.00005 in each animal; median across animals, 0.38; mean, 0.33). However, as is clear from the distribution of the BSI (Fig. 5A; Table 2), both monkeys showed a considerable variation in the magnitude of the BSI. Previous studies on face selectivity, using the same sort of index computed on net responses, used a criterion of 0.33 to define a face category selective cell (Tsao et al., 2006; Issa and DiCarlo, 2012). Employing the same criterion, 61 and 48% of the neurons can be classified as body-selective in monkeys E and B, respectively (53% across both animals). When the BSI was recomputed with the sculptures included as nonbodies the median BSIs were similar (0.41 and 0.26 in monkeys E and B, respectively) with 56 and 47% of the neurons classified as body cells.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Body category selectivity in the midSTS body patch. A, Distribution of the BSI of body-patch single neurons. The values for the two animals are indicated by different gray levels (darker corresponds to monkey B). The triangles indicate the median BSI of each monkey. The dotted line marks a BSI of 0.33, corresponding to a twofold greater net response to bodies with respect to nonbodies. BSI was computed using net responses. B, Distribution of d′ (body) of the same neurons. The same conventions as in A, except that the dotted line represents a threshold of d′ = 0.5). The neurons with d′ values significantly different from 0 (p ≤ 0.025, Permutation test) are hatched.

The body patch was defined by comparing fMRI activations for the monkey bodies and objects M. Computing a BSI index with only the net responses to these two classes yielded median BSIs of 0.45 and 0.42 in monkeys E and B, respectively. Based on these BSIs, 55 and 54% of the neurons were “monkey body” selective in E and B, respectively. Thus, using the same contrasts for single units and fMRI, approximately half of the neurons found in the body patch could be classified as body cells using the conventional criterion and category index.

To compare with previous fMRI-guided studies on face selectivity (Tsao et al., 2006; Freiwald et al., 2009; Freiwald and Tsao, 2010; Issa and DiCarlo, 2012), we computed the BSIs on net responses. Because such BSIs can be affected (in both directions) by strong inhibitions to a few stimuli of a class, we recomputed BSIs using raw responses, i.e., including baseline and ignoring the distinction between inhibitory and excitatory responses. As shown in Table 2, median BSIs computed on raw responses were, as expected, smaller than those computed on net responses but were still significantly larger than zero in each animal (Wilcoxon test; p < 0.00005 in each animal; median across animals, 0.19) with 35% and 30% of the neurons having a BSI larger than 0.33 in monkeys E and B, respectively.

In addition to BSI we also computed another category selectivity index, d′ (body), which takes into account the mean responses to the contrasting categories as well as the variability of the responses to the individual images of a category (see Materials and Methods). The median d′ (body), computed using net responses, was 0.43 and 0.21 in monkeys E and B, respectively (Fig. 5C; median d′ across animals, 0.35; d′s computed on raw response produced similar results; Table 2). As expected, the distribution of the d′ (body) was significantly biased toward positive values (Wilcoxon test; p < 0.00005 in each animal). Assessing the statistical significance of the d′ (body) for each neuron by a permutation test showed that d′s larger than 0.5 (or smaller than −0.5) were statistically significant. Taking the criterion of 0.5 (which happens to be the same one used by Ohayon et al. (2012) who also used d′ in their face selectivity study) to define body category selectivity, between 35 and 46% of the neurons (depending on the animal and on whether one computes d′ on raw or net responses; Table 2) showed a significant body selectivity. Also, pooled across monkeys, 8% of the body-patch neurons showed a d′ (body) significantly smaller than −0.5 (Fig. 5B), indicating a significant category selectivity for nonbodies.

Thus, using several category selectivity metrics, we can conclude that although the midSTS body patch shows body category selectivity at the population level, the single neurons that comprise this population differ greatly in their degree of body category selectivity. Also, independent of the used category selectivity metric, body category selectivity and the percentage of body category selective neurons are lower than that reported for face category selectivity in the face patches (Tsao et al., 2006; Issa and DiCarlo, 2012; Ohayon et al., 2012).

Stimulus selectivity of single units

Figure 6 shows three single neuron examples, whereas the stimulus selectivity of all recorded body-patch neurons is shown in Figure 7A. Both figures illustrate the variation in category and stimulus selectivity that was manifest in our sample of body-patch neurons. Most neurons responded to many stimuli of different classes, including nonbodies (Figs. 6B, 7A). In fact, some neurons showed on average stronger responses to faces compared with nonfaces, when computing a conventional FSI (Tsao et al., 2006) or a d′ (face), which contrasts the mean responses to faces versus the other images (except for animals and birds because these images depicted heads as well). The FSI was >0.33 for 16 and 21% of the body-patch neurons recorded in monkeys E and B, respectively (a twofold greater average response to faces compared with the other stimuli) and 8 and 19% of the neurons in monkeys E and B, respectively, had a d′(face) larger than 0.5 (the same criterion as Ohayon et al., 2012). A cell with a FSI of 0.98 and a d′ (face) of 1.66 is illustrated in Figure 6C. These face category selective neurons were intermingled with body-selective neurons within single penetrations. To demonstrate this, we selected neurons that had a FSI >0.33, a d′ (face) > 0.5 and a twofold stronger response to faces compared with bodies. In the nine penetrations in which there was at least one recorded neuron below the selected face-selective neuron, the median FSI and d′ (face) for the face selective neurons was 0.72 and 1.18, respectively, and reversed to a median FSI of −0.44 and a d′ (face) of −0.33 for the neighboring neuron. The median BSI and d′ (body) of the face selective neurons was −0.50 and −0.57, respectively, which increased significantly for the neighboring neurons (median BSI, 0.73; median d′ (body), 0.31; Mann–Whitney U test; p < 0.05). The same reversals of the FSI, d′ (face), BSI, and d′ (body) were present for the 10 penetrations for which there was a neuron recorded above the face selective one (median FSI, 0.86 vs −0.06; median d′ (face), 1.20 vs −0.04; median BSI, −0.48 vs 0.23; median d′ (body), −0.57 vs 0.14; Mann–Whitney U test; p < 0.05). Importantly, this also held for the five penetrations in which a face selective neuron was recorded in between two recorded neurons (median test on BSI and d′ (body), p < 0.05), showing that face category selective neurons were mixed with neurons with other stimulus preferences in this body patch.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Selectivity pattern of three example body-patch neurons. Each bar corresponds to the net response to a stimulus. The stimuli are grouped per class as indicated by the bar colors above the example images. Error bars denote the SEM across trials. Images to which the neurons are responding (the three best and two others) are shown above their corresponding bars. The BSI and d′ (body) computed on net responses are indicated for each neuron. Insets show the PSTHs of the mean responses to bodies (green), faces (red), and inanimate objects (including sculptures, dark purple) for each neuron (color code shown below example images). A, A neuron showing body category selectivity and strong within body class selectivity. Its BSI and d′ (body) computed on raw responses was 0.63 and 0.70, respectively. B, A neuron showing weak body category selectivity. Its BSI and d′ (body) computed on raw responses was 0.20 and 0.74, respectively. C, A face category selective neuron. Its BSI and d′ (body) computed on raw responses was −0.79 and −0.89, respectively. The neuron preferred profiles of human faces.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Stimulus selectivity of midSTS body-patch neurons. A, Spiking activity matrices where each row represents the normalized responses of a neuron to each of the 100 stimuli. Each column corresponds to a stimulus. The stimuli are grouped per class as indicated by the example images. Normalized response strength is indicated by a color code (see colored bar). Cells are ordered by their BSI, i.e., the cells on top of the matrix are the most body-selective. The mean normalized responses, averaged across neurons, for each stimulus are shown below the matrices (“average”). The horizontal arrows denote the example neurons and the letters corresponding to the panels of Figure 6. B, Within-category selectivity. Mean normalized responses to the 10 stimuli within a class, ranked by response strength from best (rank 1) to worst (rank 10). For each neuron, half of the trials were used for ranking the stimuli and the other half of the trials were used to average the responses. Normalization was performed with respect to the maximum responses across all 100 stimuli tested. Only neurons for which there was a significant response to at least one of the stimuli in a given class were included for that class, explaining the different number of neurons (N) for the classes. The ranking was performed for the six classes indicated by color coded legend.

Figures 6 and 7A illustrate that the neurons in the body patch responded to only some exemplars of a class. For example, the neuron shown in Figure 6A responded to a minority of the body images. This explains its relatively low d′, despite the BSI of 1 (this neuron also showed excitatory responses to a couple of nonbodies, but these were compensated by the negative net responses for many nonbody images; its BSI computed on raw responses was 0.63). The marked within-class selectivity was examined for our population of neurons by ranking the images of a class according to the elicited net response of each neuron that responded significantly to at least one of the images of that class. The statistical significance was assessed by a split-plot ANOVA (stimulus as between-trial factor and baseline versus stimulus period as repeated, within trial factor) for each of the 10 classes and an excitatory net response to at least one image of the class was required. The image ranking was performed with the mean responses obtained in 50% of the trials and the responses of the other 50% of the trials were then plotted as a function of the image rank. This avoided an erroneous induction of stimulus selectivity by the ranking procedure. This ranking analysis yielded evidence of strong within-class selectivity for all classes in both monkeys (illustrated for six classes in Fig. 7B). In fact, the net normalized response to the “worst” image of each class (Fig. 7B, rank 10) was not significantly larger than zero for each of the 10 stimulus classes in each animal (Bonferroni corrected Wilcoxon signed rank test; p > 0.05). Even for this relatively small number of images (10) per class, the single-unit responses varied within a large range, being absent for some images of the class. A highly similar within-class selectivity was also observed when ranking the stimuli only for those neurons and classes that demonstrated class-selectivity (a twofold stronger response to the class compared with controls) or only for the class that included the preferred stimulus (among the 100 stimuli tested) of a neuron. The strong within-class selectivity was not due to differences in stimulus area, contrast, or aspect ratio. Indeed, the mean normalized responses did not depend on the differences in these stimulus parameters between the preferred image of a neuron and the other images of a class (data not shown).

Because the preferred stimulus of the body classes varied among the single neurons, the response averaged across neurons appears similar across the different bodies (Fig. 7A, Average). The preference for bodies over other image classes that emerged at the population level (Figs. 1D, 4) resulted mainly from the pooling of single neurons with different stimulus preferences and strong within-body category selectivity but that are biased to respond stronger to body compared with nonbody images. This pooling averaged out the different stimulus selectivities within the body category (Fig. 7A). In fact, despite the high within-class selectivity, one can classify with a high accuracy (97 and 91% correct in monkeys E and B, respectively) whether an image comes from a body or an nonbody class by using the mean responses of the population of body-patch neurons to a stimulus (Fig. 7A, Average). This was assessed by computing the area under the receiver operating characteristic curve when comparing the distribution of the mean responses (averaged across neurons per monkey) to the individual body images (N = 40) and the distribution of the mean responses to the individual nonbody images (N = 60). Thus, although the single body-patch neurons were heterogeneous in their selectivity (Fig. 7A), the overall bias to respond stronger to bodies compared with nonbodies accounts for the body category selectivity at the population level.

Representation of stimuli in midSTS body patch

Thus far, we have showed that the overall response of the midSTS body-patch neurons to bodies was larger than to nonbodies and that individual neurons show a strong selectivity for body (and other) exemplars. This raises the question of how the population of midSTS neurons represents the individual images of the different classes. To assess this, we computed the neural response-based dissimilarities between all possible image pairs. As a metric of neural-based stimulus dissimilarity we used the Euclidean distance between the images in a multidimensional space where the responses of the single neurons defined the dimensions (Op de Beeck et al., 2001;De Baene et al., 2007;Kayaert et al., 2005; see Materials and Methods). Figure 8B shows the Euclidean distance for all possible stimulus pairs for the neurons of both monkeys combined. It is obvious that the dissimilarities are large for pairs of body images (mean Eucledian distance, 5.32; SEM = 0.02). In particular, this was the case for monkey bodies (mean distance for pairs of monkey bodies, 5.48; SEM = 0.09), mammals (5.32; SEM = 0.06) and birds (5.07; SEM = 0.09), but pairs of human bodies showed lower dissimilarities (mean distance, 4.54; SEM = 0.06). This may reflect the fact that all human body images, except one, depicted an upright standing person, and thus showed less variation in posture than the other body classes. The mean dissimilarities were the smallest for pairs of faces (mean dissimilarity, 4.13; SEM = 0.03; human and monkey faces combined) followed by pairs of inanimate objects (4.42; SEM = 0.02). The dissimilarities for face versus bodies (5.36; SEM = 0.01) or inanimate objects versus bodies (5.19; SEM = 0.01) were large but comparable to those for the body pairs.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Representation of animate and inanimate object exemplars in the midSTS body patch. A, Dissimilarity matrix of pairwise Euclidean distances for all stimulus pairs based on the gray level difference in the corresponding pixels. The stimuli are grouped per class as indicated by the example images. The dissimilarity is indicated by the color scale with red indicating high dissimilarities. Note that the matrix is symmetric with respect to the zero distance diagonal. B, Dissimilarity matrix of pairwise Euclidean distances for all stimulus pairs based on the spiking responses of 215 body-patch neurons. The same conventions as in A. C, Hierarchical clustering of the stimuli based on the spiking responses of both monkeys (N = 215 neurons). Vertical stippled lines indicate the boundaries of the major clusters. The stimulus classes are color-coded according to the legend (bottom).

To determine whether these neural dissimilarities merely reflect physical image dissimilarities, we computed the Euclidean distances between the images in the multidimensional space defined by the pixel gray levels, i.e., the input to the visual system (Fig. 8A). Comparing the two dissimilarity matrices (Fig. 8A,B), it is clear that the pixel-based dissimilarities are quite different from the neural dissimilarities. Indeed, the Spearman rank correlation between the two matrices was very small, r_S = 0.03 (n.s., permutation test), indicating the neural dissimilarities do not simply reflect physical image similarities.

We examined the stimulus representation of the body-patch neurons further by performing a hierarchical cluster analysis of the dissimilarity matrix of Figure 8B. The advantage of this technique, compared with direct testing of the similarities in response patterns between bodies and the other classes, is that it provides an unbiased description of the similarities among the responses to the stimuli, irrespectively of their class. The cluster analysis of the spiking data when both animals were combined showed two main clusters (Fig. 8C). One cluster contained 44 images of which 39 (97.5%) were bodies (10 monkey bodies, 9 human bodies, 10 mammals, and 10 birds). The percentage of bodies in this cluster (97.5%) was significantly higher than the 40% expected if bodies were randomly distributed between the two clusters (binomial test; p < 0.01). This “body” cluster contained a subcluster consisting of the nine human bodies, two elongated objects (objects H), and one vegetable with the same aspect ratio (a vertically oriented corn). The body cluster also contained one human face, which is shown in Figure 1A. Interestingly, unlike the other human face stimuli, this person had long hair, which might appear as two limbs below the neck. The other, nonbody, cluster diverged into two distinct clusters with one consisting entirely of faces. Note that human and monkey faces, despite the morphological differences between species, were dispersed within this face cluster. A similar clustering of bodies versus other images was also present in the individual data of each animal, but noisier than when pooling the data across both animals (percentage of bodies in body cluster, monkey E: 71% with 34/40 bodies in body cluster; monkey B: 65% with 40/40 bodies in body cluster).

In summary, the pairs of body images showed high within-class dissimilarities based on spiking responses of single units (Fig. 8B), which is consistent with the high within-class selectivity of the single neurons (Fig. 7B). Despite this high selectivity among the different body exemplars, the population of body-patch neurons clustered bodies versus nonbodies. This resulted from a combination of the greater responses for bodies compared with nonbodies (Fig. 1D), the relatively low dissimilarities for the nonbody image pairs, which evoked a weaker response on average, and the relatively high dissimilarities for the body–nonbody image pairs.

Because LFPs sum the activity of a population of neurons and assuming that neighboring neurons can have different preferences within the body class but still tend to prefer bodies over other stimulus classes, one would expect that the mean dissimilarities for pairs of body images would be smaller than the dissimilarities for body–nonbody pairs for the LFP power. This was indeed the case for the high and middle gamma power (Fig. 9): the mean dissimilarity for the body pairs was 4.53 (SEM = 0.01) and 4.52 (SEM = 0.01) for the high and middle gamma power, respectively, which was lower than the mean dissimilarities for the face-body (4.86; SEM = 0.02 and 4.97; SEM = 0.02) and for body-inanimate object pairs (4.68; SEM = 0.01 and 4.70; SEM = 0.01). As expected, cluster analysis showed for both these gamma bands a cluster predominantly containing bodies (39/47, 97.5%; p < 0.01 and 37/42, 92.5%; p < 0.01 for the high and middle gamma, respectively). For the low gamma power, the distinction between bodies and nonbodies was less (mean dissimilarity for body, body-face, and body-inanimate object pairs was 5.36, 5.47, and 5.36, respectively; Fig. 9) and the cluster analysis showed a cluster containing bodies (22/22), but only 55% of the bodies were represented in that cluster. The clustering of bodies versus nonbodies was weak (29/46, 72.5%; p < 0.01) and absent (21/44, 52.5%; n.s.) for the alpha and beta bands, respectively.

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

Representation of animate and inanimate object exemplars based on the LFP power in different frequency bands. Left, The dissimilarity matrices of pairwise Euclidean distances for all stimuli pairs, based on the LFP power in each frequency band (same conventions as in Fig. 8B). Right, The hierarchical clustering of all stimuli (same conventions as in Fig. 8C). The sites from both monkeys are pooled together resulting in N = 199 sites for the gamma bands and N = 196 sites for the beta and alpha bands. Note that three sites from monkey B had to be excluded from the analysis of the alpha and beta LFP power due to a close to zero maximal percentage power relative to baseline, which after normalization resulted in very large values, distorting the mean distances.

Classification of bodies versus nonbodies

Because the cluster analysis of the spiking activity showed distinct clusters of the bodies versus other stimulus classes, it was expected that one could determine whether a stimulus is a body from the population response of the midSTS body-patch neurons. This prediction was tested by having a classifier decide whether an image contains a body, or not, given only the population response vector for that stimulus. This population response vector consisted of a concatenation of the mean responses (averaged across trials) of the neurons to that stimulus. For both monkeys, separately, we trained SVMs on 70% of the stimuli of each category and tested classification performance on the remaining 30%. Hence, we tested explicitly for generalization, a hallmark of categorization. In both monkeys, the proportion of correct classifications for bodies versus nonbodies was high, and well above chance level (50%; confirmed by permuting the stimulus labels): 90 and 89% correct in monkeys E and B, respectively. Interestingly, when having the classifier deciding whether a face or nonface (excluding mammals and birds, which had heads) was present, the classification scores were also high: 92 and 97% correct in E and B, respectively. Thus, the population of body-patch neurons can be used to classify bodies versus nonbodies and, also, faces versus nonfaces. This led to the question of whether other stimulus classes can also be classified with the body-patch population responses. To answer this question we trained SVMs to classify an image as belonging to one of the 10 stimulus classes. The confusion matrices (Fig. 10) show that correct classification scores for all of the 10 classes were well above chance (Fig. 10, diagonal; chance level = 10% correct). Interestingly, objects M and objects H are both object sets, mainly differing in aspect ratio, and the body-patch neurons could classify these rather well with little confusion between these two classes (Fig. 10). The different classes of the body category (monkey bodies, human bodies, mammals, and birds) could also be distinguished reliably, except for the confusion of animals and birds in monkey E (Fig. 10).

• Download figure
• Open in new tab
• Download powerpoint

Figure 10.

Confusion matrices displaying the performance of the linear SVM classifier for the two subjects (top and bottom). In each panel, the rows indicate the presented stimulus of a particular class (input class) and the columns the classifications made by the classifier (output class). The classification scores are color coded according to the color scale shown between the panels. Numbers indicate correct classification scores as a percentage. Perfect classification corresponds to values of 100% along the diagonal. The proportions of a row add up to 100%. Each class is indicated by an example image.

Location specificity of stimulus representation in STS

The body-selective location was bordered laterally and medially by a region in which the activation for monkey bodies was still stronger compared with the objects M class, but that also showed stronger activation for monkey faces compared with objects M (Fig. 11A). It was interesting to assess whether the single unit and LFP selectivity would change away from the body patch. Thus, in monkey E, we recorded also single units and LFPs 1 and 3 mm lateral to the primary target location (Fig. 11A, position 1). The class selectivity of the mean single unit and high gamma LFP power changed when moving more lateral: mean responses to the headless monkey and human bodies became weaker and the responses to faces increased (Fig. 11B). This difference among recording locations in class selectivity was highly significant (ANOVA; interaction stimulus class and recording position, p < 0.0005 for spikes and high gamma power). A cluster analysis of the most lateral position (Fig. 11A, position 3) drew a distinction between a face cluster, consisting of 17 of the 20 faces, and all other images, including bodies (Fig. 11C). This contrasts with position 1, which showed a body cluster that included 34/40 bodies (percentage of body images in cluster, 71%) and was separated from a cluster of faces and the other objects (Fig. 11C). Thus, moving laterally away from the body patch, there is a gradual transition from a representation of mainly bodies to one of faces.

• Download figure
• Open in new tab
• Download powerpoint

Figure 11.

Comparison of the category selectivity between different medial-lateral locations within the midSTS of monkey E. A, Coronal slice of monkey E's brain, showing the activation for bodies (blue, contrast; monkey bodies, objects M) and faces (red, contrast; monkey faces, objects M). The overlapping activations are in purple. Only activations that passed the FWE corrected level of p < 0.05 are shown. The white arrows show recording locations (1, center of midSTS body patch; 2, 1 mm more lateral than 1; 3, 3 mm more lateral than 1). B, Mean normalized response per class for the spiking activity and high gamma (γ_H) band for each recording location as indicated in A. Only neurons for which valid LFPs were obtained were included. C, Hierarchical clustering of the stimuli based on the spiking responses of monkey E for the population of neurons recorded at locations 1 and 3 (A). The stimulus classes are color-coded according to the legend shown at the bottom. N indicates number of recorded neurons within each of the populations.

Selectivity for body parts in the midSTS body patch

One could argue that the relatively low BSI and d′ (body) in the midSTS body patch and the strong within-body class selectivity results from a tuning to individual body parts rather than to the whole body. Indeed, it is possible that different body-patch neurons are selective for different body parts, i.e., some neurons preferring a hand, other neurons a leg and still others a torso, etc. Because some body parts were partially occluded in some of the body images of our main stimulus set this could have contributed to the strong within-body category selectivity that we observed with this stimulus set. To examine this question, we measured the responses of neurons in this body patch to segmented body parts in a control experiment. The body part stimulus set (Fig. 2) consisted of seven classes of male monkey body parts from which three exemplars were presented at five orientations each. The midSTS body-patch neurons responded well to these body part images (mean net response to preferred image (of the 105 body part images) was 55 spikes/s (SEM = 6; N = 52 neurons)), indicating that isolated body parts are sufficient to elicit sizeable responses from the midSTS body patch. Because only a small number of these neurons were also tested with the whole-body images and these only with a small number of trials, a proper within-neuron comparison between the strengths of the responses for whole body and body parts stimuli could not be performed. However, one can compare the strength of the response to the body parts in this sample of body-patch neurons to the strength of the response to whole-body images for the body-patch neurons that were recorded with the main stimulus set (same neuronal sample as in Fig. 7A). This showed that the mean net response of the latter neuronal sample to the preferred whole-body image (44 spikes/s; SEM = 3; N = 215 cells) was comparable to that obtained for the body parts (n.s., Mann–Whitney U test). All 52 responsive neurons (assessed with a split-plot ANOVA; see Materials and Methods) showed highly selective responses to the body part stimulus set (Fig. 12A), with a profound selectivity for body part orientation. We quantified the orientation selectivity of each neuron for the body part exemplar producing the greatest response by computing a best-worst index: where R_best and R_worst are the net responses to the best and worst orientations for a particular exemplar, respectively. Note that an index of 1 means no response to the worst orientation. The median best-worst index for the body part eliciting the best response was 0.97 (25^th percentile = 0.86; 75^th percentile = 1.08; N = 52), demonstrating the strong dependence of the response on the orientation of a body part.

• Download figure
• Open in new tab
• Download powerpoint

Figure 12.

Selectivity for body parts. A, Spiking activity matrix where each row represents the normalized responses of a neuron to each of the 105 body part stimuli. The horizontal line at cell number 26 separates the neurons of the two animals (below horizontal line, monkey B). Each column corresponds to a stimulus. The stimuli were first grouped per body part class, as indicated by the example images, then by body part exemplar (three exemplars per class) and subsequently by orientation. Short and long tick marks at the top and bottom of the matrix indicate the divisions between the three exemplars of a class and between the different classes, respectively. Normalized response strength is indicated by a color code (see colored bar). The mean normalized responses, averaged across neurons, for each stimulus are shown in the bar below the matrices (average). B, Top, The dissimilarity matrix of pairwise Euclidean distances for all stimulus pairs, based on the spiking responses of 52 body-patch neurons. Bottom, The dissimilarity matrix based on the gray level difference in the corresponding pixels. The same conventions are used as in Figure 8A. The two images that demonstrated a relatively high neural dissimilarity are shown above the top panel and the arrows indicate their mean response (A) and dissimilarity values (B).

Figure 12B (top) also shows the Euclidean distances between all the body part stimulus pairs, based on the responses of the 52 body-patch neurons. First, note that there is no evidence of any clustering of the stimuli according to body part class, e.g., a clustering of all the images depicting a hand (also supported by hierarchical cluster analysis). Thus, these midSTS body-patch neurons do not appear to signal body part class per se, but instead show a pronounced selectivity for body part exemplars, viewed at specific orientations. Second, inspection of the dissimilarity matrix reveals that two images show a marked increase in pairwise dissimilarity relative to many other images (Fig. 12B, top, arrows): a penis and a leg in a grasping pose. This greater dissimilarity was not due to a greater overall response to these images (Fig. 12A, arrows): the mean net response, averaged across neurons, was 15 and 17 spikes/s for the penis and leg image, respectively, which compares well to the mean net response for all other images (15 spikes/s; SD = 3). Thus, the higher dissimilarity for these images reflects greater differences in response between these images and the other images within the neurons. The higher average dissimilarity was strongly orientation selective, being present only for one of the five orientations of these body part images (Fig. 12B, top). Interestingly, the most marked dissimilarity was demonstrated by an upright, vertically oriented, erect penis that has obvious ethological significance. Note that this increased dissimilarity was not present for the other orientations [compare the dissimilarities for the vertical penis (Fig. 12B, left arrow) with the dissimilarities depicted for the next vertical line in the dissimilarity matrix of Fig. 12B (top), which indicates the data for the same image but rotated by 45°]. Finally, note that the pixel-based dissimilarities for the body part images (Fig. 12B, bottom) reveal a different pattern compared with the neural-based ones (Fig. 12B, top). Although the Spearman rank correlation between the two dissimilarity matrices was significantly different from 0 (p < 0.0001, Permutation test), it was small: r_s = 0.17.