Abstract
Interaction with everyday objects requires the representation of conceptual object properties, such as where and how an object is used. What are the neural mechanisms that support this knowledge? While research on semantic dementia has provided evidence for a critical role of the anterior temporal lobes (ATLs) in object knowledge, fMRI studies using univariate analysis have primarily implicated regions outside the ATL. In the present human fMRI study we used multivoxel pattern analysis to test whether activity patterns in ATLs carry information about conceptual object properties. Participants viewed objects that differed on two dimensions: where the object is typically found (in the kitchen or the garage) and how the object is commonly used (with a rotate or a squeeze movement). Anatomical region-of-interest analyses covering the ventral visual stream revealed that information about the location and action dimensions increased from posterior to anterior ventral temporal cortex, peaking in the temporal pole. Whole-brain multivoxel searchlight analysis confirmed these results, revealing highly significant and regionally specific information about the location and action dimensions in the anterior temporal lobes bilaterally. In contrast to conceptual object properties, perceptual and low-level visual properties of the objects were reflected in activity patterns in posterior lateral occipitotemporal cortex and occipital cortex, respectively. These results provide fMRI evidence that object representations in the anterior temporal lobes are abstracted away from perceptual properties, categorizing objects in semantically meaningful groups to support conceptual object knowledge.
Introduction
To recognize objects, the visual system transforms a 2D retinal image into a representation of object shape that is invariant to changes in viewpoint, viewing distance, occlusion, and illumination (Marr, 1982). The representation of objects must then undergo a further level of abstraction to support object recognition at the conceptual level. The present study sought to reveal conceptual-level object representations that are abstracted away from perceptual-level object representations.
Important insights about conceptual object representations have come from neuropsychological research, including studies using lesion overlap analysis (Damasio et al., 1996) and research on patients suffering from semantic dementia, a neurodegenerative disease affecting the anterior temporal lobes (ATLs) bilaterally (Patterson et al., 2007). Semantic dementia patients lose conceptual knowledge about everyday objects while basic shape perception and other cognitive abilities are relatively intact (Warrington, 1975; Snowden et al., 1989; Hodges et al., 1992), thus suggesting a critical role for the ATL in object knowledge. Neuroimaging studies have been less conclusive about the role of the ATL in object knowledge (Martin, 2007), which has been attributed to susceptibility artifacts that reduce the likelihood of finding fMRI activity in this region (Devlin et al., 2000; Visser et al., 2010b), inappropriate baseline tasks (Visser et al., 2010b), and limited coverage of ATL in many neuroimaging studies (Visser et al., 2010b). Instead, fMRI studies have provided evidence that retrieval of object knowledge, in particular knowledge related to sensory or motor properties of objects, activates brain regions that are also active during actual perception or action of these attributes (Martin, 2007; Barsalou, 2008; Kiefer and Pulvermuller, 2012). Furthermore, a recent meta-analysis of 120 functional neuroimaging studies on semantic knowledge revealed activation in an extensive network of regions primarily outside the ATL, including the inferior parietal cortex, prefrontal cortex, posterior cingulate gyrus, fusiform and parahippocampal gyri, and middle temporal gyrus (Binder et al., 2009).
In the present study, rather than testing for brain regions in which overall fMRI activity levels increase during semantic processing, we tested for regions in which multivoxel activity patterns distinguish between visually presented objects that differ on conceptual dimensions. Multivoxel pattern analysis (MVPA; Haxby et al., 2001) has been successful in localizing representations that are abstracted away from the perceptual input (Peelen et al., 2010). In the visual domain, MVPA studies have revealed line orientation tuning (Haynes and Rees, 2005; Kamitani and Tong, 2005), size-invariant and viewpoint-invariant object representations (Eger et al., 2008), and representations of perceived object shape (Haushofer et al., 2008; Op de Beeck et al., 2008; Drucker and Aguirre, 2009), as well as individual face representations in anterior regions of the temporal lobes (Kriegeskorte et al., 2007; Baron and Osherson, 2011; Nestor et al., 2011). In the present study, we provide evidence that multivoxel activity patterns in anterior temporal cortex carry information about conceptual properties of everyday objects.
Materials and Methods
Participants.
Twenty-six healthy adult volunteers (11 females; mean age, 27.5 years; age range, 20–38 years) participated in the fMRI experiment. Fifteen healthy adult volunteers (11 females; mean age, 23.6 years; age range, 20–35 years), one of whom also participated in the fMRI experiment, rated the perceptual similarity of the objects shown in the fMRI experiment. All participants were right-handed with normal or corrected-to-normal vision and no history of neurological or psychiatric disease. Participants gave written informed consent for participation in the study, which was approved by the human research ethics committee of the University of Trento, Rovereto, Italy.
Stimuli.
The stimulus set consisted of five different exemplars of 12 objects. Half of the objects are typically found in a kitchen, and the other half in a garage (or workplace). Half of the objects are manipulated by a wrist rotation movement, and the other half by a hand-squeeze movement. Participants were presented with a printout showing the objects before the experiment and told about the location and action dimensions on which the objects differed. All participants recognized and correctly identified the objects before scanning.
Stimuli (400 × 400 pixels, 5°) were presented centrally. Stimulus presentation was controlled by a PC running the Psychophysics Toolbox package (Brainard, 1997) in Matlab (MathWorks). Pictures were projected onto a screen and viewed through a mirror mounted on the head coil.
Pixelwise similarity.
The pixelwise similarity between the 12 objects was computed using pixelwise correlations (Thierry et al., 2007) on black-and-white silhouettes of the objects. Pixelwise correlations were computed between randomly paired exemplars of the 12 objects, using each exemplar once for every cross-object comparison. Correlations were Fisher transformed, and the correlation matrices of the five exemplars were averaged, resulting in a symmetric 12 × 12 matrix of pixelwise similarity.
Perceptual similarity rating.
On each trial, two objects were shown simultaneously and participants answered the question “how much do these two objects look alike?” on a 1 (“not at all”) to 5 (“totally”) scale. Participants were instructed to judge the similarity of the shape of the objects disregarding low-level similarities, such as differences in size, illumination, or viewpoint. The next trial started after the response was collected. Each participant rated all pairs of the 12 objects once (66 trials), with one of the five object exemplars selected randomly on each trial. This resulted, for each participant, in a symmetric 12 × 12 matrix of perceptual similarity. Ratings were averaged across participants.
Independence of object similarity measures.
To test whether conceptual similarity was related to perceptual similarity, we followed the same analysis approach as that used for the analysis of neural similarity (Fig. 1). For each of the 15 participants in the rating experiment, we calculated the average perceptual similarity between objects associated with the same location (e.g., kitchen rotate, kitchen squeeze; Fig. 1), objects associated with the same action (e.g., kitchen rotate, garage rotate; Fig. 1), and objects that differed in both location and action (Fig. 1). We then tested whether perceptual similarity ratings between objects sharing one of the two conceptual dimensions were significantly different from ratings between objects that differed on these conceptual dimensions. The mean perceptual similarity rating for objects with the same location (1.7) did not significantly differ from the mean rating between objects that differed in location (1.7; t(14) = −0.1; p = 0.94). Similarly, the mean rating for objects with the same action (1.8) did not significantly differ from the mean rating between objects that differed in action (1.7; t(14) = 2.0; p = 0.06). Thus, conceptual similarity and perceptual similarity were largely independent of each other.
The same approach was used to test whether conceptual similarity was related to pixelwise similarity. The average (Fisher-transformed) pixelwise correlation between objects associated with the same location (e.g., kitchen rotate, kitchen squeeze; Fig. 1) was 0.12. The average pixelwise correlation between objects associated with the same action (e.g., kitchen rotate, garage rotate; Fig. 1) was also 0.12. Finally, the average pixelwise correlation between objects that differed in both location and action (Fig. 1) was 0.14. Thus, if anything, the objects that shared a conceptual dimension were, at the pixel level, less similar to each other than objects that did not share a conceptual dimension.
Finally, we correlated the matrix of perceptual similarity with the matrix of pixelwise similarity to test whether these variables were related to each other. The correlation was −0.15, indicating that, if anything, objects that were relatively similar perceptually were relatively dissimilar at the pixel level.
Together, these analyses confirm that the three measures of similarity (pixelwise, perceptual, conceptual) of the same set of object stimuli were orthogonal to each other, allowing for the investigation of the neural representation of these object properties independently of one another.
Task and design of fMRI experiment.
Participants performed a one-back task, detecting repetitions of either the location (kitchen, garage) or the action (rotate, squeeze) dimension of the objects in different runs. Participants pressed a response button with their right index finger when a task-relevant repetition occurred. The task was held constant throughout a run. The order of the tasks was counterbalanced across subjects.
Two presentation versions were used. In the first version (event-related; N = 14), objects were presented for 1200 ms, followed by a 600 ms fixation period. Participants performed 6 runs of 120 trials each. Each of the 12 objects was presented 10 times within a run, in random order. One of the five exemplars of the object was randomly selected on each trial. In the second version (blocked; N = 12), objects were presented for 700 ms, followed by a 425 m fixation period. Participants performed four runs of 256 trials each. Trials were presented in blocks of 16 trials, with 13 of16 trials coming from the same category (e.g., kitchen rotate), and the remaining three trials each coming from one of the other three categories. Objects within each category were randomly selected on each trial. Each block was followed by a 4000 ms fixation period. Trial order within blocks was random. Block order was counterbalanced across runs. The analysis for both presentation versions was identical, and data were analyzed together unless otherwise indicated.
fMRI data acquisition.
Functional and structural data were collected at the Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy. All images were acquired on a Bruker BioSpin MedSpec 4-T scanner. Functional images were acquired using echo planar imaging T2*-weighted scans. Acquisition parameters were a repetition time of 2 s, an echo time of 33 ms, a flip angle of 73°, a field of view of 192 mm, and a matrix size of 64 × 64. Each functional acquisition consisted of 34 axial slices (which covered the whole cerebral cortex) with a thickness of 3 mm and gap of 33% (1 mm). Structural images were acquired with an MP-RAGE sequence with 1 × 1 × 1 mm resolution.
fMRI data preprocessing.
Data were analyzed using the AFNI software package (http://afni.nimh.nih.gov/) and MATLAB (The MathWorks. Functional data were slice time and motion corrected, and low-frequency drifts were removed with a temporal high-pass filter (cutoff of 0.006 Hz). No spatial smoothing was applied.
Regions of interest definition.
Anatomical regions of interest (ROIs), Brodmann areas (BAs), were defined using the Talairach atlas implemented in AFNI (TT_Daemon) and resampled to 3× 3 × 3 mm voxels. ROIs of the two hemispheres were combined. ROI sizes (number of resampled voxels) were as follows: BA17, 228; BA18, 1063; BA19, 988; BA37, 392; BA20, 458; BA38, 451. BA17 and BA18, located in the occipital lobe, have been shown to correspond to retinotopic areas V1 and V2/V3, respectively (Wohlschläger et al., 2005). BA19 is located anterior to BA18 in the occipital lobe and includes the lateral occipital gyrus and the superior occipital gyrus. BA37 corresponds to the occipitotemporal cortex and includes the posterior fusiform gyrus and the posterior inferior temporal gyrus. Several overlapping functionally defined areas in lateral occipitotemporal cortex fall approximately at the border of BA19 and BA37, including object-selective LO (lateral occipital complex), motion-selective hMT+ (human homolog of macaque area), and body-selective EBA (extrastriate body area) (Downing et al., 2007). Another group of functionally defined areas is primarily located in BA37, including the object-selective pFs (posterior fusiform gyrus) (Lerner et al., 2001) and the face-selective/body-selective FFA/FBA (fusiform face area/fusiform body area) (Peelen and Downing, 2005). BA20, located anterior to BA37, covers part of the ventral temporal cortex and corresponds approximately to the inferior temporal gyrus. BA38 is the most anterior part of the temporal lobe, often referred to as the temporal pole.
The temporal signal-to-noise ratio (tSNR) for each participant was calculated for the first run of the experiment by dividing the mean signal in each voxel by the SD of the residual error time series in that voxel (Friedman and Glover, 2006). tSNR values were then averaged across the voxels of each anatomical ROI. Mean tSNR values, averaged across participants, were as follows: BA17, 129.2; BA18, 105.9; BA19, 90.7; BA37, 97.9; BA20, 67.2; BA38, 68.8. The percentage of voxels in each ROI that had “good” tSNR values (>20; Binder et al., 2011) was above 90% for all ROIs: BA17, 93.7%; BA18, 91.2%; BA19, 91.9%; BA37, 94.9%; BA20, 90.6%; BA38, 92.7%. These values indicate that, although mean tSNR was lower in anterior temporal cortex than in posterior temporal cortex, tSNR was sufficient to detect reliable fMRI activation in all ROIs (Binder et al., 2011).
Statistical analysis.
For each participant, general linear models were created to model the conditions in the experiment. All trials were included in the analysis. Regressors of no interest were also included to account for differences in the mean MR signal across scans and for head motion within scans. For the ROI analyses, statistical maps were transformed into Talairach space and resampled to 3 × 3 × 3 mm.
To create a neural similarity matrix for a given ROI, response patterns (beta weights) for the 12 objects were correlated with each other, resulting in a symmetric 12 × 12 matrix. These correlation values were Fisher transformed. Pixelwise information carried by response patterns in a given ROI was computed by correlating the pixelwise similarity matrix with the neural similarity matrix of that ROI. Similarly, perceptual information carried by response patterns in a given ROI was computed by correlating the perceptual similarity matrix with the neural similarity matrix of that ROI. Only the 66 unique off-diagonal values of the matrices were entered into these analyses. Correlations between matrices were computed for each participant individually, Fisher transformed, and then tested against zero using one-sample t tests (two-tailed) with participants (N = 26) as random factor. Very low p values were rounded to p < 0.0001.
Conceptual information was computed as illustrated in Figure 1. For this analysis, beta weights were computed for each of the four object groups (kitchen rotate, garage rotate, kitchen squeeze, garage squeeze), averaging activity across objects within the groups. For each ROI, response patterns to these four conditions were then correlated with each other, and correlations were Fisher transformed. Location information was computed by subtracting the average correlation between objects that differed in both location and action (e.g., correlation between kitchen rotate and garage squeeze objects) from the average correlation between objects with the same location but different action (e.g., kitchen rotate-kitchen squeeze correlation). Similarly, action information was computed by subtracting the average correlation between objects that differed in both location and action (e.g., kitchen rotate–garage squeeze correlation) from the average correlation between objects with the same action but different location (e.g., kitchen rotate–garage rotate correlation). Information values were then tested against zero using one-sample t tests (two-tailed) with participants (N = 26) as random factor. Very low p values were rounded to p < 0.0001.
Searchlight analysis.
A whole-brain pattern analysis was performed using a spherical searchlight approach (Kriegeskorte et al., 2006). For each voxel in the brain, we computed voxelwise correlations in a sphere of 8 mm radius (corresponding to 65 voxels) around this voxel. These correlations were Fisher transformed. Information values reflecting pixelwise, perceptual, and conceptual information (averaged across the two dimensions) were then computed from these correlations as described in the Statistical analysis section. Information values for each sphere were assigned to the center voxel of that sphere. The analyses were performed for each participant separately in native space. Results were transformed into Talairach space (which included resampling to 3 × 3 × 3 mm voxels), and random-effects group analyses were performed, testing for each voxel whether information values were different from zero using one-sample t tests (two-tailed). The threshold was set to t > 6.45, corresponding to p < 0.05 (Bonferroni corrected), and a minimum cluster size of five voxels. Bonferroni correction was based on the total number of 3 × 3 × 3 mm voxels that had nonzero values in the group-average anatomical brain volume (53,839 voxels).
Results
Task and behavior
Participants viewed objects that differed on two conceptual dimensions (Fig. 1): where the object is typically found (location; kitchen or garage), and how the object is commonly used (action; rotate or squeeze). Critically, these dimensions varied orthogonally to variations in low-level image characteristics and perceived object shape, thus allowing for the investigation of multiple levels of object representation in parallel. In different runs, participants (N = 26) were asked to pay attention either to the location or to the action associated with the objects, by pressing a button when an object matched the preceding object on the task-relevant dimension (see Materials and Methods). Detection performance was equally high for both tasks (action task, mean = 95.3% correct; location task, mean = 95.8% correct; difference, t(25) = 0.7, p > 0.47). Responses were faster for the location task (mean, 609 ms) than the action task (mean, 650 ms; difference, t(25) = 3.8, p < 0.001). There were no differences between the object types (e.g., kitchen versus garage objects) in mean accuracy or mean reaction time (F(1,25) < 2.0, p > 0.17, for all tests).
Schematic of analysis approach. Information about conceptual object properties was assessed by cross-correlating multivoxel fMRI activity patterns evoked by the four different object groups (illustrated by the colored vectors). Information about object location was computed by subtracting the average correlation between objects that differed in location and action (red diagonal comparisons) from the average correlation between objects with the same location (green vertical comparisons). Similarly, information about object action was computed by subtracting the average correlation between objects that differed in location and action (red diagonal comparisons) from the average correlation between objects with the same action (blue horizontal comparisons).
Representation of conceptual object properties in the ventral stream
To test for conceptual object representations, we compared, using correlation analysis, the similarity of multivoxel response patterns between object types that shared one of the two conceptual dimensions (e.g., a garlic press and a corkscrew; both are kitchen objects) with the similarity of response patterns between objects that differed in both dimensions (e.g., a garlic press and a screwdriver; Fig. 1). Importantly, the objects involved in these measures of similarity were always different objects (see Fig. 1 and Materials and Methods), and similarity in the conceptual dimensions did not covary with perceptual similarity or pixelwise similarity (see Materials and Methods). We tested for information about conceptual object properties along the ventral visual pathway implicated in object perception (Ungerleider and Mishkin, 1982). Six ROIs that covered the ventral stream were anatomically defined based on Brodmann areas in Talairach space (Materials and Methods), ranging from the most posterior region (early visual cortex; BA17) to the most anterior region of the ventral stream (temporal pole; BA38).
Activity patterns in the two most anterior regions (BA20 and BA38) carried information about both the location (t(25) > 2.4, p < 0.05, for both ROIs) and the action (t(25) > 4.2, p < 0.0005, for both ROIs) associated with the objects (Fig. 2a). No significant information about either dimension was found in the four posterior ROIs (t(25) < 1.6, p > 0.12, for all tests). Thus, information about object location and action first emerges at the level of BA20, corresponding to the anterior inferior temporal cortex (Fig. 2a). Information about the location and action dimensions in BA20 and BA38 did not differ as a function of task (Fig. 3), and significant information was observed for both the task-relevant dimension (t(25) > 3.2, p < 0.005, for both ROIs) and the task-irrelevant dimension (t(25) > 3.7, p < 0.005, for both ROIs). Finally, information about conceptual object properties (averaged across the two dimensions) in BA20 and BA38 did not significantly differ between the two presentation versions of the experiment (p > 0.53, for both ROIs) and was significant for both presentation versions of the experiment analyzed separately [blocked (N = 12), t(11) > 3.5, p < 0.005, for both ROIs; event-related (N = 14), t(13) > 3.0, p < 0.01, for both ROIs].
Results in ventral stream regions of interest. a, Conceptual information. Significant information about object location (light green bars) and object action (dark green bars) was found in the two most anterior ROIs of the ventral stream, BA20 and BA38. Conceptual information was computed as shown in Figure 1. b, Perceptual information. Response patterns in BA37 carried information about subjectively rated perceptual similarity of the objects. Perceptual information reflects the correlation between neural similarity and perceptual similarity. c, Pixelwise information. Response patterns in the three most posterior ROIs, BA17, BA18, and BA19, carried information about the pixelwise similarity of the objects. Pixelwise information reflects the correlation between neural similarity and pixelwise similarity. Error bars indicate SEM.
Pixelwise, perceptual, and conceptual information (location, action) for each ROI, separately for both tasks, is given. There were no significant differences between the tasks for any of the information measures in any of the ROIs (t(25) < 1.5; p > 0.14, for all tests). Error bars indicate SEM.
Representation of conceptual object properties in the whole brain
Are conceptual object representations specific to anterior ventral temporal cortex, or can these be found in other brain regions as well? To address this question, we next tested for conceptual object information throughout the brain, employing a whole-brain “searchlight” analysis (Kriegeskorte et al., 2006). The analysis approach was identical to that used in the ventral stream ROI analysis (Fig. 1), except that ROIs were 8 mm spheres (corresponding to 65 voxels) centered on each voxel of the brain in turn. The analysis was performed for each participant separately in native space. Information values were then transformed into Talairach space, and random-effects group analyses were performed. The threshold was set to p < 0.05 (corrected for multiple comparisons; see Materials and Methods).
No significant clusters were found when contrasting information between the two dimensions (action, location; averaged across tasks), or when contrasting information between the task-relevant and task-irrelevant dimensions (averaged across action and location dimensions). Therefore, to maximize statistical power, information for the action and location dimensions was averaged, testing for voxels in which average conceptual information values were different from zero using one-sample t tests (two-tailed). Results revealed highly significant conceptual information in bilateral anterior ventral temporal cortex [left hemisphere (LH) peak: x,y,z = −34, 8, −33; t(25) = 7.5; p < 0.0001; right hemisphere (RH) peak: x,y,z = 40, 14, −33; t(25) = 7.8; p < 0.0001; Fig. 4a], providing further evidence for conceptual object representations in anterior parts of the ventral stream, and showing that such information is specific to the most anterior parts of the temporal lobe. Information in the peak spheres did not significantly differ between the action and location dimensions (LH: F(1,25) = 1.9, p = 0.18; RH: F(1,25) = 4.4, p = 0.05) or between the task-relevant and task-irrelevant dimensions (LH: F(1,25) = 0.1, p = 0.75; RH: F(1,25) = 0.4, p = 0.53). In addition to the bilateral cluster in anterior temporal cortex, the searchlight analysis revealed a cluster in right lateral prefrontal cortex at the location of the middle frontal gyrus (peak: x,y,z = 37, 43, 8; t(25) = 7.4; p < 0.0001). Similar to the clusters in anterior temporal cortex, information did not significantly differ between the two dimensions (F(1,25) = 2.1, p = 0.16) and was not modulated by task relevance (F(1,25) = 0.2, p = 0.64).
Results of whole-brain multivoxel searchlight analyses. a, Conceptual information. Information about conceptual object properties (location, action) was found in bilateral anterior temporal cortex. Conceptual information was computed as shown in Figure 1. b, Perceptual information. Information about perceptual object properties was found in bilateral occipitotemporal cortex. Perceptual information was computed by correlating neural similarity and perceptual similarity matrices. c, Pixelwise information. Information about pixelwise image properties was found in bilateral occipital cortex. Pixelwise information was computed by correlating neural similarity and pixelwise similarity matrices. Results were overlaid on the group-average anatomical scan and thresholded at p < 0.05 (Bonferroni corrected).
Representation of perceptual and image similarity in the ventral stream
Next, we aimed to reveal brain regions in which activity patterns followed the perceptual, rather than conceptual, similarity of the objects. The perceptual similarity of the objects was assessed through pairwise ratings from a separate group of participants (N = 15; see Materials and Methods) who rated the degree to which each pair of objects subjectively looked alike, disregarding low-level image characteristic such as image size. This perceptual similarity matrix was then correlated with the neural similarity matrix in each ROI (Kriegeskorte et al., 2008). A positive correlation between these matrices indicates that objects that were perceptually similar to each other evoked relatively similar activity patterns in the ROI. Conversely, the absence of a positive correlation indicates that the variability in activity patterns in a given ROI was not related to the perceptual similarity of the objects, which provides an additional test for our claim that anterior ventral temporal regions represented the conceptual rather than perceptual properties of the objects.
Perceptual similarity was reflected in response patterns in BA37 (t(25) = 5.5, p < 0.0001; Fig. 2b), a region located in posterior occipitotemporal cortex. Activity patterns in this region were thus relatively similar to objects that were perceived as relatively similar in shape. Perceptual similarity information in BA37 did not significantly differ between presentation versions of the experiment (p = 0.09) and was significant for both presentation versions of the experiment analyzed separately (blocked: t(11) = 2.5, p < 0.05; event-related: t(13) = 5.7, p < 0.0001). BA19, neighboring BA37 posteriorly, showed a trend toward significant perceptual similarity information (t(25) = 1.9, p = 0.07). No effect of perceptual similarity was found in BA17 and BA18 (t(25) < 0.1, for both ROIs). The absence of perceptual similarity information in BA17 and BA18 suggests that participants were successful in ignoring low-level image similarity when rating the perceptual similarity of the objects. Finally, the anterior regions that showed conceptual object information, BA20 and BA38, did not carry information about perceptual similarity (t(25) <0, for both ROIs).
To reveal low-level representations of the objects, we established the low-level similarity between each object pair through pixelwise correlation analysis (see Materials and Methods). This yielded a matrix describing the similarity between the objects at the pixel level. Similar to the perceptual similarity analysis, we then correlated this matrix with the neural similarity matrix in each ROI. A positive correlation between these matrices would be expected in regions that represent low-level, image-based features of the objects, such as regions differentiating image size. This analysis also provides a further test for the independence of conceptual and perceptual representations observed in our previous analyses with pixel-based representations, thus allowing us to test for a triple dissociation between these three levels of object representation.
Activity patterns in the three most posterior regions of the ventral stream (BA17, BA18, BA19) reflected the pixelwise similarity between the objects (BA17: t(25) = 7.3, p < 0.0001; BA18: t(25) = 5.8, p < 0.0001; BA19: t(25) = 3.3, p < 0.005; Fig. 2c). Pixelwise similarity information in BA17, BA18, and BA19 did not significantly differ between presentation versions of the experiment (p > 0.16, for all ROIs). For BA17 and BA18, pixelwise similarity information was significant for both presentation versions of the experiment analyzed separately (blocked: t(11) > 5.8, p < 0.0001, for both ROIs; event-related: t(13) = 3.5, p < 0.005, for both ROIs). Activity patterns in BA37 were negatively correlated with pixelwise similarity (t(25) = −3.6, p < 0.005), which may be explained by the finding that perceptual similarity (encoded in BA37; Fig. 2b) was negatively correlated with pixelwise similarity (see Materials and Methods). No relation between activity patterns and the pixelwise similarity measure was found in BA20 and BA38 (t(25) < 0, for both ROIs).
Representation of perceptual and image similarity in the whole brain
To test for perceptual and image similarity throughout the brain, we computed information for these measures in whole-brain searchlight analyses, following the analysis approach used for the ventral stream ROIs. Information about perceptual similarity peaked in lateral occipitotemporal cortex (LH peak: x,y,z = −40, −59, −4; t(25) = 6.8; p < 0.0001; RH peak: x,y,z = 44, −59, −4; t(25) = 9.5; p < 0.0001) (Fig. 4b). Information about pixelwise similarity peaked in occipital cortex (LH peak: x,y,z = −14, −89, −4; t(25) = 13.0; p < 0.0001; right hemisphere (RH) peak: x,y,z = 20, −89, −1; t(25) = 20.2; p < 0.0001; Fig. 4c). No other significant clusters were observed. These results thus confirm the ROI analysis and provide further evidence for a triple dissociation between image-level, perceptual-level, and conceptual-level object representations in the ventral stream.
Univariate analyses in anatomical ROIs
Mean parameter estimates (reflecting response magnitude) were analyzed in repeated-measures ANOVAs with location (kitchen, garage) and action (rotate, squeeze) as factors. In the three posterior ROIs (BA17, BA18, BA19), there was a significant interaction between location and action (F1,25 > 6.6; p < 0.05, for all 3 ROIs). For rotate objects, responses were stronger to kitchen than to garage objects (t(25) > 1.7; p < 0.10, for all 3 ROIs), while for squeeze objects the reverse pattern was observed (t(25) > −2.1; p < 0.05, for all 3 ROIs). These effects may reflect the relatively large size (i.e., number of pixels) of the kitchen rotate and garage squeeze objects. BA37 showed a significant interaction between location and action (F(1,25) = 13.8; p = 0.001). For rotate objects, responses were stronger to garage than to kitchen objects (t(25) = 2.2; p = 0.04), while for squeeze objects the reverse pattern was observed (t(25) = −2.8; p = 0.01). Thus, by contrast to the posterior ROIs, BA37 appeared to respond most strongly to the relatively smaller objects. In the anterior ROIs (BA20, BA38), there were no significant main effects or interactions (p > 0.06, for all tests). Thus, unlike the MVPA, analysis of the mean responses in BA20 and BA38 did not discriminate between the objects associated with different actions or locations, responding equally to kitchen and garage objects and to rotate and squeeze objects.
Discussion
The present results show that activity patterns in anterior ventral temporal cortex carry information about how and where an object is typically used. This information was independent of the perceptual properties of the objects. Whole-brain multivoxel searchlight analysis revealed that conceptual object information was anatomically specific to the anterior part of the temporal lobe. Information about object location and object action was independent of whether participants made location or action judgments and was observed both when these dimensions were task relevant and when they were not. This suggests that the conceptual properties of the objects were retrieved even when they were irrelevant to the task. Importantly, this finding makes it unlikely that the information in ATL reflects explicit task-specific verbalization (e.g., “rotate” or “squeeze” during the action task).
In addition to investigating information about conceptual properties, our stimulus set was selected to allow for an independent investigation of information about perceptual and pixelwise object properties. Information about perceptual similarity was found in the posterior occipitotemporal cortex at the location of object-selective cortex (Malach et al., 1995). Activity patterns in this region followed the perceived shape similarity of the objects, confirming previous reports (Haushofer et al., 2008; Op de Beeck et al., 2008; Drucker and Aguirre, 2009). Finally, activity patterns in occipital cortex followed the pixelwise similarity of the objects. In summary, our results provide evidence that fMRI activity patterns in the ATL carry information about conceptual properties of everyday objects independent of the perceptual or low-level properties of these objects, which are encoded more posteriorly.
The finding that multivoxel activity patterns in the ATL carry information about conceptual object properties converges remarkably well with patient (Hodges et al., 1992; Damasio et al., 1996; Hodges et al., 2000; Mummery et al., 2000; Nestor et al., 2006; Mion et al., 2010), transcranial magnetic stimulation (Pobric et al., 2007; Lambon Ralph et al., 2009), positron emission tomography (Damasio et al., 1996; Vandenberghe et al., 1996; Noppeney and Price, 2002a,b; Bright et al., 2004; Rogers et al., 2006), and distortion-corrected fMRI (Binney et al., 2010; Visser et al., 2010a, 2012) studies that have implicated the ATL in conceptual knowledge, semantic memory, and the ability to form coherent conceptual representations (Lambon Ralph et al., 2010; Mayberry et al., 2011). Particularly relevant to the present study is the finding that patients with damage to the ATL have impaired knowledge about both the location and action of objects (Hodges et al., 2000), such that they may no longer know that a corkscrew is typically found in the kitchen and may fail to correctly demonstrate its use (Hodges et al., 2000). Our results show that object-associated location and action dimensions are selectively encoded in the ATL, indicating that the semantic deficits seen in semantic dementia patients most likely reflect damage to the ATL (Mummery et al., 2000; Nestor et al., 2006; Mion et al., 2010) rather than co-occurring damage to other brain regions such as the posterior fusiform cortex (Williams et al., 2005).
The finding that both location and action dimensions were represented in ATL suggests that this region may represent conceptual object properties related to multiple modalities; an object's action relates to motoric properties, while an object's location relates to visuospatial representations of scenes. It is important to note, however, that action categories in the present study (rotate vs squeeze) were purposely defined at a relatively abstract level to reveal regions in which representations generalize across specific sensorimotor features. For example, while both a screwdriver and a corkscrew involve a wrist rotation movement, their specific motor patterns and hand postures are quite distinct. It would be interesting for future studies to test whether activity patterns in the ATL also provide information about specific hand postures and motor patterns associated with objects, or whether these are instead represented in sensorimotor regions such as (pre)motor cortex, parietal cortex, and occipitotemporal cortex (Martin et al., 1995; Chao and Martin, 2000; Canessa et al., 2008; Bach et al., 2010; Pobric et al., 2010; Ishibashi et al., 2011; Bracci et al., 2012; Valyear et al., 2012).
Results from whole-brain searchlight analysis testing for conceptual information throughout the brain revealed a bilateral cluster in ATL as well as a cluster in the right prefrontal cortex. While these results point to a high degree of anatomical specificity of conceptual information in the anterior parts of the temporal lobes, the absence of significant clusters beyond the ATL should be interpreted with caution. It is plausible that future studies with different designs, different conceptual dimensions, and/or increased power may reveal additional areas beyond the ATL. Furthermore, although multivoxel pattern analysis is a highly sensitive technique capable of revealing neural representations, like other techniques it also has its limitations. Specifically, MVPA is ultimately limited by the spatial resolution of BOLD fMRI and will be insensitive to representations that are distributed at spatial scales that are below a certain (as yet to be determined) spatial resolution (Kriegeskorte et al., 2010; Op de Beeck, 2010; Swisher et al., 2010). Our results are therefore not necessarily in opposition to studies that used other methods, such as fMRI adaptation (Grill-Spector and Malach, 2001), to reveal semantic representations in regions beyond the ATL (Dehaene et al., 1998; Kotz et al., 2002; Vuilleumier et al., 2002; Copland et al., 2003; Rissman et al., 2003; Rossell et al., 2003; Wheatley et al., 2005). Furthermore, not all regions that are important for semantic cognition are expected to represent conceptual object properties. Regions involved in content-general processes such as executive control or the selection of semantic representations stored elsewhere, while critical for intact semantic cognition (Jefferies and Lambon Ralph, 2006), may not necessarily show neural activity that distinguishes between conceptual object properties. Finally, as mentioned in the previous paragraph, modality-specific brain regions may store object properties that are more specific than the relatively abstract properties probed in the present study.
To conclude, our results provide the first evidence that fMRI activity patterns in the ATL carry information about conceptual properties of everyday objects. Such representations abstract away from modality-specific attributes and support higher-order generalization, such as the knowledge that multiple, perceptually different objects belong to the same category (Patterson et al., 2007). In this view, object knowledge is hierarchically organized from low-level representations, which capture sensorimotor features associated with specific individual objects, to semantic representations, which capture generalized properties of objects and no longer reflect the specific, sensorimotor details associated with individual objects. Our findings support this hypothesis by showing that activity patterns in ATL, but not in occipitotemporal cortex, carry information about semantic object properties.
Footnotes
This research was financially supported by the Fondazione Cassa di Risparmio di Trento e Rovereto.
- Correspondence should be addressed to Marius V. Peelen, Center for Mind/Brain Sciences, University of Trento, Corso Bettini 31, 38068 Rovereto (TN), Italy. marius.peelen{at}unitn.it
