Abstract
The principles driving the functional organization of object representations in high-level visual cortex are not yet fully understood. In four human fMRI experiments, we provide evidence that the organization of high-level visual cortex partly reflects the degree to which objects are typically controlled by the body to interact with the world, thereby extending the body's boundaries. Univariate whole-brain analysis showed an overlap between responses to body effectors (e.g., hands, feet, and limbs) and object effectors (e.g., hammers, combs, and tennis rackets) in lateral occipitotemporal cortex (LOTC) and parietal cortex. Region of interest analyses showed that a hand-selective region in left LOTC responded selectively to object effectors relative to a range of noneffector object control conditions (e.g., graspable objects, “act-on” objects, musical instruments). Object ratings showed that the strong response to object effectors in hand-selective LOTC was not due to general action-related object properties shared with these control conditions, such as hand priming, hand grasping, and hand-action centrality. Finally, whole-brain representational similarity analysis revealed that the similarity of multivoxel object response patterns in left lateral occipitotemporal cortex selectively predicted the degree to which objects were rated as being controlled by and extending the body. Together, these results reveal a clustering of body and object effector representations, indicating that the organization of object representations in high-level visual cortex partly reflects how objects relate to the body.
Introduction
The principles driving the functional organization of high-level visual cortex have been hotly debated for decades. In addition to visual and shape properties (Haxby et al., 2000; Levy et al., 2001; Hasson et al., 2002; Op de Beeck et al., 2008), recent evidence suggests that motor-relevant properties of objects may also shape the organization of the visual system (Mahon et al., 2007; Bracci et al., 2012; Peelen et al., 2013). For example, recent studies reported that parts of the left lateral occipitotemporal cortex (LOTC) respond strongly to pictures of hands as well as to pictures of mechanical tools (Bracci et al., 2012), with tool-selective responses developing even in the complete absence of visual experience (Peelen et al., 2013). This raises the question of whether object responses in high-level visual cortex cluster according to specific action-related properties.
One such action-related property is the degree to which an object is directly controlled by the body to interact with the world, thereby physically and functionally extending the body's boundaries. For example, we can hit a ball with our hands (when playing volleyball) but also with a racket (when playing tennis). When playing tennis, the racket effectively replaces the hand as the end-effector of the action and needs to be incorporated into this “new” physical configuration of the body for the action to be successfully controlled (Arbib et al., 2009). Behavioral and neurophysiological studies on tool use have shown that the multisensory representation of the hand can indeed be extended to include a tool (Iriki et al., 1996; Maravita and Iriki, 2004; Cardinali et al., 2009), possibly reflecting the assimilation of tools into the body schema (Head and Holmes, 1911). This raises the possibility that object representations in high-level visual cortex partly follow an organization related to how the body interacts with objects, clustering representations of body effectors (e.g., hands) and object effectors (e.g., tennis rackets) that extend the body physically and functionally.
To test this hypothesis, we performed four fMRI experiments measuring responses in visual cortex to a wide range of effector and noneffector objects. The objects included in these studies were independently rated on several action-related dimensions, including the degree to which the objects were perceived as extensions of the body. Regression analysis was used to test the degree to which the response profiles of hand- and body-selective regions were related to these dimensions. Subsequently, whole-brain multivoxel representational similarity analysis (RSA) was used to test for brain regions in which multivoxel patterns of activity reflected action-related dimensions of objects. Thus, combining behavioral ratings of object properties with region of interest (ROI) and whole-brain fMRI analyses, we tested for the influence of action-related object properties on the functional organization of high-level visual cortex.
Materials and Methods
Participants
All participants were right-handed adult volunteers who gave informed consent to take part in an fMRI experiment. Forty-one unique participants were recruited, with 7 of these participants participating in two experiments. The following number of participants took part in each experiment: Experiment 1: n = 13 (6 females); mean age, 27 years; age range, 22–54 years; Experiment 2: n = 13 (9 females); mean age, 27 years; age range, 21–54 years; Experiment 3: n = 11 (5 females); mean age, 28 years; age range, 22–53 years; Experiment 4: n = 11 (6 females); mean age, 26 years; age range, 21–30 years. All studies were approved by the ethics committee of the University of Trento.
Stimuli and behavioral ratings
Across the four experiments, the stimulus set consisted of exemplars from 22 different object and body categories, as shown in Figure 1. Each object category (e.g., musical instruments) included six different exemplars of six different object types (six guitars, six tambourines, six violins, six saxophones, six trombones, and six accordions). Each body part category (e.g., upper limbs) included 36 different exemplars of the same body part (36 different upper limbs).
A new group of participants (Group 1, n = 15) took part in a behavioral experiment to rate the stimuli on the following four action-related object dimensions: (1) “body extension”; (2) “hand movement prime”; (3) “hand grasping”; and (4) “hand-action centrality.” These dimensions pertained to 12 of the following experimental conditions: tools, act-with objects, act-on objects, high-graspability objects, low-graspability objects, small/portable musical instruments, big/nonportable musical instruments, small/portable objects, large/nonportable objects, sport act-with objects, sport-related objects, and chairs. The rating experiment consisted of seven statements. Each statement was rated on a scale from 1 (strongly disagree) to 7 (strongly agree). For dimensions consisting of two statements, ratings were averaged across the two statements. The body extension dimension reflected the degree to which an object is perceived as being controlled by and to physically extend the body during its typical use, and was tested with the following two statements: (1) “This object is like a physical extension of my hand or arm; after using it for a while it almost feels to become part of my body”; and (2) “When I use this object, my hand/arm movements are directly controlling this object to physically act on another object or surface.” The hand movement prime dimension reflected the degree to which an object might prime hand movements and was tested with the following statement: “When I see this object I would like to move my hands/arms.” The hand-grasping dimension reflected the objects' graspability and was tested with the following two statements: (1) “This object is easy to pick up”; and (2) “This object is designed specifically for being easily graspable by one or both hands.” The hand-action centrality dimension reflected the degree to which skilled hand action is a central characteristic of the object, and was tested with the following two statements adopted from Mahon et al. (2007): (1) “I could probably tell what object this is just by looking at the hand movements associated with it”; and (2) “Hand movements performed when using this object are important for the object's function.” Moreover, to test the extent to which the objects were generally associated with hands, an additional group of participants (Group 2, n = 15) rated the stimuli on a hand association dimension, as well as the body extension dimension. The hand association dimension consisted of the following two statements: (1) “This object makes me think about my hand/s”; and (2) “When I see this object I mentally imagine hands.”
The rating procedure was as follows. Each participant rated all 12 object categories for which ratings applied. Each object category (e.g., portable musical instruments) consisted of six types (e.g., violin, tambourine), with six exemplars each. Due to time constraints, each participant rated one exemplar of three of the six types (e.g., three different musical instruments). Image presentation was controlled by a PC running the Psychophysics Toolbox package (Brainard, 1997) in MATLAB (MathWorks). Objects were shown at fixation on the computer screen, and participants rated the images on each of the seven questions by pressing keys 1–7. No time limit was imposed. Before the experiment, participants performed a practice block to familiarize them with the rating questions. Images used in the practice block were different from the images used in the rating experiment.
Experimental design
Each fMRI experiment consisted of six runs lasting 6 min and 20 s each. Each run comprised 44 blocks of 8 s. Each run consisted of a fully randomized sequence of nine category blocks (each repeated four times) and fixation baseline-only blocks (repeated eight times). Fixation blocks of 14 s additionally appeared at the beginning and at the end of each run. Within each stimulus block, 10 images from one category were each presented for 400 ms, followed by a blank screen for 400 ms. Once during each block, the same picture was presented two times in succession. Participants were required to detect these repetitions and report them with a button press (1-back task). Each object category consisted of two sets of 36 different grayscale images on a white background and had a size of 12° × 12° (400 × 400 pixels). One set of images was used for even runs, and the other set of images was used for odd runs. Within each run, each presented image was unique (except for the repetitions related to the 1-back task). Stimulus presentation was controlled by a PC running the Psychophysics Toolbox package (Brainard, 1997) in MATLAB. Pictures were projected onto a screen and were viewed through a mirror mounted on the head coil.
Data acquisition and preprocessing
Functional and structural images were acquired on a Bruker BioSpin MedSpec 4-T scanner with standard head coil at the Center for Mind/Brain Sciences, University of Trento. Functional scans were acquired with echoplanar imaging T2*-weighted scans. Acquisition parameters were as follows: repetition time of 2 s, echo time of 33 ms, flip angle of 73°, field of view of 192 mm, and matrix size of 64 × 64. Each volume contained 34 axial slices (covering the whole brain) with 3 mm thickness and a 1 mm gap. Structural scans were acquired with an MP-RAGE sequence, with 1 × 1 × 1 mm resolution.
For all studies, data preprocessing and analysis were performed using Brain Voyager QX (version 2.20; Brain Innovation) and MATLAB. Functional images underwent three-dimensional head motion correction, linear trend removal, high-pass temporal filtering (cutoff, three cycles per time course) and spatial smoothing (4 mm full-width at half-maximum isotropic Gaussian kernel). After coregistration of the functional images to the T1 anatomical images, the anatomical images were transformed into Talairach stereotaxic space, and this transformation was applied to the aligned functional data, which was interpolated to 1 × 1 × 1 mm.
Statistical analysis
For each participant, a general linear model (GLM) was created to model the conditions in the experiment. The GLM included regressors for the conditions of interest and the six motion correction parameters (x, y, z for translation and for rotation). Predictors' time courses were modeled with a linear model of hemodynamic response using the default Brain Voyager QX “two-gamma” function. Before computing the GLM, functional runs were z-normalized.
ROI definition.
A hand-selective region (LOTC-hand) and a body-selective region (LOTC-body) of interest were defined in individual participants to investigate the functional profile of these regions. In all studies, the left LOTC-hand region was defined using the contrast: hands > bodies + chairs. The left LOTC-body region was defined using the contrast: bodies > hands + chairs. The inclusion of chairs in the contrast allowed localization of hand- and body-selective regions in most participants, while also separating body- and hand-selective regions to some degree (Bracci et al., 2010). The threshold for ROI definition was set at p < 0.001 (uncorrected). ROIs were restricted to a cube of 10 mm width centered on the activation peak. The extracted parameter estimates used for statistical testing were independent of the data used for ROI definition. That is, when ROIs were defined using data from odd runs, parameter estimates were extracted from even runs, and vice versa. Estimates were averaged across even and odd runs. Following evidence reporting stronger lateralization for tools in the left hemisphere (Chao et al., 1999; Chao and Martin, 2000) and overlapping hand and tool responses in left LOTC (Bracci et al., 2012), we focused our analyses on the left hemisphere. However, we do not exclude the possibility that a similar organization for body and object effectors is also present in right LOTC.
Regression analysis.
We used regression analysis to assess how variability in the fMRI responses in LOTC-hand and LOTC-body regions related to the variability in the four rated action-related object dimensions and the hand-association dimension (Tables 1, 2). The fMRI response profiles consisted of the responses to the 12 conditions for which ratings were collected (Table 2), averaged across participants and experiment. For those objects that were presented in multiple experiments (e.g., tools, chairs), responses were averaged across these experiments. For the other objects, responses were the mean responses for these objects in the experiments in which they were presented. This resulted, for both ROIs, in one response value for each of the 12 objects. This fMRI response profile (normalized to a mean of 0 and an SD of 1) constituted the dependent variable in the regression analysis, with the behavioral ratings (normalized to a mean of 0 and an SD of 1) included as independent variables. Regression analyses were run for each of the independent rating datasets (n = 15). Differences between the resulting regression coefficients were tested using pairwise t tests.
Talairach coordinates for LOTC-hand and LOTC-body
Object ratings
Multivoxel pattern analysis.
Multivoxel pattern analysis (MVPA) was used to relate voxelwise variations in hand and body responses to voxelwise variations in object responses, following the approach of previous studies (Peelen et al., 2006; Downing et al., 2007; Bracci et al., 2012). For each individual participant, we defined a region in left occipitotemporal cortex selective to hands and bodies by contrasting these two categories with chairs, using the data of all six runs. The ROI was defined by selecting all active voxels (p < 0.001, uncorrected) within a cube with 2 cm width centered on the activation peak. For each participant, parameter estimates were extracted for each voxel and each condition (relative to baseline), separately for even and odd runs. The multivoxel activity patterns for bodies and hands in the odd runs were then correlated with the multivoxel activity patterns for all the other categories in the even runs, and vice versa. Correlations were Fisher transformed, and corresponding odd–even run correlations were averaged (e.g., handsodd − feeteven and feetodd − handseven). Chairs were excluded from the analysis because this condition was used as the baseline for the ROI definition, and these correlations would thus not be comparable to correlations involving the other objects. The resulting correlations were tested with pairwise t tests, comparing the correlation between hands and other conditions with the corresponding correlation between bodies and other conditions.
Whole-brain univariate analysis.
Whole-brain random-effects group analyses were performed combining data from all experiments. Seven of 48 participants took part in two experiments. In these cases, only one dataset was included in the group analysis (n = 41). Each condition in each experiment was relabeled according to the following groups: (1) “bodies” included noneffector human body parts (torsos, bodies); (2) “body effectors” included body parts used as effectors (e.g., active hand postures, resting hands, feet, upper limbs, and lower limbs); (3) “object effectors” included body-extending object effectors (e.g., mechanical tools, act-with objects, and sport-related act-with objects); (4) “objects” included all noneffector objects (e.g., act-on objects, small/portable musical instruments, big/nonportable musical instruments, small/portable objects, large/nonportable objects, high-graspability objects, low-graspability objects, sport-related objects, chairs, and vehicles). Faces and animals were excluded from this analysis.
Whole-brain representational similarity analysis.
For the object conditions that were rated on the four action-related dimensions (Table 2), we created matrices reflecting the dissimilarity on each of these four dimensions (see Fig. 6A). Dissimilarity was computed as the absolute difference between pairwise ratings. Subsequently, we used representational similarity analysis (Kriegeskorte et al., 2008) to relate these behavioral dissimilarity matrices to neural dissimilarity matrices. Neural dissimilarity was derived from the correlation between multivoxel response patterns between each object pair (1-correlation). Subsequently, we used a spherical searchlight approach (Kriegeskorte et al., 2006) to correlate behavioral dissimilarity matrices (reflecting the dissimilarity between the objects on each of the four dimensions) with neural dissimilarity matrices throughout the brain.
For each experiment and for each participant, we extracted parameter estimates for the included object conditions. For each voxel in the brain, we defined a searchlight sphere of 9 mm radius, and parameter estimates within voxels in each searchlight were used to compute pattern similarities for each condition pair using the Pearson correlation. The resulting values were averaged across participants and across experiments to create a final representational dissimilarity (1-correlation) matrix for the 12 object categories. This symmetrical 12 × 12 dissimilarity matrix contains empty cells since different experiments tested different object categories (only tools and chairs were included in all experiments; see Fig. 6A,B). Next, for each of the four behavioral dimensions (i.e., body extension, hand movement prime, hand-action centrality, and hand grasping) and for each of the 15 participants of the rating experiment, we created a similarity matrix reflecting the absolute distance between ratings of all object pairs. Finally, for each sphere, the neural dissimilarity matrix was correlated with the four behavioral dissimilarity matrices, separately for each of the 15 participants. Only values that were present in both behavioral and neural dissimilarity matrices were used in the analysis. Resulting correlation values were Fisher transformed and reassigned to the center voxel of the spheres to generate whole-brain correlation maps for each of the four dimensions. Differences between these statistical maps were tested using whole-brain group analysis.
Results
The present study consisted of four fMRI experiments, each including pictures of nine object categories (Fig. 1), with hands, bodies, tools, and chairs presented in all experiments to allow for consistent definition of regions of interest across experiments. The design, task, and scanning parameters of the four experiments were identical (see Materials and Methods).
Conditions tested in Experiments 1–4. Examples of object categories tested in the four experiments. Different color codes represent stimuli tested in each experiment (yellow for Experiment 1, blue for Experiment 2, magenta for Experiment 3, and green for Experiment 4) and across more than one experiment (orange for Experiments 1, 3, and 4; dark magenta for Experiments 1–4).
Region of interest analyses
In a first analysis, we tested whether hand- and body-selective LOTC regions respond preferentially to object effectors relative to a wide range of object control conditions that varied on several dimensions. ROIs were functionally defined in each participant. A hand-selective region (LOTC-hand; Bracci et al., 2010) was defined by the contrast, hands > [bodies + chairs], and a body-selective region (LOTC-body; Downing et al., 2001) was defined by the contrast, bodies > [hands + chairs]. LOTC-hand and LOTC-body could be defined in 39 of 41 participants (7 of 41 participants took part in two experiments). Figure 2A shows these regions in five representative participants, and Table 1 gives mean Talairach coordinates for each of the four experiments. Within these ROIs, we measured fMRI responses evoked by the conditions presented in the experiments (Fig. 1), using a split-half method to ensure that ROI definition was statistically independent of the data presented (see Materials and Methods). Each experiment included stimulus categories selected to test specific hypotheses about the object properties that might drive responses in LOTC-hand and LOTC-body.
Results of ROI analysis. A, Individual-participant LOTC-hand ([hands > whole bodies + chairs], and LOTC-body ([whole bodies > hands + chairs]) ROIs are shown for five representative participants. ROIs were restricted to cubes of 10 mm width centered on the activation peak (threshold, p < 0.001, uncorrected). LH, Left hemisphere. B, Mean responses for all stimulus categories in all experiments are rank ordered by response amplitude in LOTC-hand and LOTC-body. Bars for each condition are color coded according to the following groups: bodies include noneffector human body parts (torsos, bodies); body effectors include body parts used as effectors (active hand postures, resting hands, feet, upper limbs, lower limbs); object effectors include body-extending object effectors (mechanical tools, act-with objects, sport-related act-with objects); and objects include all noneffector objects (act-on objects, small/portable musical instruments, big/nonportable musical instruments, small/portable objects, large/nonportable objects, high-graspability objects, low-graspability objects, sport-related objects, chairs, vehicles). Faces and animals were excluded from this classification and are shown in white. As a measure of selectivity, all conditions were tested within experiments against the control category chairs (chairs were included in all experiments) using two-tailed pairwise t tests (e.g., hands in Experiment 1 vs chairs in Experiment 1). Object categories that survived the Bonferroni-corrected threshold of p < 0.0016 (p = 0.05/36 stimulus conditions) are highlighted with gray background. In LOTC-body, hands (Experiment 2) also survived Bonferroni correction, and the corrected p value is reported above the bar. Numbers below each bar represent the experiment number (1–4) of each condition. Error bars indicate SEM.
Figure 2B shows the results in hand- and body-selective left LOTC of all four experiments, rank ordered by response amplitude. Responses of all experiments were plotted together to allow for a complete overview of the response profiles. The following three types of object effectors were presented: (1) mechanical tools such as hammers, saws, and pliers; (2) everyday “act-with” objects such as pens, combs, and forks; and (3) sport act-with objects such as tennis rackets, golf clubs, and fencing swords. These three conditions all consisted of objects that are typically controlled by hands/arms to act on another surface or object, and that thereby physically and functionally extend the body during normal object use (Maravita and Iriki, 2004). Multiple control object conditions were included to capture properties that are not specific to object effectors.
In Experiment 1, everyday “act-on” objects (e.g., doorbells, light switches, and door handles) were included to control for motor affordance. Similar to object effectors (tools and act-with objects in Experiment 1), act-on objects are strongly associated with hand movements but, unlike them, they are not used by the body as action effectors. Tools and act-with object conditions were each compared with act-on objects in two 2 × 2 ANOVAs with ROI (LOTC-hand and LOTC-body) and Category (tools, act-on objects; or act-with objects, act-on objects) as within-subject factors. Results revealed no interactions between Category and ROI (F < 2 for both tests). Importantly, both ANOVAs revealed a main effect of Category (F(1,12) > 28.00, p < 0.001, for both tests), reflecting stronger responses to tools and act-with objects than to act-on objects. There was also a main effect of ROI (F(1,12) > 6.07, p < 0.05, for both tests), with overall stronger responses in LOTC-hand than LOTC-body.
To test for the influence of object graspability, responses to high-graspability objects (e.g., book, money, wallet) were compared with responses to low-graspability objects (e.g., wall lamp, wall clock, vase) in a 2 × 2 ANOVA with ROI (LOTC-hand, LOTC-body) and Category (high graspability, low graspability) as within-subject factors. There was a significant interaction between ROI and Category (F(1,12) = 12.66, p = 0.004), with stronger responses to high-graspability objects than to low-graspability objects in LOTC-hand (t(12) = 4.24, p < 0.001) but not in LOTC-body (p > 0.50). Importantly, however, pairwise comparisons showed that responses to both tools and act-with objects were still significantly stronger than responses to high-graspability objects in both ROIs (t(12) > 3.31, p < 0.006, for all tests).
In Experiment 2, musical instruments were included to control for skilled hand action. Similar to object effectors, musical instruments are associated with complex and skilled hand actions. However, musical instruments (e.g., piano) are not controlled by the body to act on other objects, and therefore do not physically extend the body to the same degree as object effectors (act-with parts of instruments such as drum sticks were not included). Two types of semantically matched musical instruments were included: small/portable musical instruments (e.g., accordion, guitar, tambourine); and big/nonportable musical instruments (e.g., piano, contrabass, timpani). Tools were tested against small/portable and big/nonportable musical instruments in two 2 × 2 ANOVAs, with ROI (LOTC-hand, LOTC-body) and Category (tools, small/portable musical instruments; or tools, big/nonportable musical instruments) as within-subject factors. Both ANOVAs revealed significant ROI × Category interactions (F(1,12) > 9.22, p < 0.01, for both tests), reflecting differences between the conditions in LOTC-hand and LOTC-body. Post hoc t tests showed that LOTC-hand responded more strongly to tools than to both categories of musical instruments (t(12) > 6.01, p < 0.001, for both tests), while LOTC-body responded more strongly to tools than small/portable musical instruments (t(12) = 3.63, p = 0.003), but not big/nonportable musical instruments (t(12) = 1.96, p = 0.08).
In Experiment 3, sport-related objects (e.g., tennis balls, golf balls, ping-pong table) were included to control for the sport-related object context associated with sport act-with objects. A 2 × 2 ANOVA with ROI (LOTC-hand, LOTC-body) and Category (sport act-with objects, sport-related objects) as within-subject factors revealed no significant interaction between ROI and Category (F(1,9) = 3.81, p = 0.08). Importantly, there was a main effect of Category (F(1,9) = 34.08, p < 0.001), reflecting stronger responses to sport act-with objects than to sport-related objects. There was also a main effect of ROI (F(1,9) = 8.06, p = 0.02), with overall stronger responses in LOTC-hand than LOTC-body.
To test for the influence of real-world object size (Konkle and Oliva, 2012), in Experiment 2 we compared responses between small objects (e.g., alarm clock, picture frame, small computer speakers) and big objects (e.g., grandmother clock, blackboard, large loudspeakers) that were functionally/semantically matched (see Materials and Methods). A 2 × 2 ANOVA with ROI (LOTC-hand, LOTC-body) and Category (small objects, big objects) as within-subject factors revealed no significant interaction or main effects (p > 0.05, for all tests). Thus, object size did not affect responses in these ROIs.
Finally, in Experiment 3, motion-implying animals (e.g., bird, butterfly, fish) and vehicles (e.g., racecar, motorbike, airplane) were included to test for effects of different types of implied motion (animate and inanimate). A 2 × 2 ANOVA with ROI (LOTC-hand, LOTC-body) and Category (animals, vehicles) as within-subject factors revealed a near-significant ROI × Category interaction (F(1,9) = 5.14, p = 0.05), reflecting a more positive difference between animals and vehicles in LOTC-body (t(9) = 4.40, p = 0.002) than in LOTC-hand (t(9) = 2.82, p = 0.02).
Together, these results indicate that responses in LOTC-hand and LOTC-body were strongest to several types of object effectors relative to other objects, including objects associated with hand actions, highly graspable objects, and objects associated with skilled hand actions. Small object size and implied inanimate motion could not account for this preference.
Next, we characterized the response profiles of LOTC-hand and LOTC-body relative to the control category chairs, included in all experiments. This analysis was aimed at establishing the selectivity profiles of these two regions, rather than testing for potential differences between LOTC-hand and LOTC-body, as in the analyses above. All statistical comparisons were performed within experiments using two-tailed pairwise t tests (e.g., hands in Experiment 1 vs chairs in Experiment 1). To account for multiple comparisons (8 categories × 4 experiments = 32), a Bonferroni-corrected threshold of p = 0.0016 (0.05/32) was adopted. Figure 2B indicates the categories that were significantly different from chairs in LOTC-hand and LOTC-body. Notably, LOTC-hand responded preferentially (relative to chairs) to all body effectors (hands, feet, upper limbs, lower limbs) and object effectors (mechanical tools, everyday act-with objects, sport act-with objects), but not to any of the other body parts or noneffector objects. Faces was the only category that was significantly lower than chairs. In LOTC-body, preferential responses (relative to chairs) were observed for all human nonface body stimuli, but not for any of the object effector conditions.
Regression analysis relating fMRI responses and behavioral ratings
For those objects to which it applied, we quantified the degree to which the objects were perceived as body-extending object effectors by collecting behavioral ratings in a separate group of participants (Group 1, n = 15; see Materials and Methods). For each object, participants rated their agreement with the following two statements: (1) “This object is like a physical extension of my hand or arm; after using it for a while it almost feels to become part of my body”; and (2) “When I use this object, my hand/arm movements are directly controlling this object to physically act on another object or surface.” The same objects were also rated on three other action-related dimensions: the degree to which an object primes hand movements, an object's graspability, and the degree to which hand action is a central characteristic of the object (see Materials and Methods for questions and rating procedure). Moreover, to test for the possibility that the response profile in LOTC-hand/LOTC-body could reflect the degree to which an object is generally associated with hands (e.g., through mental imagery), an additional group of participants (Group 2, n = 15) rated the objects on a “hand association” dimension in addition to the body extension dimension (see Materials and Methods). Table 2 gives the behavioral ratings for the relevant objects on the four action-related dimensions and the hand association dimension.
To examine the relationship between the functional response profiles of LOTC-hand/LOTC-body and the ratings on the four action-related dimensions, we used regression analysis to assess to what extent fMRI responses in LOTC-hand and LOTC-body areas (Fig. 3B) were related to the variance in the four object dimensions. Because the object conditions were tested in different fMRI experiments involving different participants (Fig. 1), fMRI data were aggregated across experiments, and inferential statistics were performed on the variability among the raters (see Materials and Methods).
Results of regression analysis. A, Single regression analyses were used to assess how variability in the fMRI responses in LOTC-hand and LOTC-body relate to the variability in the four rated action-related object dimensions (body extension, hand movement prime, hand grasping, and hand-action centrality) rated by an independent group of participants (Group 1, n = 15). Results revealed that the response profiles of LOTC-hand and LOTC-body were most strongly related to the body extension dimension (blue color-coded bar graph). Error bars indicate SEM. B, The mean fMRI responses (dependent variables) to the conditions included in the regression analyses are rank ordered by response amplitude in LOTC-hand and LOTC-body. For conditions that were presented in multiple experiments (tools and chairs), responses were averaged across these experiments. Error bars indicate SEM.
Results of the regression analyses showed that the body extension dimension was the dimension that was most strongly related to the response profiles in LOTC-hand and LOTC-body. Because the four dimensions were positively correlated (e.g., objects we perform actions with are typically associated with hand movements, are graspable, and functionally extend our body), all dimensions showed a positive relation with the response profiles of LOTC-hand and LOTC-body (p < 0.001, for all tests; Fig. 3A). Nonetheless, in both ROIs parameter estimates for the body extension dimension were significantly higher than parameter estimates for the other three dimensions (p < 0.001, for all tests; Fig. 3A). An additional regression analysis was performed to compare the body extension dimension with the hand association dimension, rated by a separate group of participants. The response profiles of left LOTC-hand and left LOTC-body showed positive parameter estimates for both dimensions (p < 0.001, for all tests) and significantly higher parameter estimates for the body extension dimension than the hand association dimension (LOTC-hand: t(14) = 7.21, p < 0.001; LOTC-body: t(14) = 5.54, p < 0.001).
These results confirm the results of the ROI analyses that directly compared responses to object effector conditions with responses to control conditions. Similar to the ROI analyses, the regression analyses indicate that responses in LOTC-hand and LOTC-body are strongest to objects that are perceived as being controlled by the body to act on other objects, thereby physically extending the body. A weaker relationship was found between fMRI responses and other action- or hand-related object properties, such as the degree to which an object primes hand movements, the degree to which it is graspable, the degree to which hand action is a central characteristic of an object, or the degree to which an objects is generally associated with hands. It should be noted, however, that these dimensions are all positively correlated with the body extension dimension and are therefore all related to the response profiles of LOTC-hand and LOTC-body (Fig. 3A). Testing responses to a set of conditions for which these dimensions are uncorrelated would better allow for fully separating the influence of these dimensions.
Mutivoxel pattern analysis in left LOTC
Several of the analyses reported here showed a similar preference for object effectors in both ROIs. To test directly whether representations of object effectors were more strongly related to hand representations or to body representations, we used MVPA (Haxby et al., 2001) to test the degree to which voxelwise variations in hand and body responses correlated with voxelwise variations in the responses to the object effector categories, following the approach used in previous studies that similarly aimed to separate overlapping neural populations (Peelen et al., 2006; Downing et al., 2007; Bracci et al., 2012).
For each experiment and each participant, we localized a region in left occipitotemporal cortex selective to both hands and bodies relative to chairs, thus encompassing both LOTC-hand and LOTC-body (see Materials and Methods). Within this region, multivoxel patterns of activity to hands and bodies were correlated with multivoxel patterns of activity to all other conditions across even and odd runs (see Materials and Methods). In all experiments, for the comparisons of hands and bodies, within-category correlations (hands–hands, bodies–bodies) were significantly higher (p < 0.001, for all tests) than between-category correlation (hands–bodies; Fig. 4), confirming that the multivoxel activity patterns for hands and bodies were reliably distinguishable from each other within left LOTC (Bracci et al., 2012). Subsequently, we tested whether the response patterns of object effectors were more similar to the response patterns of hands than to the response patterns of bodies. To this aim, pairwise t tests were used to compare the correlation of each object effector condition with hands with its correlation with bodies. Results (Fig. 4) revealed that the pattern of activity evoked by all object effectors was significantly more similar to the pattern evoked by hands than to that evoked by bodies: tools (Experiment 1: t(12) = 2.87, p = 0.01; Experiment 2: t(12) = 5.04, p < 0.001; Experiment 3: t(10) = 3.67, p = 0.005; Experiment 4: t(10) = 2.51 p = 0.03), act-with objects (Experiment 1: t(12) = 3.84, p = 0.002), and sport act-with objects (Experiment 3: t(10) = 3.50, p = 0.007). These results show that left LOTC response patterns evoked by object effectors are more strongly related to response patterns evoked by hands than to response patterns evoked by whole bodies.
Results of multivoxel pattern analysis. Multivoxel correlation matrices were computed in a left LOTC region, functionally defined in each individual participant by contrasting the average response to hands and bodies relative to chairs (see Materials and Methods). In each experiment, activity patterns for the hand and body conditions were correlated with activity patterns for the other object conditions using a split-half (odd/even) comparison. The 2 × 2 matrix (left side) shows within- and between-category correlations for hands and bodies. The 2 × 6 matrix (right side) shows correlations for hands and bodies (rows) with the other objects categories (columns) for each experiment. Each cell of the matrix represents the mean correlation value across participants within each experiment. Warm colors represent positive correlations, and cold colors represent negative correlations.
Whole-brain analyses
Univariate contrasts
To visualize the large-scale organization of body and object effectors in visual cortex, we analyzed the data from all four experiments together in a univariate whole-brain analysis (see Materials and Methods), made possible by relabeling each stimulus condition according to the following four labels used in Figure 2B: bodies, body effectors, objects, and object effectors. Subsequently, we contrasted body effectors with bodies, and object effectors with objects. By combining data across conditions and experiments, this analysis provides a powerful whole-brain test of selectivity for body and object effectors, although it does not speak to the contribution of individual categories or experiments to the overall activity observed. Figure 5 shows whole-brain group activation maps for these contrasts (p < 0.05, Bonferroni corrected). As shown in Figure 5, both body effectors (x, y, z = −48, −64, 5) and object effectors (x, y, z = −46, −68, −2) activated the left LOTC, relative to their respective control conditions. No other regions in high-level visual cortex showed such overlap, indicating that the clustering of responses to object effectors and body effectors is specific to left LOTC. Interestingly, overlapping responses to body effectors and object effectors were also observed in the left intraparietal sulcus (Fig. 5).
Results of univariate whole-brain analysis. Univariate group average activation maps (combining all four experiments) for the contrasts of body effectors versus bodies (dark blue color-coded) and object effectors versus objects (light blue color-coded) are shown at the Bonferroni-corrected threshold of p < 0.05.
Representational similarity analysis
Finally, to test for brain regions in which neural similarity between objects reflected the perceived similarity between these objects on the body extension dimension, we used RSA (Kriegeskorte et al., 2008). Whole-brain RSA was performed using a spherical searchlight approach (Kriegeskorte et al., 2006) with neural dissimilarity (1-correlation between multivoxel activity patterns) computed in spheres of 9 mm radius around each voxel of the brain. For each sphere, the neural dissimilarity matrix was correlated with matrices reflecting dissimilarity on each of the four action-related dimensions (Fig. 6A; computed as the absolute difference between pairwise ratings; Table 2). This allowed us to test for regions in which neural similarity predicted the similarity between the objects on the body extension dimension (see Materials and Methods).
Results of representational similarity analysis. A, Mean representational dissimilarity matrices for the four action-related behavioral dimensions (body extension, hand movement prime, hand grasping, and hand-action centrality) computed as the absolute distance between the ratings of each object pair. Only pairs available in the neural dissimilarity matrix were included in the analysis (see Materials and Methods). B, Mean neural dissimilarity matrices (1-correlation) for brain regions (left LOTC, left IPS, right IPS, left PM; C) in which the neural dissimilarity matrix correlated more with the body extension dissimilarity matrix than with any of the other three behavioral dissimilarity matrices (hand movement prime, hand grasping, and hand-action centrality). These matrices are included for illustration purposes only. C, Results of whole-brain RSA, showing the conjunction contrast between the body extension dimension and each of the other three dimensions, revealed a cluster in left LOTC (at p < 0.05, Bonferroni corrected). Multivoxel response patterns in this LOTC region were thus relatively similar between objects that were rated as relatively similar on the body extension dimension.
A conjunction analysis was performed to reveal clusters in which neural dissimilarity was significantly more correlated with the body extension dissimilarity than with each of the other three dissimilarity matrices. As shown in Figure 6C, results revealed a cluster in left LOTC (x, y, z = −42, −76, 6; Bonferroni-corrected threshold of p < 0.05) in which multivoxel response patterns were relatively similar between objects that were rated as relatively similar on the body extension dimension (Table 2). Two additional clusters were observed in the intraparietal sulcus (IPS; left IPS: x, y, z = −30, −63, 42; right IPS: x, y, z = 24, −47, 43) and left premotor cortex (PM; left PM: x, y, z = −21, 12, 53). The same analysis for the other three dimensions did not reveal any significant clusters. Figure 6B shows the neural dissimilarity matrices computed as 1-correlation between multivoxel response patterns for each object pair extracted from brain regions shown in Figure 6C.
Discussion
In the present fMRI study, we tested whether action-related object properties shape the functional organization of high-level visual cortex. By measuring responses to 22 different object types, we found that hand- and body-selective regions in left LOTC responded preferentially to objects that are used by the body as action effectors. The preference for object effectors in LOTC-hand and LOTC-body could not be explained by several other action-related object properties. First, LOTC-hand and LOTC-body did not respond selectively to all objects that prime hand movements, such as everyday act-on objects (e.g., door handle) or musical instruments (e.g., guitar). Behavioral ratings showed that these objects equally primed hand movements but, unlike tools and other act-with objects, were not perceived as being controlled by and extending the body. Second, the graspability of objects only weakly modulated LOTC-hand and LOTC-body: responses to graspable objects were not selective (relative to chairs), and were well below the responses observed for tools despite being rated as equally graspable. Finally, the nonselective response to musical instruments in LOTC-hand and LOTC-body, rated highest on the hand-action centrality dimension, suggests that these regions do not represent all objects for which hand action is a central characteristic, but only those objects that are used by the body as action effectors. These results were confirmed by regression analysis, showing that fMRI responses in LOTC-hand and LOTC-body reflected the degree to which objects were perceived as being controlled by and extending the body, more so than other action- or hand-related object dimensions. Interestingly, multivoxel pattern analysis within a region encompassing both hand- and body-selective voxels attributed voxelwise variations in the response to object effectors to voxelwise variations in hand-evoked activity rather than body-evoked activity, linking representations of object effectors to hand representations rather than whole-body representations. Whole-brain univariate group analyses confirmed the results of the ROI analyses, showing an overlap between body effectors and object effectors in left LOTC. Finally, whole-brain representational similarity analysis revealed a cluster in left LOTC in which multivoxel response patterns were relatively similar between objects that were rated as relatively similar on the body extension dimension, relative to other action-related or hand-related object dimensions. Together, these results indicate that the functional organization of high-level visual cortex partly reflects body–object interactions, with object effectors represented together with the body parts they extend.
Whole-brain univariate analyses showed that overlapping responses to body effectors and object effectors were also present in parietal cortex. Previous work has shown that body-centered visual receptive fields of bimodal (visual–somatosensory) neurons in parietal cortex were altered after tool use, with receptive fields extended to incorporate the tool (Iriki et al., 1996; but for a critical view, see Holmes, 2012). These neurons thus represented the tool as if it had become part of the body. This has been interpreted as reflecting the plasticity of the “body schema” (Maravita and Iriki, 2004). The concept of body schema refers to a continuously updated representation of our limbs in space generated by the integration of multiple sensory inputs, such as visual and somatosensory input (Head and Holmes, 1911; Iriki et al., 1996). During tool use, aspects of the multisensory representation of the hand can be extended to incorporate the tool (Iriki et al., 1996; Tsukamoto, 2000; Maravita and Iriki, 2004; Arbib et al., 2009; Cardinali et al., 2009). Our findings of representational overlap between body effectors and object effectors in LOTC and parietal cortex suggest that these regions may be involved in linking body and tool representations. Future work is needed to test whether LOTC contributes to the incorporation of tools into the body schema, whether this is body-part specific (Orlov et al., 2010), and how this region interacts with parietal regions previously implicated in this process.
When using an object to act with, the object replaces the hand as the end effector of the action. Successful use of the object effector requires knowledge about the hand/arm movements typically associated with it. This includes knowledge about how hand/arm movements are translated through the object, how much force needs to be applied to move the object when acting with it, and how far the action-relevant part of the object physically extends the body. Neuropsychological studies have associated left LOTC lesions with impairments in retrieving information about body parts and tools/utensils (Tranel et al., 1997, 2003; Damasio et al., 2004; Schwoebel and Coslett, 2005; Moro et al., 2008). For example, Tranel et al. (1997, 2003) reported evidence from a group of patients who, following lesions to left LOTC, were unable to retrieve conceptual knowledge about tools as well as hand actions. Furthermore, lesion and transcranial magnetic stimulation (TMS) studies have shown that LOTC is causally involved in body part discrimination, including hands (Urgesi et al., 2004; Moro et al., 2008; Pitcher et al., 2009). Future experiments could apply TMS over left LOTC to test whether this disrupts processing of both body and object effectors, as predicted by the current findings.
In a previous study (Bracci et al., 2012), we found a close overlap between left LOTC regions selective to hands and tools. These regions were located at Talairach coordinates (both regions: −46, −68, −2) that match the coordinates of the current hand-selective region (average coordinates across four experiments: −48.8, −69.6, −0.6). The results of the present study shed new light on the hand/tool overlap in this region by revealing the specific action-related properties of tools that account for this overlap. Our results may also be relevant to understanding tool representations in a region that is often labeled the posterior middle temporal gyrus (pMTG; Martin, 2007). Previous studies have reported activation to tools in the left pMTG for various contrasts (Lewis, 2006), as follows: viewing tools relative to humans (Beauchamp et al., 2002), mammals (Chao et al., 1999; Beauchamp et al., 2003; Downing et al., 2006; Bracci et al., 2012), inanimate objects (Valyear and Culham, 2010; Bracci et al., 2012) or graspable objects (Valyear et al., 2007), naming tools (Damasio et al., 1996; Martin et al., 1996; Chao et al., 1999; Tranel et al., 2005), hearing tool names (Peelen et al., 2013), and tool-use pantomiming and imagery (Choi et al., 2001; Kellenbach et al., 2003; Johnson-Frey et al., 2005). Some of these studies report nearly identical coordinates to the current hand-selective region (Beauchamp et al., 2002; Bracci et al., 2012), while others report activity more anteriorly (Chao et al., 1999; Peelen et al., 2013). It is thus unclear whether these studies all refer to the same region, or whether there are multiple tool-selective regions in left LOTC. Possibly, tasks that involve more semantic aspects of tools (e.g., naming tools or learning facts about tools) could shift activity anteriorly (Simmons et al., 2010; Simmons and Martin, 2012). More research is needed to investigate how these nearby regions in left posterior temporal cortex functionally and anatomically relate to each other, and to what extent responses in these regions reflect the degree to which objects are perceived as being controlled by and extending the body.
The present findings shed new light on the functional organization of high-order visual cortex. In particular, the overlap between responses to body effectors and object effectors in left LOTC is not easily explained by traditional accounts of the functional organization of high-level visual cortex, such as accounts related to object shape, retinal eccentricity, object size, object motion, or object domain (Haxby et al., 2001; Levy et al., 2001; Beauchamp et al., 2002; Downing et al., 2006; Op de Beeck et al., 2008; Konkle and Oliva, 2012). Instead, overlapping representations for body and object effectors indicate that nonvisual action-related object properties have a substantial influence on the organization of visual cortex. This is in line with a growing number of recent studies showing that the functional organization of high-level visual cortex is surprisingly independent of visual experience (Pietrini et al., 2004; Amedi et al., 2007; Lacey et al., 2007; Mahon and Caramazza, 2009; Mahon et al., 2009; Reich et al., 2011; He et al., 2013; Peelen et al., 2013).
For example, a recent study showed that an area in left fusiform gyrus that responds selectively when sighted participants read words (“visual word form area”) is also selectively activated when congenitally blind participants read braille (Reich et al., 2011). Together with the present results, this suggests that what is considered the high-level “visual” cortex may be much less visual than is generally assumed (but see Pascual-Leone and Hamilton, 2001). Rather than being fully driven by bottom-up visual input, the organization of visual cortex may additionally be driven by connectivity patterns to downstream regions (Mahon and Caramazza, 2011; Bracci et al., 2012), such as the connectivity between LOTC and a left lateralized frontoparietal network implicated in action perception and execution (Bracci et al., 2012; Simmons and Martin, 2012).
In summary, our results reveal a clustering of representations of body effectors and object effectors in left LOTC. This finding indicates that the functional organization of high-level visual cortex partly reflects how objects physically and functionally relate to the body.
Footnotes
This work was supported by the Fondazione Cassa di Risparmio di Trento e Rovereto. We thank Talia Konkle, Nick Oosterhof, and Paul Downing for helpful comments on the manuscript; and Nick Oosterhof for help with data analysis.
- Correspondence should be addressed to either Stefania Bracci or Marius V. Peelen, Center for Mind/Brain Sciences, University of Trento, Corso Bettini 31, TN 38068 Rovereto, Italy, stefania.bracci{at}unitn.it or marius.peelen{at}unitn.it
