Abstract
The exquisite ability of primates to grasp and manipulate objects relies on the transformation of visual information into motor commands. To this end, the visual system extracts object affordances that can be used to program and execute the appropriate grip. The macaque anterior intraparietal (AIP) area has been implicated in the extraction of affordances for the purpose of grasping. Neurons in the AIP area respond during visually guided grasping and to the visual presentation of objects. A subset of AIP neurons is also activated by two-dimensional images of objects and even by outline contours defining the object shape, but it is unknown how AIP neurons actually represent object shape. In this study, we used a stimulus reduction approach to determine the minimum effective shape feature evoking AIP responses. AIP neurons responding to outline shapes also responded selectively to very small fragment stimuli measuring only 1–2°. This fragment selectivity could not be explained by differences in eye movements or simple orientation selectivity, but proved to be highly dependent on the relative position of the stimulus in the receptive field. Our findings challenge the current understanding of the AIP area as a critical stage in the dorsal stream for the extraction of object affordances.
Introduction
The primate dorsal visual stream is critical for the analysis of space and the visual guidance of actions (Ungerleider and Mishkin, 1982; Goodale and Milner, 1992). The transformation of intrinsic object properties into motor actions during object grasping relies on a well defined network of cortical areas located in parietal and ventral premotor cortex (Gallese et al., 1994; Fogassi et al., 2001; Rizzolatti and Luppino, 2001). The anterior intraparietal (AIP) area is one of the end-stage areas of the dorsal visual stream that is crucial for grasping. Anatomical (Borra et al., 2008; Gerbella et al., 2011), neurophysiological (Murata et al., 2000; Baumann et al., 2009; Srivastava et al., 2009; Theys et al., 2012b, 2013), and computational studies (Fagg and Arbib, 1998) support the hypothesis that the AIP area extracts visual object information and relays this information to neurons in the ventral premotor cortex (area F5), which then activate the primary motor cortex.
In his influential theory of visual perception, Gibson (1979) proposed that the visual system automatically extracts the constellation of potential actions (termed “affordances”) that we can perform on an object whose image impinges on the retina. Since many AIP neurons respond selectively to objects during passive fixation and grasping, it is widely believed that the extraction of object affordances for grasping purposes occurs in the AIP area (Fagg and Arbib, 1998; Rizzolatti and Luppino, 2001; Cisek and Kalaska, 2010).
Visual-dominant AIP neurons respond during visually guided grasping and during object fixation, but not during memory-guided grasping in the dark (Murata et al., 2000). AIP neurons also encode the three-dimensional (3D) profile of curved surfaces as defined by binocular disparity (Srivastava et al., 2009). Moreover, neurons in the posterior AIP (pAIP) area can be selective for two-dimensional (2D) images of objects, a selectivity that is primarily based on the contours of the object shape (Romero et al., 2012). At least for this subpopulation of AIP neurons, the presence of binocular disparity is not necessary for evoking selective responses (Romero et al., 2013). However, no study has investigated which aspects of object contour are encoded by AIP neurons.
The analysis of object shape for purposes of object recognition relies on the ventral visual stream (Logothetis and Sheinberg, 1996). Tanaka and colleagues (1991, 1993) used a stimulus reduction approach to determine the critical features (shape, texture, color, or combinations thereof) eliciting responses in neurons in the inferior temporal cortex (ITC). The critical features for neurons in the anterior ITC were generally moderately complex, and a substantial fraction of these neurons (42%) did not respond to reduced versions of the original 3D object, such that a critical feature could not be determined. Here, using a similar stimulus reduction approach, we identified the minimum effective shape features (MESF) for AIP neurons. Remarkably, AIP neurons responded to even the smallest shape fragments in the test, which measured a mere 1°. The neural selectivity for fragments was highly dependent on position. These findings challenge our current understanding of visuomotor transformations in the dorsal stream and the role of the AIP area in extracting object affordances.
Materials and Methods
Surgical and recording procedures.
The experimental protocol was similar to that previously described (Romero et al., 2012). All technical procedures were performed in accordance with the National Institute of Health's Guide for the Care and Use of Laboratory Animals, and approved by the Ethical Committee at the Katholieke Universiteit Leuven Medical School. Two male rhesus monkeys (Macaca mulatta; monkey H, 7 kg; monkey C, 6 kg) were trained to sit in a primate chair. Next, a head post (Crist Instruments) was affixed to the skull with ceramic screws and dental acrylic. For this and all other surgical procedures, monkeys were kept under isoflurane anesthesia (1%) and strict aseptic conditions. Intensive training in passive fixation began after 6 weeks of recovery. Once a monkey had acquired an adequate level of performance, a craniotomy was made, guided by anatomical magnetic resonance imaging (MRI; Horsley-Clark coordinates: 2P, 12L) over the left hemisphere (Fig. 1A). To verify the recording positions, glass capillaries were filled with a 2% copper sulfate solution and inserted into a recording grid at five different locations—for reference—while structural MRI was performed (slice thickness, 0.6 mm). During the experiment, we recorded single-unit activity with standard tungsten microelectrodes (impedance, 1 MΩ at 1 kHz; FHC) inserted through the dura by means of a 23 gauge stainless-steel guide tube and a hydraulic microdrive (FHC). The neural activity was amplified and filtered between 300 and 5000 Hz. Spike discrimination was performed online using a dual time window discriminator, and displayed with LabView and custom-built software. The position of the right eye was continuously monitored with an infrared-based camera system (Eye Link II, SR Research) sampling pupil position at 500 Hz. The stimuli were presented on a monitor (resolution, 1280 × 1024 pixels; Vision Research Graphics) equipped with a fast-decay P46-phosphor operating at 120 Hz at a viewing distance of 86 cm (56 pixels per degree) on a dark background (luminance 8 cd/m2). A photocell attached to the stimulus display in the lower right corner detected the onset of a white square (covered with black tape to obscure it from view) in the first video frame containing the stimulus. All recorded signals (eye position, neural activity, and photocell pulses) were digitized and processed at 20 kHz with a digital signal processor (C6000 series, Texas Instruments). The active–silent transitions observed between the medial bank, sulcus, and, finally, the lateral bank of the intraparietal sulcus (IPS) were useful for the identification of the recording area. All neurons were recorded in the pAIP area (Fig. 1A) within 7 mm of the tip of the IPS, which is consistent with previous studies in AIP areas (Baumann et al., 2009). In monkey H, we verified that no responses could be recorded during visually guided saccades (Romero et al., 2012). In monkey C, we used an electrode inserted into one of the grid positions where most neurons were recorded to verify the location of the recording area by anatomical MRI (data not shown). Furthermore, also in this monkey, we recorded neurons that responded during grasping, typical of the AIP area.
Stimuli and tests.
In the passive fixation task, each trial began with the onset of a small square in the center of the screen (fixation point: 0.2 × 0.2°). If the position of the eye remained within an electronically defined 1° square window around the fixation point for ≥300 ms, a visual stimulus was presented. The fixation time depended on the specific requirements of the task (preonset fixation, 300/500 ms; postonset fixation, 300/800 ms; see below). Trials were considered to be correct when the monkey held fixation until the offset of the stimulus, receiving a drop of water as reward.
The search stimulus set consisted of 21 2D area-equalized static images of natural and artificial objects, including faces, hands, fruits, branches, and several artificial graspable objects, as illustrated for one stimulus (the image of a banana) on the left panel of Figure 1B. These stimuli will be termed “surfaces” because they contained all texture, perspective, and shading information. Because of the large variation in surface area in our original set, we split the images into two categories (round vs elongated stimuli; Fig. 1C), and area-equalized the stimuli according to the category to which they belonged (surface area for round stimuli, 34.4°2; diameter, ∼6.6°; surface area for elongated stimuli, 25.7°2). For a second stimulus set (contour stimulus set), we created gray contour versions of the same images by generating silhouette and outline stimuli (Photoshop 8.0, Adobe; Fig. 1B, left). The surface area included by the contours was identical within the stimulus category (round or elongated), but, because of the variation in the shape of the contour, the total number of bright pixels varied by ∼±3.5% between stimuli (Romero et al., 2012). The luminance in a 6.5 × 6.5° window around the center of the stimuli measured 1.4 cd/m2 for the surface stimuli, 5 cd/m2 for the silhouettes, and 0.4 cd/m2 for the outlines. Finally, a third stimulus set (fragments stimulus set) consisted of the intact (full) outline stimuli, and the isolated fragments obtained by the progressive segmentation of these outline images into 4, 8, and 16 different elements (Fig. 1B, right). The fragmentation was done manually in Photoshop along the main axes of the outline, as illustrated in Figure 1C. Because the surface area enclosed by the intact outline was equalized for each stimulus category (round and elongated), lengths of the intact outlines varied between stimuli (between ∼15 and 20°). Therefore, the size of the fragments ranged from 3.8 to 5.1° for the 4-fragment stimuli, 2.0 to 2.5° for the 8-fragment stimuli, and 1.0 to 1.3° for the 16-fragment stimuli. During the experiment, all stimuli were randomly interleaved and presented on a black background at a viewing distance of 86 cm.
For the majority of the neurons (70 of 82, 85%), we searched for responses during the presentation of the search stimuli (i.e., images of objects) at the fixation point (passive fixation search test). If the neuron responded to ≥1 of these stimuli, we selected the object image evoking the strongest response (termed the “preferred image”) and chose a second object image to which the neuron responded weakly (termed the “nonpreferred image”) for subsequent testing. Both images were members of the same stimulus category (round or elongated), so that their surface areas were equal. In the remaining neurons (12 of 82, 15%), we searched for responses using a visually guided grasping (VGG) test. In the VGG test, a single object (bar, the same for all neurons; see Fig. 3A) attached to a plate was placed in the monkey's view. The monkey had to rest his right hand on a sensing device in complete darkness for a variable time (intertrial interval, 3000–5000 ms). Next, a light inside the object was illuminated and the monkey had to fixate the object (keeping its gaze inside a ±2.5° fixation window). If he maintained fixation for 500 ms, then an audible go signal was given for initiating the grasping movement, which consisted of quitting the resting position and reaching, grasping, and pulling the object on the plate (holding time, 500–900 ms). We did not assess the selectivity for different 3D objects. For each responsive neuron in the VGG task, we then ran the passive fixation search test (images of objects presented at the fixation point on the display), which included four additional stimuli: the image of the object on the plate and its corresponding outline and, as a control, the image of the same object, rotated 90°, and its corresponding outline (see Fig. 3A).
After the search tests, we mapped the receptive field (RF; area mapped: 14 × 10° around the fovea) of each responsive neuron using the preferred image (surfaces) and repeated the passive fixation search test at the RF center. All the following tests were always performed by placing the selected stimuli at the RF center. As in our previous studies (Romero et al., 2012, 2013), neurons showing image-selective responses were next studied in the shape test, which included presentations of the preferred and nonpreferred object images in three different versions: surfaces (stimulus set of the passive fixation search test), silhouettes, and outlines (Fig. 1B, example stimuli). We presented silhouette stimuli to verify the shape selectivity in some neurons responding weakly to the outline stimuli. For both the search (passive fixation search test) and the shape tests, the monkey had to maintain gaze within a 1° fixation window for at least 500 ms before the visual stimulus was presented in the center of the display for 800 ms. The eye position was continuously monitored until the end of each trial.
To identify the MESF for AIP neurons, the standard reduction test was performed, where we presented the preferred and nonpreferred intact outlines together with their corresponding 4, 8, and 16 fragments at the RF center. Due to the large number of conditions in this test, the fixation time was reduced to 300 ms for both the prestimulus and poststimulus onset periods and the median number of trials per condition was seven (minimum number, five). In the standard reduction test, the fragments were presented at the location they occupied in the full contour stimulus. To evaluate whether the observed AIP responses to fragments were influenced by small changes in stimulus position, we ran a center reduction test, in which the same fragments used in the standard reduction test were now presented at the center of mass of the full shape outline.
The fifth and final test was the orientation test, created to assess orientation selectivity in the AIP area. In this test, straight lines (1.3°; similar in size to the smallest fragments) or curved line segments (Fig. 1D) were presented over the RF (center of mass of the intact contour) in eight different orientations (22.5° apart). All curved line segments had the same curvature (radius of curvature, 0.38 cm). The fixation time was 300 ms in both the prestimulus and poststimulus onset periods. Because of the large number of tests performed on each neuron, we did not explore whether fragment selectivity was anatomically clustered in the AIP area.
Data analysis.
All data analysis was performed in Matlab (Mathworks). For each trial in the search and the shape tests, the baseline firing rate was calculated from the mean activity recorded in the 400 ms interval preceding the stimulus onset. Net neural responses were then calculated by subtracting the baseline from the mean activity observed between 50 and 450 ms after the onset (hence negative firing rates were observed in some cases). Next, we calculated ANOVAs on the net responses to the preferred and nonpreferred surfaces, silhouettes, and outlines to assess the significance of shape selectivity for each of these stimulus types. For the reduction tests, we compared the activity in the 300 ms interval before stimulus onset to the activity between 50 and 300 ms after the onset, and computed ANOVAs to assess the significance of the fragment selectivity at each fragment level (4, 8, and 16 fragments).
For the standard reduction test, we also assessed the significance of the responses using t tests of the activity after the stimulus onset (50–350 ms) compared with the average activity in the interval before the stimulus onset. We designated the MESF as the lowest level of fragmentation (4-fragment, 8-fragment, or 16-fragment stimuli) for which we observed a response that was ≥70% of the response to the full contour (outline) and was not significantly smaller than that response (one-sided t test, p > 0.05). These same criteria were used in the stimulus reduction studies of Tanaka and colleagues in ITC (Tanaka et al., 1991). Population response latencies were computed as the first of three consecutive time bins after stimulus onset (bin size, 10 ms) showing significantly increased responses compared with baseline (last preonset bin; t test, p < 0.05).
To quantify fragment selectivity in the standard and in the center reduction tests, we first ranked the responses of each AIP neuron tested, ranking them separately for the standard reduction test and the center reduction test, and plotted the average response across neurons as a function of the stimulus rank. To determine whether the fragment preference was similar in the two reduction tests we ranked the stimuli based on the responses in the standard reduction test and then plotted the average responses to the same stimuli in the center reduction test.
Results
We recorded the activity of 82 single AIP neurons (N = 31 in monkey H; N = 51 in monkey C) that responded selectively to images of objects during passive fixation. Because the results were highly similar for the two monkeys, the data from these animals were combined. The recording positions in the pAIP area were verified by means of structural MRI (0.6 mm resolution; Fig. 1A). For most neurons in this study (N = 70; N = 31 in monkey H; N = 39 in monkey C), we used a passive fixation search test, in which 21 images of objects were presented in the center of a display (at the fixation point). However, in a subset of neurons (N = 12), we used a delayed VGG task to search for responses while recording in the same grid positions as in the passive fixation search test. Neurons active during grasping were then tested in the passive fixation search test. All neurons responsive during grasping were also responsive during the passive fixation search test. The strong responses we observed in the grasping task confirmed that the recordings were performed in the AIP area.
Recording sites and stimuli. A, Lateral and coronal views of the macaque brain obtained by anatomical MRI indicating the recording position in the AIP area (arrows). B, Example image used in the search test and its corresponding contour versions (outlines, left). These panels illustrate the different conditions presented in the shape (left) and reduction tests (right). Each outline was divided into 4, 8, and 16 fragments along the main axes of the shape indicated by the dotted lines on the right. C, Outline stimuli used in the shape test and in the standard reduction test. The red dashed lines indicate the main axis along which the intact contour was fragmented. D, Stimuli used in the orientation test. Upper row, Straight line segments. Lower row, Curved line segments. We used 8 different orientations for the straight line segments and 16 different orientations for the curved line segments.
Responses of AIP neurons to contour fragments
The example neuron in Figure 2A responded to the intact outline of the tangerine (on average, 55 spikes/s) but not to the outline of the hand [Fig. 2A, left, peristimulus time histograms (PSTHs) and RF map]. However, this neuron responded even more strongly to one of the four-fragment stimuli (the leaf of the tangerine, Fig. 2A). Surprisingly, several of the 8-fragment and 16-fragment stimuli also evoked robust responses that reached >70% of the response to the intact outline. This example neuron showed significant selectivity at each of the levels of fragmentation (ANOVA on the net responses: p = 0.003, p = 0.012, and p = 0.0004 for the 4-fragment, the 8-fragment, and the 16-fragment levels, respectively). Following Tanaka and colleagues (1991), we defined the MESF for an AIP neuron as the smallest fragment that evoked a response that was ≥70% of the response to the intact outline and not statistically smaller (one-sided t test p < 0.05) than the response to the intact outline. In the case of the example neuron in Figure 2A, the simplest stimulus that activated the neuron (MESF) was a small, curved line segment measuring 1° (lower row, orange box). Given the strong selectivity for small line segments observed in this cell, we also tested its orientation tuning by using a set of straight bars and curves but found no significant orientation selectivity (ANOVA: p = 0.125 and p = 0.148 for straight lines and for curved lines, respectively; data not shown). A second finding was that the effectiveness of the 8-fragment and 16-fragment stimuli was apparently related to the effectiveness of the 4-fragment stimuli from which they were derived. For example, the leaf of the tangerine elicited strong responses as a 4-fragment stimulus, but the constituent 8-fragment and 16-fragment stimuli also evoked strong responses individually (Fig. 2A, right). To quantify the relationship between the different shape levels, we calculated the correlation between the 8-fragment responses and the 16-fragment responses. Because each 8-fragment stimulus was decomposed into two 16-fragment stimuli, we always considered only the highest response within a pair of 16-fragment stimuli derived from each 8-fragment stimulus. For the example neuron in Figure 2A, the correlation between the 8-fragment and the 16-fragment shape levels equaled 0.91 (p = 0.002). Shape fragments derived from the nonpreferred outline (Fig. 2A, bottom) did not evoke any robust responses (all responses were <30% of the response to the preferred intact outline).
Standard reduction test, example neurons. A, PSTH of a neuron responding selectively to the outline image of a tangerine (preferred image; left, top row) but not to the outline of a human hand (nonpreferred image; left, bottom row). Above the PSTH is a color map showing the responses in the receptive field test of the example neuron. The center of the display is located at 0,0. The right panel is a color tree plot showing the normalized net responses (firing rate minus baseline activity) of the same neuron to all stimuli in the standard reduction test (4-fragment stimuli, first row; 8-fragment stimuli, second row; 16-fragment stimuli, third row). We normalized these responses to the highest response in the test. B, PSTH of a second example neuron responding selectively to the intact outline of a key (preferred image; left, top row) compared with the outline of a monkey hand (nonpreferred image; left, bottom row). In contrast to the previous example neuron, this neuron responded to fragments from both the preferred and nonpreferred outlines. Same conventions as in A.
The second example neuron (Fig. 2B) was also highly selective for the intact outline version of the key (left, PSTHs and RF map) and responded strongly to several of the 4-fragment, 8-fragment, and 16-fragment stimuli of the preferred outline. As in the previous case, the MESF was a small curved line segment. However, in contrast to the previous example neuron, this cell also responded to several fragments derived from the nonpreferred outline stimulus (image of the hand; Fig. 2B, bottom), reaching 90% of the maximal response in the test for the best eight-fragment stimulus of the nonpreferred shape. Thus, even ineffective outline stimuli can contain shape elements that are highly effective for the neuron under study. Alternatively, our reduction method introduced new shape features that fortuitously activated the neuron. As in the first example neuron, simple orientation tuning could not account for the observed responses in the reduction test (ANOVA: p = 0.326 and p = 0.168 for straight lines and curved lines, respectively; data not shown). Overall, both example neurons in Figure 2 demonstrate that the minimum effective shape features for AIP neurons can consist of exceedingly short line segments.
The example neurons in Figure 2 were recorded during the passive fixation of object images on a display. It is therefore possible that these example neurons constitute a special subclass of AIP neurons distinct from the well known grasping-related AIP neurons investigated in other studies (Taira et al., 1990; Sakata et al., 1997; Murata et al., 2000; Baumann et al., 2009). Therefore, we also used a VGG task to search for responsive neurons in the same grid positions as the neurons recorded with the passive fixation search test (monkey C only). Figure 3B shows the activity of an example neuron during visually guided grasping. This neuron fired strongly after the onset of the light inside the object (Fig. 3B, left) and during the grasping movement (Fig. 3B, right; activity aligned on movement onset). In the passive fixation search test, the same neuron proved to be highly selective for images of objects (ANOVA on the net responses: p = 0.021; Fig. 3C), responding to the image of the bar (the object in the VGG; 22 spikes/s) and maximally to the image of the spoon (26 spikes/s). This example neuron also responded to the outline of the graspable object (24 spikes/s; Fig. 3D), reacted moderately to the 4-fragment and 8-fragment stimuli (max response: 15 and 18 spikes/s, respectively), and most strongly to two of the 16-fragment stimuli (23 and 19 spikes/s, respectively; Fig. 3D, lower right). Thus even AIP neurons responding to the object presentation in a grasping task can be activated by very small line segments measuring a mere 1°. All neurons tested during grasping were also responsive in the passive fixation search test, and 83% of these neurons (10 of 12) showed significant selectivity to particular images of objects (ANOVA on the net responses: p < 0.05). On average, these 12 AIP neurons did not respond more to the 3D object than to 2D images of objects (t test comparing the responses to the 3D object to the response to the best 2D image: p = 0.0580; 7 of 12 neurons responded more to the 3D object). Since this subset of AIP neurons recorded during object grasping behaved in a manner very similar to the population of neurons recorded using the passive fixation search test (images on a display, N = 70), the data from both groups were combined for the rest of the article.
VGG test. A, Surface and outline images of the object in the VGG test (left) and the same object rotated by 90° (right). B, Perievent time histogram showing one of the neurons tested in the VGG task. This neuron responded to the visual presentation of the object to be grasped (left; activity aligned to the light onset in the object), but also maintained a sustained activity before and during the whole reach-to-grasp movement (right; activity aligned to the onset of the movement). C, Responses in the passive fixation search test for the neuron in B. As for the majority of our grasping neurons, this cell also showed selective visual responses to 2D images, including those of the object to be grasped. D, Same example neuron in the standard reduction test, illustrating the responses to the outline of the object to be grasped. Same conventions as in Figure 2.
Although our search tests merely contained images of objects (passive fixation search test) or real objects (VGG test), we rarely encountered AIP neurons that did not respond to any of the fragments in the standard reduction test (derived from either the preferred or the nonpreferred outline): 90% of our AIP neurons (74 of 82) responded significantly (t test poststimulus vs prestimulus onset epochs; p < 0.05) to at least one of the 16-fragment stimuli, compared with 80% (66 of 82) and 84% (69 of 82) for the 4-fragment and 8-fragment stimuli, respectively.
For each neuron, we designated the MESF as the smallest fragment that evoked a response that was ≥70% of the response to the intact outline and not statistically smaller than the response to the intact contour (one-sided t test; p > 0.05; Fig. 4A). For just more than half of the neurons (42 of 82, 51%), the MESF was one of the 16-fragment stimuli, while for 16 other neurons (16 of 82, 20%), the MESF was one of the 8-fragment stimuli (Fig. 4A). Thus a majority of the cells in our AIP population (58 of 82, 71%) were activated by remarkably small line segments measuring 1–2.5°, and only a small fraction of neurons (9 of 82, 11%) could be activated by ≥1 of the four-fragment stimuli but not by any of the smaller line fragments. The remaining 18% of these AIP neurons (15 of 82) did not respond to any of the fragments by >70% of the response to the intact outline.
Standard reduction test, population data. A, Pie chart illustrating the proportions of the MESF (see Materials and Methods) in our population of AIP neurons (N = 82). B, Average normalized population responses in the standard reduction test. We averaged the activity across all neurons showing significant selectivity (ANOVA: p < 0.05) for each level of fragmentation (intact outline, gray; 4-fragment stimuli, red; 8-fragment stimuli, blue; 16-fragment stimuli, green), normalized by dividing by the highest response, and calculated the population latency as the first of three consecutive bins higher than the baseline (equal to last bin before stimulus onset; t tests on 10 ms bins, p < 0.05). The dashed vertical lines indicate the time of stimulus onset (left) and the latency of the population response to the intact outlines (right).
Most neurons responding to fragments by >70% of the response to the intact outline were also significantly selective for these fragments as determined by ANOVA (p < 0.05): 45% (37 of 82), 40% (33 of 82), and 45% (37 of 82) of the total population of neurons was significantly selective for the 4-fragment, 8-fragment, and 16-fragment stimuli, respectively. In a small number of neurons, (7 of 82 for the 4-fragment stimuli, 4 of 82 for the 8-fragment stimuli, and 7 of 82 for the 16-fragment stimuli) we observed strong responses (>70% of the response to the full preferred contour) to ≥1 of the fragments derived from the nonpreferred contour but not to those derived from the preferred contour.
The average population response to the intact preferred outline and to the best 4-fragment, 8-fragment, and 16-fragment stimuli in the standard reduction test is illustrated in Figure 4B. For this plot, we averaged and normalized the activity across all neurons showing significant selectivity (ANOVA: p < 0.05) at each level of fragmentation (the population response using the MESF criterion yielded highly similar graphs). We defined the latency of the population response as the first of three consecutive bins significantly higher than the baseline (last bin before stimulus onset; t tests on 10 ms bins, p < 0.05). Consistent with previous studies (Srivastava et al., 2009; Romero et al., 2012; Theys et al., 2012a), the latency of the population response was remarkably short (50–60 ms for the intact outline and the 4-fragment stimuli, and 60–70 ms for the 8-fragment and the 16-fragment stimuli, possibly due to their low total luminance).
The exquisite selectivity for very small line segments of AIP neurons was also reflected in the average number of fragments that evoked significant responses: most neurons (63 of 82, 77%) responded to ≤3 16-fragment stimuli derived from the preferred contour, whereas a minority (7 of 82, 9%) responded to ≥6 fragments. The example neurons in Figures 2 and 3 also illustrate the wide variability observed in the correlation between the responses at the different fragment levels: on average, these correlations were not significantly higher than zero; the average correlation between the 4-fragment and the best 8-fragment responses was only −0.07 (first quartile, −0.58; third quartile, 0.38; p = 0.119), and the average correlation between the 8-fragment and the best 16-fragment responses was 0.04 (first quartile, 0.01; third quartile, 0.10; p = 0.1085). The absence of significant correlations between fragment levels indicates that the effectiveness of a shape fragment at a higher level could not easily be predicted from the responses to the lower-level fragments (except in a few neurons like the one illustrated in Fig. 2A).
We wanted to determine to what extent the responses to the intact outlines could be predicted from the responses to the 4-fragment, 8-fragment, and 16-fragment stimuli by calculating the correlation between the intact outline responses and the sum of the fragment responses. Figure 5 shows the average net response to the intact contour plotted against the sum of the responses to the individual fragments. Although the sum of the fragment responses was almost always larger than the response to the intact contour (data points above the unity line), we obtained significant (p < 0.001) but modest correlations for each level of fragmentation (ranging from 0.58 to 0.71). The correlation between the intact contour responses and the sum of the 16-fragment responses (r = 0.58; p = 1.492e-05) was not significantly smaller than the correlations between the intact contour responses and the sum of the 4-fragment and 8-fragment stimuli (r = 0.71; p = 1.147e-06 and r = 0.69; p = 5.764e-09 for 4-fragment and 8-fragment stimuli, respectively). In other words, similar correlations were found across all three levels of fragmentation (Fisher z test: z = −0.93; p = 0.176 for the comparison between the 16-fragment and 4-fragment stimuli and z = −0.76; p = 0.223 for the 16-fragment and 8-fragment stimuli). Hence, a simple linear combination of the fragment responses could only explain 34–50% of the variance in the intact contour responses, suggesting that nonlinear interactions between the fragments contributed to the AIP responses. Slightly higher correlations were observed between the responses to the best 4-fragment stimulus and the sum of the responses to the 8-fragment stimuli derived from that 4-fragment stimulus (r = 0.83, p = 2.179e-10), and between the response to the best 8-fragment stimulus and the sum of the responses to the 16-fragment stimuli derived from that 8-fragment stimulus (r = 0.77, p = 2.147e-08). Competitive (inhibitory) interactions between the fragments—possibly related to the substructure of the RF—may be responsible for stronger responses to 16-fragment stimuli compared with larger fragments.
Standard reduction test, scatter plots. The average net response to the intact contour (x-axis) is plotted against the sum of the average responses to the 4-fragment (top), 8-fragment (middle), and 16-fragment stimuli (bottom) for all neurons tested in the standard reduction test (n = 82). The dotted line in each plot indicates the unity line.
To investigate whether differences in eye movements could have influenced the responses in the standard reduction test, we analyzed the mean eye position and the variability of the eye position. We calculated the absolute differences in the mean and the SD of the eye position signal between the prestimulus epoch (−300–0 ms before stimulus onset) and the poststimulus epoch (0–300 ms after stimulus onset). Both horizontal and vertical deviations of the eye position were very small, averaging <0.1° for both monkeys (99th percentile: 0.14 and 0.08° for the horizontal and vertical eye position differences of monkey C, and 0.13 and 0.18° for those of monkey H). Moreover, the increases in the variability of the eye position signal after the stimulus onset were almost negligible (on average 0.02 and 0.005° for the horizontal and vertical eye position of monkey C, and 0.006 and 0.02° for those of monkey H). Only one neuron showed differences in the vertical eye position between stimuli, as evidenced by a significant interaction between the factors prestimulus versus poststimulus onset and condition (two-way ANOVA: p = 0.001). In a similar vein, the range of eye positions during stimulus presentation (i.e., range of the absolute differences between the eye position on any given trial and the mean eye position in that test) was very small: averaged across all neurons of the standard reduction test the 95th percentile of the horizontal and vertical position range was 0.24 and 0.37°, respectively. Therefore, our results were most likely not influenced by any differences in eye movements between stimuli.
Overall, small line fragments proved to be surprisingly effective for many AIP neurons, even for those neurons responding to real objects during grasping. None of the AIP neurons we recorded was unresponsive to all the line fragments in our test.
If AIP neurons were to signal grasping affordances or potential grasping points on objects, we would expect that, for a given outline, similar shaped fragments would be preferred by different AIP neurons. We investigated this possibility by looking at the responses to an object outline and its fragments (standard reduction test) tested on different neurons. The image of the spectacles was tested on 13 AIP neurons; Figure 6 shows the responses to the 16-fragment stimuli of this image. Similar to the entire AIP population tested, for 8 of 13 neurons (62%) the MESF was one of the 16-fragment stimuli. In total, 11 neurons responded significantly (t test p < 0.05) to ≥1 of the contour fragments, and all of these neurons were selective for these line fragments (ANOVA: p < 0.05). For each of the fragments, we calculated the proportion of responsive neurons. No significant differences were observed between the lowest (27%) and the highest proportions of responsive neurons (64%, z test). Although the average net response of these 11 neurons to the 16-fragment stimuli varied between 12.6 and −7.3 spikes/s (Fig. 6, right), these differences were not significant (ANOVA: F = 0.6, p = 0.209) due to the large response variability between neurons. Similar results were obtained for the images of the Kong (tested on eight neurons), the scissors (seven neurons), and the tangerine (six neurons). Thus across the population of AIP neurons, no preference for any particular line fragments was observed.
Selectivity for 16-fragment stimuli across neurons. Each row illustrates the normalized net responses of an AIP neuron to the 16-fragment stimuli derived from the outline of the spectacles, which was tested in 13 neurons. The right panel shows the 16-fragment stimuli ranked according to the average response of these 13 AIP neurons (indicated on the right). Although 11 neurons responded significantly (t test p < 0.05) to ≥1 of the 16-fragment stimuli, no systematic preference for particular fragments was observed in our AIP population.
Position dependency of fragment responses in the AIP area
In the standard reduction test, we always presented the line fragments at the position they occupied in the intact outline in the center of the RF (Fig. 7A). As a consequence, the different fragments were not presented at exactly the same position in the visual field. To assess the effect of these position differences, we also presented all fragments at the same position, which corresponded to the center of mass of the intact outline (center reduction test; Fig. 7A), in a subset of neurons (N = 37). It is important to emphasize that, in both types of reduction tests, the stimuli were presented at the center of the RF (which averaged 10°2), and that for most contour stimuli, the position difference between the standard reduction test and the center reduction test measured only 1.7° (range, 1.2–2.5°). The example neuron in Figure 7A responded to the intact outline (derived from the image of the plant) and to the 4-fragment, 8-fragment, and 16-fragment stimuli in the standard reduction test (Fig. 7A, left; ANOVA: p = 0.026). In the center reduction test, the neuron also responded to several 16-fragment stimuli (Fig. 7A, right; ANOVA: p = 0.039), but the selectivity was radically altered (compare, for example, the response to the sixth fragment from the left in both tests). The same phenomenon was present for the fragments derived from the nonpreferred contour: the neuron was selective for the 16-fragment stimuli in both tests (ANOVA: p < 0.05), but the neural preference was entirely different. This remarkable change in stimulus selectivity could not be explained in terms of simple spatial selectivity. To illustrate the effect of stimulus position on fragment selectivity in this example neuron, we diagrammed the responses to the individual fragments on the original intact outline, i.e., at the position where the fragments appeared in the standard reduction test (Fig. 7B, left). The neuron gave the strongest response to the 4-fragment stimulus located at the bottom left side of the preferred contour (Fig. 7B, top), but the best 16-fragment response was recorded for the fragment located at the top right of the contour (Fig. 7B, bottom row). Conversely, for the nonpreferred contour, the best 4-fragment stimulus was located at the bottom right side of the contour (Fig. 7B, top), but the best 16-fragment stimulus appeared on the top left side of the contour (Fig. 7B, bottom).
Standard reduction test and center reduction test, example neuron. A, Tree plot illustrating the normalized net responses to the preferred (top) and nonpreferred outlines (bottom) in the standard reduction test (stimuli presented at the location occupied in the intact outline; left, top), and in the center reduction test (stimuli presented at the center of mass of the intact outline; right, top). In the top row, the difference in stimulus position is schematically illustrated for one of the fragments. The X indicates the center of mass of the outline; the red point is the fixation point. B, Fragment responses of the neuron in A, plotting the activity evoked by all tested fragments of the preferred (left) and nonpreferred (right) outline (color code indicates the normalized net response at each fragmentation level) at their position on the outline.
To quantify fragment selectivity in the two reduction tests, we first ranked the responses of each AIP neuron tested, separately for the standard reduction test and the center reduction test, and plotted the average response across neurons (N = 37) as a function of the stimulus rank (Fig. 8A–C). Clearly, the selectivity was equally strong in both types of reduction tests, since the average net response varied from >20 spikes/s (for the best fragment) to 0 spikes/s (for the least preferred fragment). The half-maximum response was observed for the fourth (8-fragment stimuli) and the eighth (16-fragment stimuli) stimulus rank in both the standard and the center reduction tests. These results indicate that both in the standard and in the center reduction tests, the AIP population showed strong selectivity for the 4-fragment, 8-fragment, and 16-fragment stimuli.
Standard reduction test and center reduction test, population analysis. A–C, Average ranked net responses (mean ± SEM) in the standard reduction test (cyan) and in the center reduction test (orange) for all neurons tested in both tests (N = 37). In these plots, the ranking was performed independently in the two tests. D–F, Average responses in standard reduction test (cyan) and center reduction test (orange), ranked according to the responses in the standard reduction test.
However, ranking the stimuli based on the responses in the standard reduction test and then plotting the average responses to the same stimuli in the center reduction test (Fig. 8D–F), revealed that the neural preference (i.e., the stimulus eliciting the strongest response) differed markedly in the two tests. The slopes of the linear fit to the center reduction test responses differed significantly from the slopes obtained for the standard reduction test for the 4-fragment (slope, −6.7 ± 1.01; R2 = 0.79, p = 0.112), the 8-fragment (slope, −2.8 ± 0.2; R2 = 0.61, p = 0.023), and the 16-fragment stimuli (slope, −1.3 ± 0.05; R2 = 0.12, p = 0.314). We compared the slopes of the linear fits to the data of the center reduction test to the slopes obtained after randomly shuffling the responses between conditions (mean of 1000 permutations); for all levels of fragmentation, the slopes of the center reduction test differed significantly from those of the random permutations (slope, 0.041, −0.013, and −0.009 for the 4-fragment, 8-fragment, and 16-fragment stimuli), indicating that the preference was partially (albeit very weakly in the case of the 16-fragment stimuli) preserved. Furthermore, the correlations between the fragment responses in the two reduction tests were invariably small (average r = 0.09 ± 0.02, r = 0.17 ± 0.03, and r = 0.07 ± 0.01 for the 4-fragment, the 8-fragment, and the 16-fragment stimuli, respectively). These results indicate not only that AIP neurons encode small line fragments (instead of shape contours), but also that the fragment selectivity is not independent of the spatial position: AIP neurons respond to small line fragments at a particular position in their RF, but this selectivity is entirely different for even the smallest position shifts. Yet the fragment selectivity reported here cannot be explained as a pure position effect, given the strong selectivity we observed for exceedingly small line fragments in the center reduction test (in which all stimuli appeared at the same position). Besides, a purely spatial response would evoke high correlations between the fragment levels since the 8-fragment and 16-fragment stimuli occupied similar positions on the contour, yet we consistently observed low correlations between responses to the different fragment levels.
Visual acuity decreases sharply with increasing eccentricity. Therefore, the differences in the stimulus position between the standard reduction test and the center reduction test may have introduced small differences in the spatial blurring of these small line fragments and thereby caused differences in neural selectivities between the two tests. Furthermore, since a significant number of neurons (18 of 37, 49%) in the center reduction test were tested with line fragments presented at the foveal location, the presence of the fixation point (which measured 0.2 × 0.2°) may also have interfered with the fragment selectivity of these neurons. To investigate these two potentially confounding factors, we performed the ranking analysis separately for cells tested at the fixation point (foveal cells, N = 18) and cells tested at a parafoveal position (parafoveal cells, N = 19). The results were virtually identical for both foveal and parafoveal cells: a limited preservation of the fragment preference for the 4-fragment and 8-fragment stimuli, and an almost complete loss of the fragment preference for the 16-fragment stimuli (data not shown).
Because all line fragments were presented at the same position in the center of the RF, the center reduction test also enabled us to investigate whether the small line fragments that drive AIP responses may share some common features. Figure 9A shows the best three 16-fragment stimuli (i.e., evoking the strongest responses) for all neurons tested in the center reduction test, superimposed and color-coded according to the strength of the response (normalized to the highest response in the test). Some neurons responded maximally to simple straight lines (e.g., ch037) or curved line segments (e.g., h029), but for the majority of the AIP population it was difficult to identify a single feature that drove the responses (e.g., ch012, ch049, h031).
Feature selectivity. A, The most effective (i.e., eliciting the strongest responses) three 16-fragment stimuli are illustrated for all neurons tested in the center reduction test (N = 37). The color of the fragments indicates the normalized (to the highest response in the test) response of the neuron. We only included fragments evoking ≥70% of the response to the intact contour. B, Orientation selectivity. Preferred orientation (left column) and preferred 16-fragment stimulus (right column) for the 20 neurons (rows, indicated by numbers 1–20) tested in the orientation test and the center reduction test. The colors indicate the average net response normalized to the highest response in either test.
To identify whether the fragment responses we measured could be explained by simple orientation selectivity, we presented straight lines measuring 1.3° at eight different orientations at the center of mass of the preferred outline in 50 AIP neurons (orientation test). Almost half of these neurons (23 of 50, 46%) were significantly tuned for orientation (ANOVA: p < 0.05). However, the average response to the preferred orientation (16 spikes/s) was lower than that to the preferred fragment (25 spikes/s). Twenty neurons were tested in the orientation test and in the center reduction test. Figure 9B shows the best oriented line (left column) and the best 16-fragment stimulus (right column) for each neuron (numbers in the left column indicate individual neurons). Although many AIP neurons responded to straight lines, the preferred orientation rarely matched with the preferred 16-fragment stimulus. Therefore, simple orientation selectivity could not account for the fragment responses observed in AIP.
Discussion
We determined the critical shape features for AIP neurons that responded to 2D images of objects. These object-selective neurons typically responded to exceedingly small line fragments measuring merely 1–2°. Significantly, the neural preference for the line fragments was not preserved over small shifts in retinal position. These findings indicate that object responses in the AIP area stem from the combination of a selectivity for small line fragments and spatial position.
Our neurons were located in the pAIP area. However, neurons in the anterior AIP (aAIP) area also respond to images of objects that were used in a visually guided grasping task, though were less frequently selectively (P. Pani and P. Janssen, unpublished observations). The anatomical location of highly visual neurons in the pAIP area is consistent with earlier studies (Baumann et al., 2009) reporting a posterior-to-anterior gradient of visual responsiveness in the AIP area. The boundary between the pAIP area and anterior lateral intraparietal (LIP) area, where shape responses can also be recorded (Janssen et al., 2008), is poorly defined. Therefore, and to remain consistent with previous studies (Gallese et al., 1994; Murata et al., 2000), we prefer to use a functional criterion to demarcate the boundary between the LIP and AIP areas: spatially selective saccadic activity is typical for the LIP area, whereas object responses, foveal RFs, and grasping activity is mostly observed in the AIP area. The absence of saccadic responses (Romero et al., 2013) and the presence of strong grasping-related activity indicate that the recording sites in this study were effectively located in the AIP area.
The AIP area is one of the end-stage areas of the dorsal visual stream that is considered critical for visuomotor transformations during object grasping. Given the position of the AIP area in the hierarchy of visual areas, our finding that the neural coding of object shape in the AIP area is in fact largely based on a selectivity for small line fragments is surprising; the smallest line fragments in our test appear to be more suitable for activating neurons in early visual areas (V1 or V2, where neurons also respond to angle stimuli; Hubel and Wiesel, 1959, 1968; Hammond and Andrews, 1978; Hegde and Van Essen, 2000; Ito and Komatsu, 2004). However, consistent with this low-level visual selectivity, the current and previous studies (Srivastava et al., 2009; Romero et al., 2012; Theys et al., 2012a) have observed very short response latencies in the AIP area, on the order of 50–60 ms after stimulus onset for the fastest neurons. These short response latencies and the low-level visual selectivity in the AIP area both suggest that relatively few processing stages may separate the AIP area from early visual areas. Since recent anatomical studies have mainly targeted the aAIP area (Borra et al., 2008), relatively little is known about the anatomical connectivity of the pAIP area. Nakamura et al. (2001) injected anterograde tracers in V3A and observed labeling throughout the lateral bank of the IPS, including a region that possibly corresponds to our pAIP area and that projects to the aAIP area. Therefore the pAIP area may receive inputs from V3A and send visual object information to the aAIP area. Thus pAIP neurons appear to represent an early stage in the hierarchy of visuomotor processing. Similarly, in the 3D domain, many AIP neurons selective for disparity-defined curved surfaces (concave and convex) are also selective for a simple step in disparity (Srivastava et al., 2009), as are neurons in area V2 (Thomas et al., 2002).
The fragment selectivity in the AIP area did not arise from simple orientation selectivity. AIP neurons frequently preferred small line fragments containing a particular curvature, similar to V4 neurons (Pasupathy and Connor, 2001, 2002). Indeed, neurons in ventral area V4 are also frequently tuned to curvature, and show consistent selectivity across shifts in position (half the RF size or more; Pasupathy and Connor, 2001). Nandy and colleagues (2013) have recently reported selectivity for orientation and curvature in V4. However, compared with previous work, the position invariance in their study was much more limited (Nandy et al., 2013), suggesting differences in orientation tuning across the RF, although the rank ordering of position was preserved. Sharpee et al. (2013) recently observed that V4 neurons that were tuned to more curved contours had smaller ranges of position invariance. Our AIP neurons frequently preferred different line fragments at different positions along the contour (Fig. 7, example neuron), and it was difficult to identify a single feature present in contours and fragments that drove the AIP response. However, in a previous study using multidimensional scaling analysis (Romero et al., 2013), we showed that a population of pAIP neurons represents simple object features, such as aspect ratio and orientation. Although orientation tuning could not explain the fragment selectivity we observed, future studies should systematically test the interaction between curvature, orientation, and spatial position in the AIP area.
In some cases we observed stronger responses to small fragments derived from a larger stimulus evoking little or no response. These observations suggest that competitive interactions between line fragments may exist in the AIP area. Similar findings have been reported for the ITC, where simplification of an object could evoke activity in columns that were not active during presentation of the intact object (Tsunoda et al., 2001).
Given our findings, the traditional view of the role of the AIP area in the extraction of object affordances may have to be revised. Object affordances constitute object parts that can be grasped. In robotics and computer vision, affordances are frequently characterized by collinearity and coplanarity (Bohg et al., 2009; Popović et al., 2011), which were present in our full-contour and four-fragment stimuli. However, the great majority of our AIP neurons responded equally well to 8-fragment and 16-fragment stimuli lacking those features. Therefore, the shape features encoded by AIP neurons—even those responding during object grasping—may be simpler than grasping affordances. Obviously, the grasping deficit resulting from AIP inactivation (Gallese et al., 1994) illustrates that the input from this area is critical for the network that transforms visual information into motor commands, but additional processing must be necessary (possibly in the subsectors of area F5 to which the AIP area projects, or in parietal area PFG) to encode larger object parts that can be grasped.
The fragment selectivity in our study was strongly affected by very small changes in stimulus position within the RF. Particularly for the 16-fragment stimuli, no selectivity was retained at the population level across these small position shifts. Note that the lack of position tolerance in the AIP area does not imply that AIP neurons were no longer selective for line fragments at a second location (i.e., at the center of mass of the shape contour); rather, the preference was radically different across the two positions tested. Since all tests were conducted at the center of the RF, and given the small size of the stimuli, it is unlikely that accidental interactions between the line fragment and the shape of the RF can explain these observations. These results imply that AIP neurons always signal a line fragment at a particular position. The dependency of the fragment selectivity on position poses serious challenges for the readout of AIP activity by higher cortical areas: a population of AIP neurons can effectively signal which object (fragment) is present, but the population activity will be entirely different for small shifts in the position of the object or small shifts in the eye position, which occur frequently during natural vision (Flanagan and Johansson, 2003). In contrast, position tolerance is a key feature of neurons in the ITC (Schwartz et al., 1983; Op De Beeck and Vogels, 2000): although ITC neurons can be sensitive to small position shifts, individual ITC neurons usually preserve the rank order of shape preferences across position changes (DiCarlo and Maunsell, 2003).
In a previous study (Romero et al., 2012), we showed that some AIP neurons, primarily those with foveal RFs, can exhibit a limited degree of position invariance. However, in that study neurons were tested with relatively large (5°) images of objects at a limited number of positions (nine), spaced 5° apart, and a substantial fraction of the neurons responded only at the fixation point (and therefore did not tolerate any position change). Here, we tested AIP neurons with the smallest stimuli to which the neurons responded and with much smaller position shifts (2.5°) within the RF. Our data resemble those reported for the neighboring anterior LIP area (Janssen et al., 2008) acquired with a different stimulus set. In the LIP area, the rank order of the shape preference was not preserved across changes in size, and only a small fraction of the neurons (19%) retained some shape selectivity across position (Janssen et al., 2008). Therefore, the marked influence of spatial position may be a general feature of the action-oriented dorsal stream, which distinguishes it from the ventral stream. As such, our data demonstrate that the coding of “shape” (i.e., selectivity for object shape and tolerance to stimulus transformations that preserve stimulus identity) may not occur in the IPS areas as it does in the ITC. Thus our results provide the first solid basis with which to develop computational models on the object representation in the posterior parietal cortex.
Footnotes
This work was supported by Geconcerteerde Onderzoeksacties (GOA 2010/19), Fonds voor Wetenschappelijk Onderzoek Vlaanderen grants (G.0495.05, G.0713.09), Programmafinanciering (PFV/10/008), Interuniversitaire Attractiepolen VII-11, and ERC-2010-StG-260607. We thank Piet Kayenbergh, Gerrit Meulemans, Stijn Verstraeten, Marc Depaep, Wouter Depuydt, and Inez Puttemans for technical assistance, and Steve Raiguel for comments on the manuscript.
The authors declare no conflicts of interest.
- Correspondence should be addressed to Peter Janssen, MD, PhD, Laboratorium voor Neurofysiologie en Psychofysiologie, Leuven Medical School, Herestraat 49, bus 1021, B-3000 Leuven, Belgium. peter.janssen{at}med.kuleuven.be
