Abstract
Differences in the horizontal positions of retinal images—binocular disparity—provide important cues for three-dimensional object recognition and manipulation. We investigated the neural coding of three-dimensional shape defined by disparity in anterior intraparietal (AIP) area. Robust selectivity for disparity-defined slanted and curved surfaces was observed in a high proportion of AIP neurons, emerging at relatively short latencies. The large majority of AIP neurons preserved their three-dimensional shape preference over different positions in depth, a hallmark of higher-order disparity selectivity. Yet both stimulus type (concave–convex) and position in depth could be reliably decoded from the AIP responses. The neural coding of three-dimensional shape was based on first-order (slanted surfaces) and second-order (curved surfaces) disparity selectivity. Many AIP neurons tolerated the presence of disparity discontinuities in the stimulus, but the population of AIP neurons provided reliable information on the degree of curvedness of the stimulus. Finally, AIP neurons preserved their three-dimensional shape preference over different positions in the frontoparallel plane. Thus, AIP neurons extract or have access to three-dimensional object information defined by binocular disparity, consistent with previous functional magnetic resonance imaging data. Unlike the known representation of three-dimensional shape in inferior temporal cortex, the neural representation in AIP appears to emphasize object parameters required for the planning of grasping movements.
Introduction
A large body of evidence has highlighted the importance of the ventral visual stream in object recognition (Logothetis and Sheinberg, 1996; Tanaka, 1996). Neurons in inferior temporal cortex (IT) preserve their shape selectivity under various stimulus transformations such as changes in retinal position, size, or the visual cue defining the shape, and partial occlusion (Schwartz et al., 1983; Sáry et al., 1993; Ito et al., 1995; Kovacs et al., 1995).
We previously demonstrated that neurons in the lower bank of the rostral superior temporal sulcus (STS)—a region in IT that we labeled TEs—provide a robust representation of three-dimensional shape (Janssen et al., 1999b, 2000a). TEs neurons respond selectively to disparity-defined concave and convex surfaces, and most of these neurons preserve their depth structure preference over different positions in depth, a hallmark of higher-order disparity selectivity. TEs neurons are able to signal small differences in the depth structure, are mainly sensitive to the sign of curvedness (concave vs convex), and respond primarily to the second spatial derivative of disparity (Janssen et al., 2000b). Although higher-order disparity selectivity was subsequently demonstrated in caudal intraparietal (CIP), V4, and middle temporal (MT)/V5 areas (Taira et al., 2000; Hinkle and Connor, 2002; Watanabe et al., 2002; Nguyenkim and DeAngelis, 2003), the detailed three-dimensional shape representation in TEs is currently the only known representation of disparity-defined curved surfaces (Hegdé and Van Essen, 2005).
The dorsal visual stream has been implicated in the analysis of space and the planning of actions (Ungerleider and Mishkin, 1982; Andersen and Buneo, 2002). Dorsal stream areas—particularly those in the lateral bank of the intraparietal sulcus (IPS)—may also analyze object shape. Monkey functional magnetic resonance imaging (fMRI) studies suggest shape-related activations in several regions of the IPS (Sereno et al., 2002; Denys et al., 2004; Sawamura et al., 2005), and single lateral intraparietal (LIP) neurons show selectivity for simple two-dimensional shapes (Sereno and Maunsell, 1998; Lehky and Sereno, 2007; Janssen et al., 2008). Furthermore, neurons in neighboring anterior intraparietal (AIP) area respond selectively to real-world objects during passive fixation and grasping (Murata et al., 2000; Gardner et al., 2002). It remains unclear, however, to what extent the object-selective AIP responses represent a neural coding of two-dimensional contour, depth structure, size, orientation, or any other feature present in graspable objects.
An awake monkey fMRI study (Durand et al., 2007) showed that a region in the anterior lateral bank of the IPS (AIP/LIP) is more activated by disparity-defined curved and slanted surfaces than by flat surfaces presented at different positions in depth, suggesting the presence of neurons selective for the depth structure of objects. Because fMRI cannot directly demonstrate neural selectivity nor tuning for stimulus features, the objective of the present study was (1) to verify the hypothesis that anterior IPS neurons respond selectively to disparity-defined slanted and curved surfaces and (2) to determine the tuning of anterior IPS neurons to different degrees of curvedness and different approximations of curved surfaces to elucidate the representation of three-dimensional shape at the end stage of the dorsal stream.
Materials and Methods
Stimuli and tests.
All stimuli and tests were identical with the ones described by Janssen et al. (2000b). The basic stimulus set consisted of 32 pairs of disparity-defined curved surfaces (four pairs of depth profiles combined with eight two-dimensional shapes) (supplemental Fig. S1, available at www.jneurosci.org as supplemental material), filled with a 50% density random-dot pattern. The two members of a pair of three-dimensional shapes used the same monocular images, and interchanging the monocular images between the eyes yielded two three-dimensional shapes that differed only in the signs of their disparity gradients (convex surfaces become concave and vice versa). The combination of a two-dimensional contour and a depth profile generates a three-dimensional shape. The vertical extent of the stimuli measured 5.5°, dot size was 2 arcmin, and stimulus contrast (ΔI/I) was 4.6. Disparity only varied along the vertical axis of the shape. The depth profiles in the search test consisted of a half-period sine profile (sin(ω), ω ε [0, π/2]), an inclined surface (sin(ω), ω ε [−π/4, π/4]), a full-period sine profile (sin(ω), ω ε [−π/2, π/2]), and a Gaussian depth profile (supplemental Fig. S1, available at www.jneurosci.org as supplemental material).
The stimuli were presented dichoptically by means of a double pair of ferroelectric liquid crystal shutters (two superimposed shutters in front of each eye; Displaytech). All four shutters were operating at 60 Hz, synchronized with the vertical retrace of the display monitor (vertical refresh rate, 120 Hz) equipped with a fast-decay P46 phosphor (VRG). Thus, each eye was stimulated at 60 Hz. There was no measurable cross talk between the eyes: no luminance was measured behind the shutter of the right eye during presentation of the left eye image alone because of the fast-decay phosphor, and no luminance was measured behind the shutter in closed state because of the double-shutter system (contrast ratio open vs closed state, 1:106). The viewing distance was 86 cm. The timing of stimulus onset was registered using a photodiode placed at the bottom of the display and stimulated by a bright square appearing simultaneously with the stimulus.
We searched for responsive neurons while disparity-defined curved surfaces were presented foveally during passive fixation. In the “search test,” we only presented 32 curved surfaces (one member of each pair of three-dimensional shapes): the eight two-dimensional shapes combined with four depth profiles (supplemental Fig. S1, top row, available at www.jneurosci.org as supplemental material).
In the “disparity test,” we presented two pairs of three-dimensional surfaces with identical two-dimensional contours binocularly, selected on the basis of the responses observed in the search test (one stimulus to which the neuron fired strongly and a less effective stimulus), together with monocular presentations of the same stimuli to the left and the right eye separately. The three-dimensional shape pair yielding the strongest selectivity (i.e., the largest response difference between the two members of the pair) was subsequently tested in detail. The most effective stimulus was labeled the preferred three-dimensional shape; the other member of that pair (with identical two-dimensional contour but opposite depth profile) was termed the nonpreferred three-dimensional shape.
In the “position-in-depth test,” the preferred and the nonpreferred stimuli were presented binocularly at five positions in depth. The average disparity varied from −0.5° (near) to +0.5° (far), which was larger than the disparity amplitude in the stimulus (0.65°). Because the disparity in the stimulus varied between −0.83 and −0.18° for the extreme near position in depth, and between +0.18 and +0.83° for the extreme far position in depth, each stimulus in the position-in-depth test consisted entirely of either near or far disparities for the extreme positions in depth.
In the “disparity order test,” we presented the original pair of three-dimensional shapes at the fixation point along with lower order approximations of the original depth profile. First-order approximations consisted of a plane inclined in depth, to approximate either the top or the bottom part of the original three-dimensional shape. A least-square approximation yielded a wedge-shaped stimulus with the same disparity amplitude as the original three-dimensional shape (0.65°). The three different zero-order approximations (discrete stimulus) consisted of three planes at different positions in depth (disparity amplitude, 0.65°), with a central plane of varying size.
In the “sensitivity test,” the preferred and the nonpreferred stimuli were presented foveally at the fixation plane. We used six amplitudes of the disparity variation for preferred and nonpreferred three-dimensional shape ranging from 1.3 to 0.03°, approximately evenly distributed on a logarithmic scale. In the “position-frontoparallel test,” we presented the preferred and the nonpreferred three-dimensional shape at the fixation point and at eight positions in the frontoparallel plane centered around the fixation point at 2.5° eccentricity. The stimulus duration was 800 ms in all tests.
Recording procedures.
Under isoflurane anesthesia and aseptic conditions, two male rhesus monkeys (monkey D, 7 kg; monkey H, 5 kg) were implanted with a head post and a recording cylinder (Crist Instrument), oriented vertically above the IPS. A second cylinder was implanted orthogonal to the recording cylinder in the coronal plane of the same hemisphere. This cylinder allowed real-time imaging of the IPS and the electrode tip during the recordings by means of high-resolution ultrasound (Philips HDI 5000 SonoCT: scan frequency, 17 MHz; resolution, 0.5 mm; Philips). All surgical techniques and veterinary care were performed in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Katholieke Universiteit Leuven ethical committee. Monkey D showed excellent stereopsis in a test using visual evoked potentials and dynamic random-dot stereograms, as described by Janssen et al. (1999a). Standard extracellular recordings were made using tungsten microelectrodes (impedance, 1 MΩ at 1 kHz; FHC), inserted vertically through the dura by means of a 23 gauge stainless-steel guide tube positioned in a grid (Crist Instrument). Structural magnetic resonance imaging (MRI) (0.6 mm slice thickness), using glass capillaries filled with a 1% copper sulfate solution inserted into several grid positions, confirmed that the recordings were made in the anterior part of the lateral bank of the IPS.
The horizontal and vertical positions of each eye were monitored using an infrared-based camera system (EyeLink II; SR Research) sampling at 500 Hz. Each trial started with the onset of the fixation point in the center of the display. If the positions of both eyes remained within a 1° square window for at least 400 ms, the stimulus was presented for 800 ms. Correct trials were rewarded with a drop of water. Eye position signals, neural activity, and photocell pulses were digitized and processed at 20 kHz on a digital signal processor (DSP) (C6000 series; Texas Instruments). Spikes were discriminated on-line by the DSP using a dual time-window discriminator and displayed using LabView and custom-built software.
If a unit was well isolated and responsive to at least one of the stimuli in the search test (N = 149), we selected the stimulus for which the neuron fired most strongly (the preferred three-dimensional surface) and a second one to which the neuron did not respond (having a different depth profile, but the same two-dimensional contour as the preferred). Both of these three-dimensional surfaces were subsequently tested in the disparity test, together with their counterparts composed of the same monocular images but interchanged between the eyes, and with monocular presentations to the left eye and the right eye alone. If the neuron showed significant response differences between the members of a pair of three-dimensional surfaces that could not be accounted for by the pattern of the monocular responses, the pair of three-dimensional surfaces that evoked the strongest selectivity was studied further in the position-in-depth test (N = 112). If the neuron displayed higher-order disparity selectivity (N = 100), we proceeded to the disparity-order test, the position-frontoparallel test, and the sensitivity test.
Data analysis.
We calculated net neural responses by subtracting the mean activity in the 400 ms immediately preceding stimulus onset from the mean activity between 50 and 450 ms after stimulus onset. Data analysis was performed using Matlab (Mathworks) and Statistica (StatSoft).
In the search test, we computed an index to quantify how many of the two-dimensional contours in our stimulus set (supplemental Fig. S1A, available at www.jneurosci.org as supplemental material) elicited significant responses, defined as Swidth = (N − SUM/MAX)/(N − 1), in which N is the number of conditions, SUM is the sum of the responses, and MAX is the maximum response of the neuron (Rainer et al., 1998). The index is 1 if the neuron responds to only one of the two-dimensional contours, and zero if the neuron responds to all stimuli equally.
In the disparity test, an ANOVA of the responses to the four three-dimensional surfaces in the test (stereo condition) was used to assess the significance of the disparity selectivity. To verify that the observed selectivity did not reflect a selectivity for the monocular images, we calculated a stereo difference index (SDI) (Janssen et al., 1999): SDI = (Dstereo − Dmono)/(Dstereo + Dmono), in which Dstereo represents the response difference in the stereo condition between preferred and nonpreferred three-dimensional surface, and Dmono represents the difference in the sum of the monocular responses for each three-dimensional surface. An SDI = 0 indicates that the response difference in the stereo condition equals the response difference in the sum of the monocular responses; an SDI >1 indicates that the difference in the monocular responses has a sign opposite that of the binocular responses. If the SDI was >0.5, the stereo selectivity was considered to arise from binocular rather than monocular mechanisms (Janssen et al., 1999b). The binocular summation index was calculated as follows: (response to preferred three-dimensional surface) − (sum of the monocular responses)/(sum of monocular responses). To quantify the degree of selectivity, we calculated a stereo selectivity index (SSI) on the responses of the three-dimensional shape pair yielding the largest selectivity as follows: (response to the best three-dimensional surface − response to the worst three-dimensional surface)/(response to the best three-dimensional surface).
Response latencies were calculated by identifying the first of three consecutive 20 ms time bins in which the population response to the preferred three-dimensional surface was significantly higher than in the preceding time bin (t test, p < 0.05). The latency of the neural selectivity was likewise computed as the first of three consecutive 20 ms time bins in which the population response to the preferred three-dimensional shape significantly exceeded the response to the nonpreferred three-dimensional shape (t test, p < 0.05). Alternative calculations based on a Poisson statistic on the individual neurons yielded very similar latencies.
In the position-in-depth test, we analyzed the traces of the left and the right eye from 50 to 450 ms after stimulus onset. The EyeLink II infrared eye tracker can reliably detect small eye movements but does not measure the actual vergence angle of the two eyes. Previous studies, however, have demonstrated that monkeys—like human observers—generally obtain stable vergence and that fixation disparities (i.e., misalignment of the eyes at the fixation point) are generally small compared with the measurement error of the recording system (Read and Cumming, 2003). Large fixation disparities would obviously result in diplopic vision, which would seriously hinder accurate fixation in a 1° electronically defined window around the fixation point. Therefore, we can assume that the vergence angle of our monkeys was correct at the time of stimulus onset. For the position-in-depth test, it was critical to measure changes in vergence angle after stimulus onset: vergence eye movements “tracking” the stimulus in depth could potentially reduce the variation in mean disparity in the test and thereby preclude any conclusion about higher-order disparity selectivity. In the beginning of every recording session, we calibrated the eye position signal by presenting the fixation point (without the stimulus) at two positions on the horizontal meridian 4° from the center of the display. This calibration allowed to estimate the deviation of the two eyes (in degrees) evoked by presenting the stimuli at different positions in depth. Furthermore, we tested the ability of the EyeLinkII system to detect vergence eye movements by presenting the fixation point at different positions in depth (0.125° near and far). We were able to reliably detect vergence eye movements of 0.125°. Thus, although the exact vergence angle could not be measured with our infrared eye tracker, we were able to detect small changes in vergence angle evoked by the stimulus (see Results).
The stimulus was presented if both eyes remained for 400 ms within a 1° window centered on the fixation point. We measured changes in the vergence angle by calculating the deviation of the two eyes after stimulus onset compared with a 50 ms interval (25 data points) immediately preceding stimulus onset. The eye displacements were zero-centered by subtracting the mean position of each eye, over a 50 ms window before stimulus onset, from each sample, and were smoothed by convolving with a Gaussian kernel (kernel width, 25 ms). For each stimulus, we tested whether the mean difference between the left and the right horizontal eye position and the SD of the eye position difference deviated significantly after stimulus onset.
To determine to what extent three-dimensional shape and position in depth can be decoded from the population of AIP neurons tested in the position-in-depth test, we used a linear support vector machine (SVM) (machine learning package Spider) (Weston and Watkins, 1998) for classification (http://www.kyb.tuebingen.mpg.de/bs/people/spider/main.html). We trained the SVM using a one-versus-one approach with a threefold cross-validation for optimizing the SVM parameter. The analysis was run on the responses of a representative sample of 73 AIP neurons (41 in monkey D; 32 in monkey H): for the classification of position in depth, we used the responses to the preferred three-dimensional shape at each of the five positions tested, and for the classification of three-dimensional shape, we used the responses to the concave and convex surface at every position in depth. For every neuron, we randomly selected four trials for training and two for testing the SVM. To obtain robust results, we performed the training and test procedure 10,000 times by permuting each time the order of neurons and the selection of trials for training and testing. Decoding accuracy was expressed per category as the proportion of correct classifications (position in depth or three-dimensional shape), averaged across all permutations. Chance level was 0.2 for the position-in-depth analysis and 0.5 for the three-dimensional shape analysis. To test the validity of the SVM analysis, we ran the SVM on the data of the position-in-depth test after random shuffling of the labels, separately for three-dimensional shape and position in depth: classification performance was at chance level for both three-dimensional shape (mean accuracy, 0.50; SD, 0.11; 10,000 permutations) and position in depth (mean accuracy, 0.2; SD, 0.13; 10,000 permutations).
In the analysis of the sensitivity test, we flipped the normalized sensitivity profiles for individual neurons so that the preferred three-dimensional shape was always represented on the right of the graph. Neural responses were normalized to the highest response in the test. We classified each neuron into one of three classes: for monotonic neurons, the largest disparity amplitude in the test evoked either the strongest response or a response statistically indistinguishable from the response to the second-largest amplitude, whereas tuned neurons showed an optimal amplitude of the disparity variation and a significant decrease in response on either side of the optimal amplitude. The remaining neurons were classified as broadband.
We calculated the Euclidean distances between the stimuli based on the following formula: (Σin (Ri1 − Ri2)2)1/2, where Ri1 is the mean response of neuron i to stimulus 1 and n is the number of neurons tested. We used nonmetric multidimensional scaling in Statistica (Statsoft) using a matrix of distances between each pair of stimuli as input. An optimal number of dimensions for the resulting stimulus configuration was chosen by examining the scree plot for a significant reduction in stress of the configuration by adding an extra dimension.
In the disparity order test, we computed ANOVAs with three-dimensional structure (preferred vs nonpreferred) and stimulus type (original vs reduced version) as factors. A significant interaction between three-dimensional structure and stimulus type (original vs first-order stimulus) was considered to be indicative of second-order disparity selectivity. First-order neurons showed no significant interaction between three-dimensional structure and stimulus type.
In the position-frontoparallel test, we calculated t tests on the responses to preferred and nonpreferred stimulus to assess the three-dimensional shape selectivity at each position.
Results
We recorded activity in 149 responsive single neurons in area AIP (Fig. 1) of two awake rhesus monkeys (78 neurons in monkey D; 71 neurons in monkey H) using exactly the same paradigm and stimuli as in previous studies in TEs (Janssen et al., 1999b, 2000b). We reconstructed the recording area using structural MRI and high-resolution (0.5 mm) ultrasound imaging (Philips HDI 5000 SonoCT) of the IPS and the electrode tip during the recordings (Fig. 1A,B). All recordings were verified to be in the lateral bank of the IPS. Figure 1C shows the recording area projected onto a flat map of the IPS. The recordings were made in the anterior part of the lateral bank of the IPS, anterior to the region activated during saccades in a separate fMRI experiment (Durand et al., 2007).
Recording sites. Ultrasound image (A) and anatomical MRI (B) of the recording area. The white arrows indicate the IPS, and the green arrow indicates the microelectrode in situ on the ultrasound image (A), also schematically represented by a yellow line on the anatomical MRI (B). MPRAGE, Magnetization-prepared rapid-acquisition gradient echo. C, Recording area projected onto the flat map of the IPS in the left hemisphere. Dark blue, Monkey D; red, monkey H. The white circles indicate the boundary of saccade-related activations in a separate fMRI experiment (Durand et al., 2007). The light blue area indicates the recording sites in area LIP for monkey D.
We also recorded in more posterior grid locations in monkey D (indicated in light blue in Fig. 1C). We did not observe any responses to foveally presented three-dimensional shapes in this region, but many neurons responded to simple two-dimensional shapes presented parafoveally and before and during saccades to small targets presented in the contralateral hemifield (Janssen et al., 2008). These observations are consistent with previous single-cell studies and with the fMRI results of Durand et al. (2007), and strongly suggest that these more posterior recording positions were located in the anterior part of area LIP. Neurons responsive to three-dimensional shapes were concentrated in the more anterior grid locations (Fig. 1C, dark blue and red contours), but saccadic activity was virtually absent in this region and simple two-dimensional shapes presented in the contralateral hemifield were not effective in driving the neurons (Janssen et al., 2008). Therefore, the recording area corresponds to area AIP, consistent with Murata et al. (2000).
In the search test, most AIP neurons preferred a given two-dimensional contour (supplemental Fig. S1, available at www.jneurosci.org as supplemental material). We calculated Swidth to quantify the average number of contours in the search test to which the neurons responded (Rainer et al., 1998). The median Swidth of the subpopulation of neurons for which search test data were stored (N = 52) equaled 0.63 (0.62 for monkey D; 0.66 for monkey H; NS). Hence AIP neurons display sensitivity to the two-dimensional contours of the stimulus. The focus of the present study, however, was to study the neural selectivity for the depth structure of three-dimensional surfaces in AIP.
Higher-order disparity selectivity
Figure 2 shows an example AIP neuron responding strongly to the convex profile in contrast to its concave counterpart composed of the same monocular images (t test, p < 0.05). Presentations of the constituent monocular images failed to elicit any appreciable response, and no significant stimulus-dependent deviations in the binocular eye positions were detected (Fig. 2). To determine whether the observed selectivity reflects a selectivity for the monocular images, we adopted the same criterion as used in TEs (Janssen et al., 1999b): if the difference in the stereo conditions was at least three times larger than the difference in the sum of the monocular responses (SDI ≥ 0.5) (see Materials and Methods), the stereo selectivity was considered to arise from binocular rather than monocular mechanisms. The SDI for this example neuron equaled 1.06. Note also that the average deviation of the two eyes (0.03°) after stimulus onset was negligible compared with the disparity amplitude in the stimulus (0.65°). To ensure that the observed selectivity was attributable to the depth structure of the stimulus (i.e., to spatial variations of disparity) and not to a mere tuning for local disparities, we presented the same pair of curved surfaces at five positions in depth, with all other stimulus attributes held constant. A neuron was deemed responsive to the spatial variation of disparity (i.e., higher-order disparities) if the response to the nonpreferred three-dimensional shape did not significantly exceed the responses to the preferred three-dimensional shape at any of the positions in depth. The example neuron preserved its selectivity at every position in depth, whereas the binocular eye position traces showed only minor deviations (Fig. 3). Hence this neuron clearly displays selectivity for the depth profile of the stimulus (i.e., higher-order disparity selectivity).
Disparity test. An example neuron is shown illustrating disparity selectivity in AIP. PSTHs of the responses to the convex (top) and concave (bottom) three-dimensional surfaces in the stereo condition (left column) and the monocular presentation to the left (middle column) and right (right column) eyes (20 ms bins). The vertical calibration bar indicates 80 spikes/s, and the horizontal bar indicates the duration of stimulus presentation (800 ms). Eye position traces of the left (black) and right (blue) eyes are depicted above the responses in the stereo conditions. The inset shows the monocular images used in the test.
Position-in-depth test. Average net responses of the same neuron as in Figure 2 to the preferred (red) and the nonpreferred (blue) three-dimensional shape for every position in depth. The labels on the x-axis show the average disparity of the stimulus. Eye position traces are shown for the two extreme positions in depth and the center position (middle). The vertical dashed line indicates stimulus onset.
Eighty-three percent (123 of 149) of the responsive AIP neurons showed significant disparity selectivity, which could not be accounted for by the pattern of responses to the monocular presentations (74% in monkey D; 92% in monkey H). A small fraction of the neurons recorded (5 of 149; 3%) showed significant selectivity between the two members of a pair of three-dimensional shapes but also an SDI <0.5 and were excluded from additional analysis. The median SDI in our population of 123 selective AIP neurons equaled 1.31 (Fig. 4A) (1.34 for monkey D; 1.29 for monkey H; NS). Differences between preferred and nonpreferred responses observed in AIP proved to be substantial (Fig. 4B): disparity-selective AIP neurons responded on average 2.8 times more to the preferred three-dimensional shape than to the nonpreferred counterpart (median SSI, 0.64) (Fig. 4B). The example neuron in Figure 2 also shows that AIP neurons demonstrate binocular summation (binocular summation index, 1.58) in the sense that the response in the binocular condition (preferred three-dimensional shape) was greater than the sum of the monocular responses. The median binocular summation index for the preferred three-dimensional shape was +0.07 in our population of disparity-selective AIP neurons. Hence AIP contains a sizeable proportion of disparity-selective neurons, consistent with Durand et al. (2007). Figure 4C shows the population peristimulus time histogram (PSTH) of all disparity-selective AIP neurons for the preferred and the nonpreferred three-dimensional shape. The latency of the neural selectivity was remarkably short: the first bin in which the response to the preferred three-dimensional shape differed significantly from the response to the nonpreferred three-dimensional shape occurred 60–80 ms after stimulus onset.
Population analysis of disparity test. A, Histogram of SDI pooled across subjects for all neurons (N = 123) showing significant selectivity in the disparity test (ANOVA of the net responses to the four three-dimensional shapes in the test). The last bar in A includes all cells that had an SDI of 4.5 or higher. B, Histogram of SSI for the same population of neurons (N = 123). C, Population PSTH in the disparity test. Mean normalized responses to the preferred (black) and nonpreferred (gray) three-dimensional shape are shown (20 ms bins) for all AIP neurons showing significant selectivity in the disparity test (ANOVA, p < 0.05). The vertical dashed line indicates the first bin with significant selectivity.
We tested 112 of the 123 disparity-selective AIP neurons with the preferred and nonpreferred three-dimensional shapes at five positions in depth (52 neurons in monkey D; 60 neurons in monkey H). The large majority of these neurons (100 of 112; 89%) was selective for the spatial variation of disparity rather than just disparity, and was therefore classified as higher-order neurons. The proportions of higher-order neurons were highly comparable between the two monkeys (83% for monkey D; 95% for monkey H). We ranked the positions in depth according to the mean net response of each neuron to the preferred three-dimensional shape, and calculated the average normalized response for each position rank to preferred and nonpreferred three-dimensional shape (Fig. 5). The vast majority of the higher-order neurons (94 of 100) showed significant response differences for at least four of five positions (median number of positions with selectivity, 4). Although the average firing rate at the worst position (rank 5) was less than one-half the response at the preferred position (rank 1), across neurons, the average response to the preferred three-dimensional shape was significantly higher than the response to the nonpreferred three-dimensional shape for every position rank (t test, p < 0.05).
Population analysis of position-in-depth test. A, Ranked mean normalized responses for all higher-order neurons (N = 100). Solid line, Response to preferred three-dimensional shape; dashed line, nonpreferred three-dimensional shape. 1, Position in depth to which the neuron showed the strongest response; 5, position in depth to which the neuron showed the weakest response. The responses were normalized to the highest response in the test. B, Average responses of all higher-order neurons to preferred (solid line) and nonpreferred (dashed line) three-dimensional shape in the position-in-depth test. Error bars indicate SEM.
Although almost all AIP neurons preserved their three-dimensional shape preferences over different positions in depth, most neurons were significantly modulated by position in depth: 76% (76 of 100) of the neurons showed either a significant main effect of position in depth and/or an interaction between position in depth and three-dimensional shape (ANOVA of the net responses with three-dimensional shape and position in depth as factors, p < 0.05). The middle position in depth (at the fixation plane) yielded the strongest selectivity in 35 of 100 neurons, and was never the worst position in depth. We did not observe any overall preference for near or far positions in depth (Fig. 5B).
To determine the extent to which the position in depth of the stimulus can be decoded from the responses of a population of AIP neurons, we used a support vector machine for classifying the stimuli based on single test trials from a representative sample of 73 AIP neurons. The SVM accuracy exceeded 0.98 for every position in depth (supplemental Fig. S3A, available at www.jneurosci.org as supplemental material). For comparison, the classification accuracy for concave versus convex across positions in depth equaled 1.0, which can be expected since all of these neurons showed significant three-dimensional shape selectivity. Hence both the depth structure and the position in depth of the stimulus can be decoded from the responses of a population of AIP neurons.
A small fraction of the neurons tested (9%) failed to preserve their selectivity over different positions in depth. An example neuron is shown in supplemental Figure S4 (available at www.jneurosci.org as supplemental material). This neuron reversed its selectivity for different positions in depth: the neuron preferred a concave surface at far positions, and a convex surface at near positions, which is reminiscent of three-dimensional shape responses in area V4 as reported by Hegdé and Van Essen (2005). Nevertheless, the example neuron responded significantly less to the flat shapes compared with the curved shapes across positions in depth. We tested seven neurons that failed to show invariance for position in depth with flat shapes at five disparities, and three of those seven neurons responded significantly less to the flat shapes than to the curved stimuli. Although we excluded this type of neuron from additional analysis, neurons with similar properties may have contributed to the AIP activation reported by Durand et al. (2007).
Additionally, 27 higher-order neurons were tested with flat shapes at the same positions in depth as for the curved shapes. More than one-half of the neurons tested (15 of 27; 56%) were significantly tuned for position in depth or zero-order disparity. Consistent with the data obtained by Durand et al. (2007), the average response to flat shapes was significantly smaller than to curved shapes (t test, p < 0.05; N = 15). Hence, in addition to showing selectivity for spatial variations of disparity over a surface, most AIP neurons are also modulated by the average disparity of the stimulus.
Only a small minority of the recordings of higher-order neurons (6 of 100) yielded significant changes in either the mean difference in eye positions or in the SD of the eye position difference at the extreme positions in depth (+ and −0.5°). For the extreme far positions in depth, a slow 0.1–0.15° divergence eye movement was detected in the eye traces average across all neurons tested (supplemental Fig. S2, available at www.jneurosci.org as supplemental material), but the amplitude of this vergence response was much smaller than the range of disparities used in the test (1°). In the initial 250 ms after stimulus onset, the average change in vergence angle remained within 0.1° of the prestimulus vergence angle. Toward the end of the stimulus presentation time, this vergence eye movement decreased the effective range of disparities in the test to 0.85°. We also attempted to classify three-dimensional shape (concave–convex) using the SVM on the changes in vergence angle in the disparity test, in the intervals 50–450 ms and 50–250 ms after stimulus onset. Classification performance was above chance level (mean accuracy, 0.73; SD, 0.08) for the 50–450 ms interval because of minute differences in vergence angle between convex and concave surfaces (on average, 0.002° for monkey D and 0.08° for monkey H), but at chance level for the 50–250 ms interval (mean accuracy, 0.56; SD, 0.09) (supplemental Fig. S3B, available at www.jneurosci.org as supplemental material) (average difference in vergence angle, 0.001° for monkey D and 0.003° for monkey H). In contrast, three-dimensional shape could be reliably decoded based on the spike rates in the interval between 50 and 250 ms after stimulus onset (mean accuracy, 1.0; SD, 0) (supplemental Fig. S3B, available at www.jneurosci.org as supplemental material). Hence it is extremely unlikely that the neural selectivity—which emerged as early as 70 ms and peaked at 150 ms after stimulus onset (Fig. 4C)—was affected by vergence eye movements. Thus, AIP neurons preserve their selectivity over different positions in depth and are therefore higher-order disparity selective.
First- and second-order disparity selectivity
A stimulus reduction approach can potentially reveal the stimulus features extracted by the neural population under study (Tanaka et al., 1991; Janssen et al., 2000b). We tested 66 higher-order AIP neurons with smoothly curved surfaces and various reduced versions (approximations) of these stimuli (Figs. 6A, 7A). Tilted planes (first-order stimuli), in which disparity varied linearly along the vertical axis of the shape were tested to approximate either the top or the bottom part of the original three-dimensional shape (Fig. 6A, second column; supplemental Fig. S5, top, available at www.jneurosci.org as supplemental material). We also used a linear second-order approximation (wedge stimulus) for the cosine and Gaussian depth profile (Fig. 7A, second column), in which the disparity variation was a least-square approximation of the entire smoothly curved surface of the cosine and Gaussian depth profile. Finally, we used three different discrete approximations consisting of separate planes spanning the same space in depth. The size of the central part varied between the three different discrete approximations.
First-order disparity selectivity. A, Example neuron. The icons under each PSTH illustrate the depth structure of the stimulus: from left to right, original three-dimensional shape, first-order stimulus, and three discrete approximations. The neuron showed robust selectivity for the inclined three-dimensional shape (top row) and the first-order stimulus. The three discrete approximations yielded similarly strong selectivity. The horizontal bar indicates the duration of stimulus presentation; the vertical calibration bar indicates 100 spikes/s. B, Mean net response (right) and mean absolute response difference (left) for first-order stimuli plotted against the mean net response and mean response difference to the preferred three-dimensional shape. The blue dots represent the first-order neurons, and the black dots represent second-order neurons.
Second-order disparity selectivity. A, Example neuron. The neuron showed strong selectivity for the original three-dimensional shape pair (left column) but not for the first-order stimuli (rightmost column). Robust selectivity was present for the wedge and the discrete approximations. B, The mean net response to the preferred three-dimensional shape (right) and the mean absolute response difference for the original three-dimensional shape pair (left) are plotted against the mean net response to the wedge approximation and the mean net response difference for the wedge stimuli (only second-order neurons). C, The mean net response to the preferred three-dimensional shape (right) and the mean absolute response difference for the original three-dimensional shape pair (left) are plotted against the mean net response to the best discrete approximation and the mean net response difference for the best discrete stimulus. The diagonal line represents the line of equal responses (right panel) or equal response differences (left panel). The arrow points to the example neuron shown in A.
More than one-half of the neurons tested (33 of 63; 52%) were at least as selective for first-order stimuli as for the original second-order stimuli and were therefore labeled first-order neurons (ANOVA on the net responses to original and first-order stimulus pairs; NS). An example neuron responding to the inclined three-dimensional shape is shown in Figure 6A. All three discrete approximations yielded almost equally strong selectivity. The scatter plot in Figure 6B demonstrates that the population of first-order neurons showed robust selectivity for simple linear gradients of disparity. The mean response to the preferred first-order stimulus equaled 31.6 ± 4.2 spikes/s (compared with 42.0 ± 3.4 for the original smoothly curved three-dimensional shape), and the mean response difference for first-order stimuli equaled 23.3 ± 3.2 spikes/s (compared with 23.6 ± 2.4 spikes/s for the original three-dimensional shape pair). Note, however, that the criterion we used to categorize neurons into first- and second-order neurons was inherently conservative: we labeled neurons according to the lowest order of disparity for which selectivity could be observed. The possibility remains that some AIP neurons display combined first- and second-order disparity selectivity, as in curved surfaces that are tilted in depth. Only a small fraction of the neurons (6 of 63; ∼10%) were selective for the inclined surfaces and the first-order stimuli but not for any of the other curved surfaces used in the search test. The latter subset of neurons could be considered as purely first-order.
Thirty (30 of 63; 48%) neurons were significantly more selective for curved surfaces than for first-order stimuli and were therefore classified as second-order neurons (Fig. 7A, example neuron). These results demonstrate that the neural population responding to disparity-defined curved surfaces consists of first-order and second-order neurons. None of the second-order AIP neurons, however, could reliably discriminate between the original smoothly curved three-dimensional shape and its linear approximation (Fig. 7A). Most AIP neurons responded very similarly to the original three-dimensional shape and its linear approximation (42.04 ± 17.53 spikes/s and 39.6 ± 18.5, respectively) (Figs. 7B, 8), and the response differences (that is, between preferred and nonpreferred depth profile) for the linear approximations (27.4 ± 13.8) were comparable with the response differences for the original three-dimensional shapes (29.7 ± 11.7; data points evenly scattered around the diagonal in Fig. 7B).
Population analysis disparity order test. Mean net responses to the preferred (open bars) and nonpreferred (shaded bars) original three-dimensional shape (left), the linear approximation (wedge stimulus), the first-order stimuli and three discrete approximations (right), for all second-order neurons in AIP. The stars indicate significant response differences (t test, p < 0.05). Error bars indicate SEM.
More surprisingly, a large majority of second-order neurons (22 of 30; 73%) preserved their selectivities for at least one discrete approximation of the original three-dimensional shape, as illustrated by the example neuron in Figure 7A. Our population of AIP neurons responded on average significantly more to two of three discrete approximations of the preferred depth structure than to those of the nonpreferred depth structure (t test, p < 0.05) (Fig. 8). The mean response to the best discrete approximation of the preferred depth structure equaled 34 ± 3.9 spikes/s, and the mean response difference for the best discrete approximation equaled 16.4 ± 2.7 spikes/s, compared with 42.04 ± 17.53 spikes/s average response and 29.7 ± 11.7 response difference for the original three-dimensional shape pair. Intriguingly, the time course of the neural selectivity differed between the linear approximation and the discrete approximation (Fig. 9B,C). The selectivity for the linear approximation remained high throughout the stimulus period, whereas the discrete responses changed markedly: the first period of 50–250 ms after stimulus onset showed robust selectivity, but the later part of the response (250–700 ms) showed much weaker—although still significant—selectivity. Hence the selectivity for discrete approximations was primarily present in the initial part of the response, suggesting an additional refinement of the stimulus representation later in time. Note also that a similar decrease in neural selectivity was present in the responses of the first-order neurons to the first-order stimuli (Fig. 9A). At least in the initial part of the response, AIP neurons appear to be relatively insensitive to the presence of sharp edges formed by disparity discontinuities: the relative positions of the different stimulus parts were frequently sufficient to elicit neural selectivity (Figs. 7C, 8). The tolerance of AIP neurons to manipulations resulting in these impoverished renderings of the original stimuli reveals that the representation of three-dimensional shape in AIP is essentially a coarse one.
Time courses of responses and selectivity in the disparity order test. In the top row, the average responses to preferred (full line) and nonpreferred (dashed line) three-dimensional shape are plotted as a function of time (0 is stimulus onset). Blue, Original, Smoothly curved three-dimensional shape pair. Green, First-order stimulus (A), wedge stimulus (B), and best discrete approximation (C). The bottom row plots the selectivity index (best − worst/best) as a function of time for the original three-dimensional shape pair (blue) and the respective approximations (green), in a 200 ms window sliding by 50 ms and starting at 50 ms after stimulus onset. For the first-order stimuli and the discrete approximations, the selectivity was significant even for the later part of the response (p < 10−5). Error bars indicate SEM.
Invariance for position in the frontoparallel plane
We tested 37 higher-order AIP neurons with preferred and nonpreferred three-dimensional shapes presented at the fixation point and at eight positions located 2.5° away from the foveal position. Figure 10 shows the average response to preferred and nonpreferred three-dimensional shape at every position tested. The response difference between preferred and nonpreferred three-dimensional shape was significant at every position tested except two (down on the vertical meridian and up contralateral). For more than one-half of the neurons tested (19 of 37; 52%), the center position evoked the strongest selectivity. Within the limitations of our testing procedure, we can conclude that the receptive fields of our population of AIP neurons are bilateral and include the fovea. Thus, the stimuli were properly centered in the receptive field of the neurons tested.
Position frontoparallel test. The mean responses (±SEM) to preferred (open bars) and nonpreferred (dark bars) three-dimensional shape are plotted for all nine positions tested (N = 37). FP, Fixation point; U, upper visual field; D, lower visual field; VM, vertical meridian; HM, horizontal meridian; C, contralateral; I, ipsilateral.
Neural selectivity for convex and concave surfaces might be based entirely on first-order selectivity if neurons extract a local (linear) disparity gradient in the stimulus. For example, the upper part of a concave surface consists of a disparity gradient tilted in depth toward the observer, whereas the upper part of a convex surface has the opposite tilt (supplemental Fig. S5, available at www.jneurosci.org as supplemental material). The two vertical positions in the position-frontoparallel test (2.5° up and 2.5° down) were chosen such that the linear disparity gradients were approximately matched between convex and concave surfaces: the lower part of the convex surface presented at a position 2.5° up matched the upper part of the concave surface presented at the fixation point, and vice versa for the position 2.5° down. If a neuron would extract a local linear disparity gradient, it would reverse its selectivity at one of the positions on the vertical meridian. None of the neurons tested, however, showed a reversal in three-dimensional shape preference, indicating that their selectivity included that for second-order disparity.
Representation of degree of curvedness
To investigate how well AIP neurons can discriminate between different three-dimensional surfaces, we tested 53 higher-order neurons with the same range of disparity-defined curvatures as those used in TEs (Janssen et al., 2000b). The preferred and the nonpreferred stimuli were presented foveally at the fixation plane, and the amplitude of the disparity variation ranged from 1.3 to 0.03°, approximately evenly distributed on a logarithmic scale. [We will use the term “curvedness” to describe the signed amplitude of the disparity variation present in the stimulus (i.e., the difference in disparity between the central part and the upper and lower parts in a concave surface).] In both animals, the average tuning function obtained in AIP was smooth and monotonic, with no special sensitivity to the change in the sign of curvedness (Fig. 11A; supplemental Fig. S6, available at www.jneurosci.org as supplemental material). This monotonic tuning function was not the result of averaging tuning functions with different maxima: the large majority of AIP neurons (37 of 53; 70%) displayed monotonic tunings (Fig. 11B), and only a small proportion of the AIP neurons (12 of 53; 23%) showed significant tuning for different degrees of curvedness. Very few neurons (3 of 53; 6%) were selective for the change in the sign of curvedness (−0.03 vs +0.03°; t test, p < 0.05).
Sensitivity test. A, Mean normalized responses (±SEM) to the 12 amplitudes of the disparity variation in the sensitivity test. Positive numbers indicate preferred amplitudes, and negative numbers indicate nonpreferred amplitudes. B, Classification of neurons in the sensitivity test. The thin lines indicate the mean net responses of individual neurons tested in the sensitivity test (N = 53). The bold black curve in each panel plots the mean response for each class of neurons. Negative values on the abscissa refer to the nonpreferred curvature, and positive values refer to the preferred curvature. C, Representation of curvedness in the sensitivity test. Distances between the stimuli in the sensitivity test in the neural space of AIP. Multidimensional scaling on the distance matrix of each area showed that a two-dimensional solution explained >98% of the variance in both areas. Green labels, Convex surfaces; blue labels, concave surfaces.
To quantify the neural representation of three-dimensional shape in AIP, we computed the Euclidean distances between each pair of stimuli, based on the neural responses in AIP. We used multidimensional scaling (MDS) to visualize the interstimulus distances in a two-dimensional solution (Fig. 11C). A single dimension explained 98% of the variance in AIP. More importantly, however, the MDS solution clearly shows that the distances between the stimuli in the neural space of AIP accurately reflect the extent, in depth, of the stimulus: pairs of stimuli with a similar degrees of curvedness (e.g., −0.65 and −0.32°) occupy neighboring positions in neural space, whereas pairs of stimuli with greater differences in curvedness are located further apart (e.g., −1.3 and −0.03°). Note also that the distances across the convex–concave category boundary are not appreciably larger than the within-category distances (compare, for example, the distance between 0.03 convex and 0.03 concave to that between 0.06 and 0.16 convex). Hence the three-dimensional shape representation in AIP is clearly based on the degree of curvedness, and this representation preserves distances between the stimuli on a logarithmic scale of curvedness. Together, the evidence strongly suggests that the three-dimensional shape representation in AIP is metric in the sense that the firing rate of a population of AIP neurons faithfully describes the distances between pairs of three-dimensional stimuli on a log scale of curvedness.
Comparison with TEs
The present study was designed with identical stimuli and apparatus as previous studies in TEs (Janssen et al., 2000a,b). We did not record in the animals used in the TEs studies, but the consistency of our results across animals and the magnitude of the differences in three-dimensional shape representation suggest that at least a qualitative comparison between AIP and TEs is warranted.
A number of neuronal properties were highly similar in the two areas. We observed robust selectivity for three-dimensional shape in a high proportion of neurons, most neurons responded to the combination of a two-dimensional contour and a depth profile, and many neurons in the two areas showed binocular summation. In both areas, the large majority of the neurons preserved their selectivity across different positions in depth and different positions in the frontoparallel plane, and in both areas the center position (at the fixation point and centered around zero disparity) appeared to evoke the strongest selectivity. Conversely, important differences between TEs and AIP were observed in the latency of neural selectivity, in the sensitivity of individual neurons for disparity discontinuities, and in the tuning for different degrees of curvedness. TEs neurons did not start to signal disparity differences until 130 ms after stimulus onset (Janssen et al., 1999b), compared with 70 ms in AIP. To ensure a balanced comparison between AIP and IT, we randomly sampled 56 neurons—the number of neurons used in the latency calculation for TEs neurons (Janssen et al., 1999b)—from the AIP population of 117 neurons, and report the latency as the median of the first of three consecutive 20 ms time bins across all tests (N = 10,000; t test, p < 0.05). To rule out the possibility that we observed shorter latencies because of higher firing rates in AIP neurons, we confined our latency analysis to the neurons (N = 56) responding with 14.1–35 spikes/s (mean net response, 22.8 spikes/s), which is comparable with the mean responses observed in TEs (20.1 spikes/s) (Janssen et al., 2000a). The latency in this subpopulation of AIP neurons averaged 70 ms. Thus, posterior parietal cortex signals differences in the depth structure well before IT. These results are in line with previous observations using two-dimensional shape stimuli in area LIP (Lehky and Sereno, 2007; Janssen et al., 2008): IT neurons also exhibit longer latencies than LIP neurons.
At least in the initial part of their response, AIP neurons were significantly more selective for the discrete approximations than TEs neurons (ANOVA of the mean responses to the discrete approximations, interaction between region and stimulus type, p < 0.05). Second-order TEs neurons, in contrast, did not demonstrate three-dimensional selectivity for the discrete approximations in any part of their response (Janssen et al., 2000b, their Fig. 7B,C). Thus, AIP neurons appear to be less sensitive to the presence of sharp edges formed by disparity discontinuities compared with TEs neurons: the relative positions of the different stimulus parts are frequently sufficient to elicit neural selectivity.
Finally, the average tuning function for curvedness in AIP differed conspicuously from the average tuning function in TEs: TEs neurons showed a sharp increase in neural response at the transition to the preferred sign of curvedness (Janssen et al., 2000b). The large majority of AIP neurons (37 of 53; 70%) displayed monotonic tuning, compared with only 46% with monotonic tuning in TEs. AIP neurons rarely displayed selectivity for the −0.03 versus +0.03 disparity amplitudes, but 66% of TEs neurons preserved their selectivities for the smallest differences in curvedness. Figure 12A plots the normalized response difference to 1.3 compared with 0.65° of the preferred three-dimensional shape (i.e., the within-category response differences) as a function of the normalized response difference for −0.03 compared with +0.03° (i.e., the across-category response differences). In TEs, the median across-category response difference was much larger (0.37) than the within-category response difference (−0.09), whereas the opposite was true for the AIP population (0.05 across-category compared with 0.14 within-category; ANOVA, interaction between area and category type, p < 0.01). The MDS analysis on the data previously obtained in TEs showed that a single dimension explained only 84% in TEs, compared with 98% in AIP; the scree plot (supplemental Fig. S7, available at www.jneurosci.org as supplemental material) showed a more pronounced improvement in the goodness-of-fit in TEs, compared with AIP, as a consequence of adding a second dimension. In the neural space of TEs, convex surfaces cluster to the left and concave surfaces to the right (Fig. 12B): the across-category distances between the stimuli are clearly much larger than the within-category distances. TEs neurons emphasize the sign of curvedness but provide less information about the metric differences between the three-dimensional surfaces. Therefore, the representation of three-dimensional shape in TEs is more categorical based on the sign of the curvedness, as opposed to a more metric representation of three-dimensional shape in AIP.
Comparison with TEs. A, Within- versus across-category response differences in the sensitivity test for AIP (black) and TEs (red symbols). The normalized response difference at the point of change in the sign of curvedness (+0.03 compared with −0.03°) is plotted against the normalized response difference for the maximum-disparity-amplitude pair (0.65 compared with 1.3°). Each dot corresponds to a neuron. B, Distances between the stimuli in the sensitivity test in the neural space of TEs. The conventions are the same as in Figure 11C.
We have pooled all higher-order AIP neurons in the analysis of the sensitivity test. However, more AIP neurons are first-order disparity selective compared with IT. Could this account for the observed difference in the representation of three-dimensional shape? We studied 12 neurons in the sensitivity test, which were also tested in the disparity order test and proved to be second-order neurons. As in the total population tested, the large majority of these neurons (75%) showed monotonic tuning for the degree of curvedness, and the average tuning function of these 12 neurons was very similar to the tuning curve of the entire population tested (N = 53). An additional 11 neurons that had been tested in the sensitivity test were also subjected to the position-frontoparallel test. None of these neurons reversed its selectivity for any of the positions tested, and almost all of them (10 of 11; 91%) showed monotonic tuning in the sensitivity test. We computed the neural distances between the stimuli in the test, based on the responses of these 23 neurons with proven second-order selectivity (12 from the disparity-order test and 11 from the position-frontoparallel test). The MDS solution (supplemental Fig. S8, available at www.jneurosci.org as supplemental material) was qualitatively very similar to the solution shown in Figure 11. Thus, although the sensitivity test included both first- and second-order neurons, our conclusions regarding the sensitivity test certainly pertain to the subset of second-order neurons. Furthermore, the differences between AIP and TEs in this test do not reflect the different proportions of first-order neurons in these two areas.
Discussion
Disparity selectivity has been observed throughout the monkey visual cortex, in early visual areas (Poggio and Fischer, 1977), dorsal stream areas MT/V5 (Maunsell and Van Essen, 1983), MST (medial superior temporal) (Roy et al., 1992), LIP (Gnadt and Mays, 1995), and CIP (Shikata et al., 1996), and more recently also in ventral stream areas IT (Janssen et al., 1999b; Uka et al., 2000) and V4 (Hinkle and Connor, 2002; Watanabe et al., 2002). AIP neurons respond during the visual presentation and grasping of real-world objects (Sakata et al., 1995; Murata et al., 2000; Gardner et al., 2007), and inactivation of AIP results in deficits in the preshaping of the hand to accommodate the shape of an object (Gallese et al., 1994). Human psychophysical studies have demonstrated conclusively that binocular disparity represents the most important depth cue for the control of grasp (Watt and Bradshaw, 2003). Therefore, the presence of disparity selectivity in AIP is consistent with its presumed role in the visuomotor transformations necessary for grasping objects.
Our data provide critical single-cell evidence in support of a previous monkey fMRI study (Durand et al., 2007) showing that AIP is activated more strongly by curved surfaces than by flat surfaces at different positions in depth. Theoretically, the stronger activation for curved surfaces than flat surfaces could have originated from a population of neurons responding equally strongly to convex and concave surfaces, but less to flat surfaces. Here, we have demonstrated conclusively that neural selectivity for higher-order disparities (i.e., response differences for concave and convex surfaces and invariance of the three-dimensional shape preference across positions in depth) at least partially underlies the differential fMRI activation in the study by Durand et al. (2007). Although our recording positions were centered on the activation by curved surfaces in the study by Durand et al. (2007), the fMRI activation was clearly much larger than our recording area [compare Fig. 1C with Durand et al. (2007), their Fig. 4]. Technical factors can partially account for this difference: voxel size was 2 × 2 × 2 mm, and the data were smoothed and averaged across three animals. A second contributing factor was observed in this study: a small number of neurons showed selectivity for three-dimensional shape but lacked invariance of the three-dimensional shape preference across positions in depth. These neurons are reminiscent of previous observations in V4 (Hegdé and Van Essen, 2005) and may have contributed to the fMRI effect. Since it is virtually impossible to sample an area homogeneously with single-cell recordings, the actual proportion of these partially invariant neurons may be much higher than what we observed. Finally, the apparent absence of responsive neurons in area LIP (posterior to the saccade boundary) in one monkey is puzzling and should be explored further using combined single-cell and local field potential (LFP) recordings: the possibility remains that part of the fMRI activations can only be observed at the input level (LFP) but not in the spiking output (Logothetis et al., 2001).
The overwhelming majority of AIP neurons preserved their three-dimensional shape preference across positions in depth, but the absolute responses depended substantially on the mean disparity of the stimulus. The SVM analysis showed that both the depth structure and the mean disparity of the stimulus can be accurately decoded from the responses of a population of AIP neurons.
Our data shed light on a key question that is difficult to tackle using only fMRI: how do posterior parietal neurons represent three-dimensional (parts of) objects? Individual AIP neurons respond selectively to the combination of a two-dimensional contour and a depth profile, frequently show binocular summation, and preserve their selectivity across positions in depth, positions in the frontoparallel plane, and differences in stimulus size (S. Srivastava, G. A. Orban, and P. Janssen, unpublished observations). By itself, however, invariance does not reveal the fundamental nature of the three-dimensional shape coding, and more detailed tests were necessary to assess the three-dimensional shape representation in AIP. Accordingly we have shown that AIP neurons provide detailed information about the degree of curvedness in the stimuli, not by virtue of a collection of neurons tuned to different values of this continuous quantity (Tudusciuc and Nieder, 2007), but rather through a mostly monotonic relationship between spike rate and curvedness (Orban et al., 1981). Conversely, many AIP neurons do not signal the presence of disparity discontinuities in reduced versions of the smoothly curved surfaces, at least in the initial part of their response. Thus, AIP neurons extract, or have access to, relatively rudimentary three-dimensional shape information. Yet the same neurons provide reliable and fast information about the degree of curvedness. Although the data for TEs and AIP were obtained in different animals, we feel it is safe to conclude that the coding of three-dimensional shape in AIP is at least qualitatively distinct from the coding in TEs.
Our data do not provide an account of the receptive field structure that gives rise to higher-order disparity selectivity in AIP. Because of the range of stimulus disparities we used in the position-in-depth test, we can exclude mere tuning for absolute disparities for the large majority of the neurons. The receptive field of most AIP neurons appeared fairly homogeneous in the central 5° of the visual field. Finally, the disparity order test revealed that, for many neurons, a simple step in disparity was sufficient to evoke selective responses. Because the AIP neurons we recorded required a particular two-dimensional contour combined with a specific depth profile (comparable with TEs neurons), it remains to be seen to what extent the receptive structure of AIP neurons can be charted with dynamic noise stimuli (Bridge and Cumming, 2008). Higher-order disparity selectivity does not emerge in AIP since it is already present in earlier visual areas such as CIP, in which such a mapping might be more realistic. In any case, the data presented here shed light on one of the end products of the dorsal stream (i.e., a representation of three-dimensional shape), not on the mechanisms that give rise to higher-order disparity selectivity.
Sakata et al. (1995) and Murata et al. (2000) showed that AIP neurons respond selectively to the visual presentation of objects during grasping and passive fixation. Because the animals always fixated the objects binocularly, the stimulus contained gradients of binocular disparity and other depth cues. Our results demonstrate that at least part of this object selectivity can be attributed to a coding of the disparity-defined three-dimensional structure of objects (Sakata et al., 1997). It remains to be determined to what extent the presence of binocular disparity per se is necessary for AIP selectivity.
The faster, coarser, and more metric representation of three-dimensional shape in AIP compared with TEs neatly ties with the proposed behavioral role of the two processing streams. AIP is a central player in the visuomotor transformations involved in grasping. IT, however, is crucial for object recognition, and the neural coding of shape by individual IT neurons correlates with perceptual judgments of both two-dimensional (Sheinberg and Logothetis, 1997; Op de Beeck et al., 2001) and three-dimensional shape (Janssen et al., 2003). The perception-versus-action dichotomy is reflected in the data presented here: grasping requires a rapid, approximate computation of object depth structure, orientation in depth, and position in depth that scales with the extent in depth of the stimulus (and is hence metric) to control grip aperture, whereas object recognition calls for a more elaborate and categorical representation of depth structure.
We labeled neurons according to the lowest disparity order for which selectivity was significant, but the invariance of the three-dimensional shape preference with respect to position in the frontoparallel plane suggests that most AIP neurons accomplish more than the simple extraction of a linear disparity gradient at a particular position on the surface. Although we did not manipulate combinations of tilt and curvedness, our results suggest that some AIP neurons may exhibit combined selectivities for first- and second-order disparity.
AIP projects selectively to the rostral lower bank of the STS (Borra et al., 2008), the division of IT labeled TEs, because of the high proportion of three-dimensional shape selective neurons (Janssen et al., 2000a). The faster latency and coarse coding we observed in AIP are, at least in principle, compatible with the hypothesis that AIP sends three-dimensional shape information to TEs, where it is elaborated and transformed into a categorical code. Interestingly, earlier ventral stream area V4 does not possess an invariant representation of three-dimensional shape (Hegdé and Van Essen, 2005). Area CIP, in contrast, projects to AIP and is sensitive to the orientation in depth of large planar surfaces, possibly an intermediate processing phase toward the computation of three-dimensional shape (Tsutsui et al., 2002; Sakata et al., 2005). We cannot exclude the possibility that input from a third area such as TEO is necessary for certain aspects of three-dimensional surface selectivity in AIP and TEs. In any case, our results suggest that the end stage of the dorsal visual stream extracts more rudimentary three-dimensional shape features than TEs, but does so much earlier. Combined behavioral, electrophysiological, and inactivation experiments will be necessary to elucidate the nature of the interactions between these two end stages of the dorsal and the ventral visual streams.
Footnotes
-
This work was supported by Neurobotics (European Union Project IST-001917), Fonds voor Wetenschappelijk Onderzoek Grant G.0495.05N, Geneeskundige Stichting Koningin Elisabeth, GOA (Geconcerteerde Onderzoeksactie) 2005/18, Human Frontier Science Program CDA (Career Development Award) 0057/2005-C, and EF (Excellentie financiering) 05/014. We thank Gerrit Meulemans, Piet Kayenbergh, Marc Depaep, Stijn Verstraeten, Thembi Frank, and Inez Puttemans for technical assistance.
- Correspondence should be addressed to Peter Janssen, Laboratorium voor Neuro- en Psychofysiologie, Katholieke Universiteit Leuven Medical School, Herestraat 49, bus 1021, B-3000 Leuven, Belgium. peter.janssen{at}med.kuleuven.be
