Abstract
The lower bank of the superior temporal sulcus (TEs), part of the inferior temporal cortex, contains neurons selective for disparity-defined three-dimensional (3-D) shape. The large majority of these TEs neurons respond to the spatial variation of disparity, i.e., are higher-order disparity selective. To determine whether curved boundaries or curved surfaces by themselves are sufficient to elicit 3-D shape selectivity, we recorded the responses of single higher-order disparity-selective TEs neurons to concave and convex 3-D shapes in which the disparity varied either along the boundary of the shape, or only along its surface. For a majority of neurons, a 3-D boundary was sufficient for 3-D shape selectivity. At least as many neurons responded selectively to 3-D surfaces, and a number of neurons exhibited both surface and boundary selectivity. The second aim of this study was to determine whether TEs neurons can represent differences in second-order disparities along the horizontal axis. The results revealed that TEs neurons can also be selective for horizontal 3-D shapes and can code the direction of curvature (vertical or horizontal). Thus, TEs neurons represent both boundaries and surfaces curved in depth and can signal the direction of curvature along a surface. These results show that TEs neurons use not only boundary but also surface information to encode 3-D shape properties.
The lower bank of the rostral superior temporal sulcus (STS), TEs, contains neurons selective for disparity-defined three-dimensional (3-D) shapes (Janssen et al., 1999b, 2000a). This area is part of the inferior temporal cortex (IT), which is known to be critical for object recognition (Dean, 1976;Ungerleider and Mishkin, 1982; Logothetis and Sheinberg, 1996). The large majority of TEs neurons preserve their 3-D shape preference over different positions in depth, exhibiting selectivity for the spatial variation of disparity, rather than mere absolute disparities (Janssen et al., 1999b). In fact, these neurons respond to the variation of the disparity gradient over space (Janssen et al., 2000b), i.e., display second-order disparity selectivity (Howard and Rogers, 1995). The 3-D shape selectivity is remarkably susceptible to disparity discontinuities such as sharp edges or steps in disparity, and most neurons preserve their selectivity for even very slight variations in disparity (Janssen et al., 2000b). Taken together, these results indicate that TEs neurons provide an accurate representation of robust 3-D features of an object such as the sign of its disparity curvature (concave versus convex).
The stimuli in our previous experiments were disparity-defined 3-D shapes filled with a texture of random dots and having both the surface and boundary curved in depth along the vertical axis. With two-dimensional (2-D) stimuli, IT neurons have been shown to respond selectively to either boundary (shape) or to surface characteristics (texture or color), or to combinations of the two (Desimone et al., 1984; Tanaka et al., 1991; Komatsu et al., 1992; Missal et al., 1997). Thus, the first aim of the present study was to determine the contribution of surface and boundary information to the 3-D shape selectivity of TEs neurons. Boundary disparities can be particularly useful for retrieving the 3-D structure of sparsely textured surfaces, such as that shown by a folded sheet of white paper. Indeed, it has been widely recognized that contours play an important role in stereopsis (Ramachandran et al., 1973; Mayhew and Frisby, 1976; Mayhew and Frisby, 1980). Using a stimulus reduction approach (Tanaka et al., 1991), we recorded the responses of single neurons to stimuli in which the disparity varied either along the boundary of the shape, or only along its surface. Thus, we investigated whether a boundary in depth or a disparity variation along the surface were by themselves sufficient to elicit 3-D shape-selectivity.
In the previous studies, the stimuli were curved only along the vertical axis. The second aim of the present study was therefore to investigate whether TEs neurons can also represent differences in second order disparities along the horizontal axis, such as that in a vertically oriented cylinder. In real-world objects, the (local) disparity can vary along either the vertical or the horizontal axis, or along both (Koenderink, 1990). If TEs neurons were to represent (parts of) real world objects, they should also be selective for curved surfaces along the horizontal axis.
MATERIALS AND METHODS
Subjects. Recordings were made in three hemispheres of two juvenile rhesus monkeys (monkey J. and monkey C.). Both subjects showed stereopsis, as demonstrated by means of visual evoked potentials (Janssen et al., 1999a). A head post, a scleral search coil, and a recording well were implanted consecutively in sterile conditions and under deep isoflurane anesthesia. Analgesics (Tramadol 2.5 mg/kg) were administered for 24 hr postoperatively. Horizontal and vertical movements of the right eye were recorded with the scleral search coil technique (Judge et al., 1980) at 200 Hz. Monkey C. was implanted with a second coil in the left eye to directly measure any vergence eye movements. Monkeys were trained to keep their gaze within 0.7° (monkey J.) or 0.9° (monkey C.) of a fixation target (0.2° diameter) in the center of the display. After 1000 msec of stable fixation, the stimulus was presented for 600 msec. The monkeys were rewarded with a drop of apple juice for maintaining fixation during the entire duration of the trial.
Recording sites. Standard extracellular recordings were made with tungsten microelectrodes (Frederick Haer Co., Bowdoinham, ME) in the rostral lower bank of the STS (targeted Horsley–Clark coordinates: 16 mm anterior and 22 mm lateral). Before surgery, an anatomical MRI was obtained. The recording positions were verified using CT scan (slice thickness 1 mm) with the guiding tube in situ (Janssen et al., 1999b).
Stimuli. The stimuli were disparity-defined, concave and convex 3-D shapes. Because our previous study showed that most 3-D shape-selective TEs neurons respond selectively to convex and concave 3-D shapes (Janssen et al., 2000b), we only imposed convex and concave depth profiles onto each of 10 simple 2-D shapes (Fig.1A). Each concave–convex pair of 3-D shapes used the same pair of monocular images. Interchanging the monocular images creates two 3-D shapes that differ only in the sign of their binocular disparity (convex surfaces become concave and vice versa). The stimuli were presented dichoptically by means of a double pair of ferroelectric liquid crystal shutters (Displaytech, Longmont, CO), which were placed in front of the monkeys' eyes. Each shutter opened and closed at a rate of 60 Hz, synchronized with the vertical retrace of the monitor (digital multisync monochrome monitor with P46 ultrarapid decay phosphor; Vision Research Graphics, Durham, UK). Stimulus luminance measured behind the shutters operating at 60 Hz equaled 2.5 cd/m2 (contrast [(maximum luminance − minimum luminance)/minimum luminance] = 4). No cross-talk between the monocular images was measured using a photomultiplier 475R (Brandenburg, Surrey, UK) equipped with a Hamamatsu (Hamamatsu City, Japan) R453 tube. The average vertical and horizontal diameter of the stimulus was 6.2 and 6.3°, respectively. A fixation target was superimposed on the stimulus. The fixation distance was 86 cm.
Stimuli. A, Two-dimensional shapes used to derive the 3-D stimuli of the search test. B,Vertical 3-D shape. The monocular images are shown for the correlated (first row), the decorrelated (second row), the solid 3-D shape (third row), and the 3-D rim (fourth row). The icons on theright schematically illustrate the perceived 3-D structure when the images are crossed fused. In all four vertical 3-D shapes, the outer contours were identical and curved in depth along the vertical axis. The surface of the correlated vertical 3-D shape was also curved in depth, as indicated by the luminance gradient in the icon of the correlated vertical 3-D shape. The surface of the 3-D boundary stimuli was uninformative about depth, indicated by the homogeneously dark (decorrelation), gray (solid), or blank (rim) surface in the icons on the right.C, Surface stimuli. The monocular images are shown for the restricted surface (top row) and the large surface (bottom row) stimuli, together with schematic illustrations of the perceived 3-D structure (right). The disparity varied along the surface of the shape in the vertical and in the horizontal direction. Both scale bars on the left of the monocular images indicate 2°. The scale bar for the restricted surface also applies to B and D. Note that the icons on the right only represent the 3-D percept, whereas the actual 2-D shape of the stimulus can be seen in the monocular images. Thus, only the large surface was square-shaped.D, Horizontal 3-D shape. The disparity varied only along the horizontal axis on both surface and boundary.
We used three sets of stimuli: (1) 3-D shapes in which both surface and boundary were curved in depth, (2) stimuli in which only the boundary was curved in depth, and (3) stimuli that only contained a disparity variation along the surface of the shape. The magnitude of the disparity variation equaled 0.26°, and the dot size equaled 0.032° for all 3-D stimuli. In the “correlated vertical 3-D shape”, the disparity varied only along the vertical axis of both surface and boundary, as in the previous experiments (Janssen et al., 1999b,2000a,b). A Gaussian function defined the variation in disparity over space. The shape was filled with a texture of random dots (dot density 50%) (Fig. 1B, top row). Three types of stimuli contained only boundary disparities. The “decorrelated vertical 3-D shape” was identical to the correlated 3-D shape, except that the random dot patterns of the left and right eye images were uncorrelated (Fig. 1B, second row). This manipulation eliminates the disparity information within the surface but preserves the disparity information along the boundary of the shape (schematically illustrated by the darker surface in the icon on the right side of Fig.1B). Because of false correspondences between the dots in the two images, the surface of this stimulus is generally not perceived as entirely flat. Most observers report an unstable 3-D percept of the surface, together with a clear perception of a concave or convex boundary. In the “solid vertical 3-D shape”,the monocular shapes were identical to the correlated 3-D shape, except that the random dot texture was replaced with a white surface (100% dot density) (Fig. 1B, third row). Whereas the uncorrelated dots along the surface of the decorrelated 3-D shape conveyed conflicting (rivalrous) information, the surface of the solid shape pair was congruent between the two eyes, although uninformative about 3-D structure. The most reduced 3-D stimulus was a “vertical 3-D rim”, consisting of the outer contours of the correlated 3-D shape (width, 0.19°), with the central area left blank (Fig.1B, fourth row). The rim was filled with a 50% random dot texture. Thus, unlike in the decorrelated and the solid vertical 3-D shapes, no surface was present in the 3-D rim stimulus.
Two types of stimuli contained no disparities along the boundary. In the “restricted surface stimulus”, disparity varied along both the vertical and the horizontal axis in such a way as to produce zero disparity at the boundary (Fig. 1C). The Gaussian disparity gradient along the vertical axis was multiplied by a second Gaussian function that varied from 0 to 1 along the horizontal axis. The resulting disparity was maximal in the center of the shape and smoothly approached zero toward the boundary. The “large surface stimulus” was identical to the restricted surface in its central part, but extended over a 13 × 13° square area at zero disparity. Thus, the large surface differed from the restricted surface in two respects: the textured surface was two times larger than the restricted surface and the 2-D shape visible in the monocular images was removed (Fig.1C).
The correlated horizontal 3-D shape only contained a Gaussian disparity variation along the horizontal axis within both the surface and the boundary of the shape (Fig. 1D). In the decorrelated horizontal 3-D shape, the dots were uncorrelated between the left and right eyes, again eliminating the disparity information in the surface but preserving the disparity information along the boundary. Note, however, that the amount of boundary disparity for a horizontal gradient is much smaller than for a vertical gradient and depends on the 2-D shape.
Disparity gradients along the horizontal axis result in texture density cues at each point where the disparity value changes, e.g., at the change from 0 to 0.032°, from 0.032 to 0.064°, etc. (Cobo-Lewis, 1996). In each stimulus containing a horizontal gradient of disparity, i.e., in the restricted surface, the large surface, and the horizontal 3-D shape, we removed these texture density stripes by randomly eliminating dots such that the 50% density was restored over the entire surface of the shape. Thus, no monocular texture density gradients were present in the stimulus.
Testing procedure. We searched for responsive neurons using a set of 30 stimuli (three 3-D profiles combined with ten 2-D shapes). One depth profile consisted of a concave, vertical 3-D shape, the other two were concave and convex restricted surfaces. All stimuli were presented foveally while the monkeys performed a fixation task. Responsive neurons were then tested further with 3-D stimuli derived from a single 2-D shape. Neurons were tested with concave and convex vertical 3-D shapes and restricted surfaces, combined with monocular presentations of these stimuli to verify the stereoscopic nature of the response. Next, 3-D shape-selective neurons were tested with preferred and nonpreferred 3-D shapes (either the vertical 3-D shape or the restricted surface) presented at three different positions in depth (“position-in-depth test”). The disparity difference between the near and the far positions equaled 0.39°. On the basis of the position-in-depth test, neurons were classified as either zero-order disparity selective (responsive to position-in-depth) or higher-order disparity selective, i.e., responsive to the spatial variation of disparity (Janssen et al., 2000b). Only the latter neurons were tested further in the “boundary–surface test.”
The first purpose of the boundary–surface test was to determine the contribution of surface and boundary information to the response difference between concave and convex 3-D shapes. To test whether a 3-D boundary was sufficient for 3-D shape-selectivity, we presented concave and convex decorrelated vertical 3-D shapes and solid vertical 3-D shapes. The boundary–surface test also included a vertical correlated 3-D shape pair with boundary and surface in depth. To determine whether a 3-D surface is sufficient for 3-D shape-selectivity, we presented stimuli in which the disparity varied only along the surface, i.e., the restricted and the large surface stimuli. The boundary–surface test also included a concave and a convex correlated horizontal 3-D shape to determine the selectivity for disparity variations along the horizontal axis, the second purpose of the test.
Neurons that appeared to be selective for either the decorrelated or the solid shape were additionally tested with a concave and convex vertical 3-D rim, together with the preferred and nonpreferred correlated vertical, decorrelated, and solid 3-D shape (“rim test”).
The stimuli in the boundary–surface test differed from the stimuli used in our previous studies with regard to dot size and disparity magnitude (Janssen et al., 2000a,b). To investigate the extent to which 3-D boundary selectivity can also be observed for larger dot sizes and larger disparity magnitudes, we studied a separate sample of 3-D shape-selective neurons in TEs with the stimulus used in our previous experiments. Additionally, we wanted to exclude any selectivity for the monocular images underlying the response differences for the 3-D boundaries. The “outline control test” therefore consisted of binocular presentations of a pair of correlated vertical 3-D shapes, decorrelated vertical 3-D shapes, and monocular presentations of the decorrelated 3-D shapes. As in our previous studies (Janssen et al., 2000a,b), the magnitude of the disparity varied over a range of 0.65° within the stimulus, the dot size was 0.065°, and dot density was 50%.
To determine to what extent the 3-D surface-selectivity can also be observed for different random dot textures, we tested neurons that had responded selectively to the restricted surface in the boundary–surface test by using concave and convex surfaces with 3 different dot sizes: 0.032°, 0.065°, and 0.13° (“dot size test”). The latter stimuli were composed of white and black dots on a gray background (mean luminance, 1.08 cd/m2; dot density, 25%; Michelson contrast, 67%).
Finally, some neurons selective for the horizontal 3-D shape were tested with monocular controls and horizontal decorrelated 3-D shapes.
Data analysis. Net neural responses were computed trialwise by subtracting the number of spikes counted in a 400 msec interval immediately preceding stimulus onset from the number of spikes in a 400 msec interval starting 80 msec after stimulus onset. The significance of 3-D shape-selectivity was assessed using ANOVA (p < 0.05). The selectivity was judged not to arise from purely monocular mechanisms if the difference in response between the dichoptic presentations was at least three times larger than the difference between the sum of the responses to the two monocular presentations (Janssen et al., 1999b).
In the position-in-depth test, the neurons were tested with either the correlated vertical or the restricted surface stimulus. A neuron was classified as responsive to the spatial variation of disparity if at no position in depth did the response to the nonpreferred 3-D shape significantly exceed any response to the preferred 3-D shape, as assessed by a post hoc least significant difference (LSD) test (p < 0.05; Janssen et al., 2000b). Note that sensitivity to the spatial variation of disparity does not necessarily imply invariance of the response strength over position-in-depth. Higher-order disparity selectivity could represent either first- or second-order disparity selectivity (Janssen et al., 2000b). In the boundary–surface test, response differences between convex and concave stimuli were tested for statistical significance using an LSD test. To correct for multiple comparisons, the type 1 error was set to 0.005.
To quantify the degree of selectivity within a pair of 3-D shapes, we computed a selectivity index (SI), defined as SI = [(net response to the preferred 3-D shape − net response to the nonpreferred 3-D shape)/net response to the preferred 3-D shape]. The SI indicates the differential response normalized to the higher response for a given pair of 3-D shapes. To compare the 3-D shape-selectivity over different types of 3-D stimuli, we computed a normalized response difference (NRD) defined as NRD = [(net response to the preferred 3-D shape − net response to the nonpreferred 3-D shape)/maximal response of the neuron]. The NRD gives the differential response normalized to the highest response to any of the stimuli compared.
RESULTS
We recorded the responses of 196 3-D shape-selective neurons in area TEs. Of these neurons, 120 were initially tested with two pairs of 3-D shapes and monocular presentations of these shapes, and were subsequently studied using the position-in-depth test. On the basis of this test, 16 neurons were classified as responsive to zero-order disparities and were excluded from the analysis. The 104 higher-order disparity-selective neurons entered the boundary–surface test. The remaining 76 neurons were studied in the outline control test (see Materials and Methods). Testing of a subset of 42 neurons suggested that the large majority (36 of 42) of these 76 neurons were higher-order disparity selective.
Position-in-depth test
To decide between zero- and higher-order disparity selectivity, each neuron was tested with concave and convex 3-D shapes at three different positions in depth before the boundary–surface test. Because of the wide range of disparities used, this test also allows us to evaluate the possible influence of vergence eye movements. Figure2A shows the responses of a single 3-D shape-selective neuron in the position-in-depth test. The icons above the peristimulus time histograms (PSTHs) illustrate the position in depth of the stimulus with respect to the plane of fixation (indicated by the fixation target). This neuron preserved its 3-D shape preference for all three positions tested, implying selectivity for the spatial variation of disparity contained in the stimulus, i.e., higher-order disparity selectivity (Janssen et al., 1999b, 2000b). Below the histograms, the mean horizontal positions of the right and left eyes are plotted. The average response latency of the present sample of TEs neurons was 90 msec. Therefore, in principle, eye movements can start to influence the response rates no earlier than 90 msec after the onset of the eye movement. In the present test, we detected a small vergence eye movement (0.1°) after stimulus presentation (convergence for near and divergence for far presentations). However, it is noteworthy that the neuronal selectivity was clearly present in the initial part of the response at all positions tested and was not influenced by the change in eye position.
Position-in-depth test. A, Example neuron. The icons above the peristimulus time histograms depict the position-in-depth of the stimulus for an observer viewing from the left (left column, near; right column, far). The horizontal scale bar (top right) indicates 0.25° disparity. Below the PSTHs, the mean position of left (red) and right eye (blue) are shown. The vertical bar on theleft of the eye position traces indicates 1°, and eachhorizontal bar indicates the duration of stimulus presentation (600 msec). The green lines are plotted as a reference at 200 msec after stimulus onset. The large scale bar on the right indicates 72 spikes/sec. The neuron fired strongly and selectively to the convex 3-D shape at every position in depth, and the selectivity was present in the initial part of the response. B, Population PSTH of all neurons that preferred the convex 3-D shape (N = 52). Same conventions as in A. The 3-D shape-preference is preserved at every position in depth. C, Population PSTH for all neurons that preferred the convex 3-D shape and for which binocular eye movements were recorded (N = 12). The mean positions of left (red) and right eye (blue) are plotted below the histograms. Same conventions as in A. Again, the response difference between convex and concave can be observed in the initial part of the response at every position in depth and is not influenced by the small (0.1°) vergence eye movement. D, Population PSTH of all neurons that preferred the concave 3-D shape (N= 52). Same conventions as in A.
Figure 2B shows population PSTHs, normalized to the highest bin count in the PSTHs of each neuron, of the responses of the 52 of 104 neurons for which the preferred stimulus was a convex 3-D shape. The 3-D shape preference is preserved at every position in depth. We obtained binocular eye position traces for 28 higher-order disparity-selective neurons, 12 of which preferred the convex 3-D shape. The population PSTH for these 12 neurons is shown in Figure2C. Below each histogram, the mean horizontal positions of the left and right eyes are plotted. Again, the neuronal selectivity was present in the initial part of the response and was preserved over all positions tested. As for the neuron in Figure 2A, the difference in response was largely unaffected by the change in eye position. Figure 2D shows the population PSTH for the neurons that preferred the concave 3-D shape (N = 52), which is very similar to that for the convex cells (Fig.2B). Consistent with our previous studies (Janssen et al., 1999b, 2000a,b), we can conclude that the 3-D shape-preference of TEs neurons is invariant for position-in-depth and does not reflect vergence eye movements.
Selectivity for 3-D boundaries in the boundary–surface test
To determine whether 3-D boundaries can be sufficient for 3-D shape selectivity, we tested neurons selective for the correlated vertical 3-D shape pair with stimuli containing only boundary disparities. Two example neurons are shown in Figure3, A and B. The neuron in Figure 3A responded selectively to the concave–convex vertical 3-D shape pair (first column), but was even more selective for the decorrelation and the solid shape pair, in which only the boundary was curved in depth (second and third column). The restricted and the large surface pairs did not evoke significant response differences (last two columns). Clearly, the mere presence of a disparity variation along the boundary of the shape was sufficient for 3-D shape selectivity in this neuron. The neuron in Figure 3B, however, showed the opposite response behavior: strong selectivity for the correlated vertical 3-D shape pair, but no significant response difference for the decorrelated or the solid shape. The apparent surface-based selectivity of this neuron was consistent with the significant response difference for the restricted surface pair.
Selectivity for 3-D boundaries. A,Neuron showing selectivity for the correlated vertical, the decorrelated, and the solid shape pair but no selectivity for the 3-D surfaces. B, Neuron showing selectivity for the correlated vertical 3-D shape and for the restricted surface but no significant response differences for the decorrelated, the solid 3-D shape, or the large surface stimulus. The responses to the preferred 3-D shape (either concave or convex) are plotted in the top row (below the icons), the responses to the nonpreferred 3-D shape in the bottom row. The scale bars inA and B indicate 65 spikes/sec.
Of 104 neurons, a total of 61 showed selectivity for the correlated vertical 3-D shape pair. [The remaining 43 neurons were only selective for the restricted surface stimulus (see below).] Thirty-seven neurons (61%) were selective for either the decorrelated or the solid shape pair. Because a 3-D boundary alone was sufficient to evoke significant response differences, they were termed “boundary neurons.” Figure4A shows the normalized population PSTH for these 37 boundary neurons, for the correlated vertical 3-D shape, the decorrelated, and the solid shape. The graph illustrates the substantial selectivity for 3-D boundaries in this population of cells. The median selectivity index (SI) equaled 0.88, 0.80, and 0.70 for correlated, decorrelated, and solid shape pair, respectively (Wilcoxon matched pairs tests, NS). The slightly higher selectivity for the decorrelated 3-D shape was reflected in the larger number of neurons selective for the decor-related (29) compared with the solid shape pair (23). Furthermore, the boundary selectivity emerged rapidly after stimulus onset: response differences were already significant by 100–120 msec after stimulus onset in each of the three conditions (Fig. 4A, arrow). In Figure4B, the population responses are plotted for those boundary neurons for which binocular eye movements were recorded (N = 12). The higher number of neurons selective for the solid (10) compared with the decorrelated shape pair (7) produced the relatively larger response difference for the solid shape pair compared with the decorrelated shape pair in this sample of neurons. Figure 4C shows the mean difference in the horizontal eye position for concave and convex 3-D shapes in the correlated vertical, decorrelated, and solid shape conditions, for the same neurons as in the second row (N = 12). As was the case in the position-in-depth test, the small vergence responses (0.1°) could not account for the observed 3-D shape selectivity.
Time course of 3-D boundary selectivity.A, Normalized population PSTH for all boundary neurons (N = 37), showing the average normalized response to the preferred (red) and nonpreferred (blue) correlated vertical (full line), the decorrelated (dashed line), and the solid shape pair (dotted line). B,Normalized population PSTH for all boundary neurons for which binocular eye movements were recorded (N = 12). Same conventions as in A.C, Mean difference in horizontal position between the left and the right eye, for the correlated vertical (first row), the decorrelated (second row), and the solid shape pair (bottom row), for convex (green) and concave (black) 3-D shapes. The vertical calibration bar on theleft indicates 1°. The pink vertical lineis a reference positioned at 200 msec after stimulus onset.
Although boundary neurons responded selectively to at least one of the 3-D boundaries, neurons selective for the decorrelated 3-D shape frequently showed a significant effect of decorrelation on either response strength or degree of selectivity. A 2 × 2 ANOVA on the net responses to the correlated vertical 3-D shape pair and the decorrelated shape pair revealed either reduction in the net response (significant main effect) and/or reduced difference in response between the convex and concave decorrelated shapes (significant interaction) in 18 of 29 neurons (62%). Thus, even this population of boundary neurons was frequently affected by the removal of surface information.
The stimuli in the boundary–surface test differed from the stimuli used in our previous studies in both dot size and disparity magnitude (see Materials and Methods). To investigate to what extent 3-D boundary selectivity can be observed for larger dot sizes and larger disparity magnitudes, we tested 76 3-D shape-selective TEs neurons with the correlated vertical 3-D shapes used in Janssen et al. (2000a,b), and decorrelated 3-D shapes, including monocular controls. Table1 compares the results of this outline control test to the findings in the boundary–surface test. No significant differences were observed in the proportion of neurons in which decorrelation showed a significant effect, in the number of neurons selective in the decorrelation condition, in the median SI for the correlated vertical 3-D shape pair, or in the median SI for the decorrelated shape pair (Wilcoxon matched-pairs test, NS). Importantly, for no neuron tested in the outline control test did the selectivity in the decorrelation condition arise from a selectivity for the monocular images per se. The response difference in the decorrelated conditions averaged 10 times the difference in the sum of the monocular responses, which did not significantly differ from the ratio (20:1) for the correlated 3-D shape (Wilcoxon matched pairs test, NS). Thus, 3-D boundary selectivity is also present for stimuli consisting of larger dots and a higher disparity magnitude.
Comparison between the results of the boundary–surface and the outline control test
Rim test
The most reduced version of a 3-D-boundary stimulus consisted of a 3-D rim, which contained only the outer contours of the original correlated 3-D shape. In the rim test, 20 neurons showing boundary selectivity in the boundary–surface test were studied further with the 3-D rim pair, the correlated, decorrelated, and solid vertical shape pairs. Figure 5A shows the responses of a boundary neuron that is equally selective for the 3-D rim than for the other boundary stimuli (ANOVA, interaction between 3-D structure and boundary stimulus;F(3,64) = 0.56; NS). Overall, 11 of 20 neurons tested (55%) displayed a significant selectivity to the 3-D rim pair. The population PSTH of those 11 rim neurons is shown in Figure 5B. The median SI equaled 0.77 for the rim pair, which did not differ significantly from that for the correlated 3-D shape (0.83). For at least this population of rim neurons, the removal of all surface information did not result in a reduction of the 3-D shape selectivity compared with a stimulus in which both boundary and surface were curved in depth. In the remaining nine boundary neurons, however, no selectivity was present for the 3-D rim pair, indicating that even in boundary neurons the mere presence of a surface was required for 3-D shape selectivity.
Rim test. A, Example neuron showing selectivity for all 3-D boundaries tested. The calibration bar on theright indicates 67 spikes/sec. B, Normalized population PSTH for all neurons selective for the 3-D rim (N = 11). Same conventions as in Figure 3.
To compare both selectivity and response strength in the population of boundary neurons for the correlated vertical, decorrelated, solid shape, and rim, we plotted the NRD for the 3-D boundary stimuli as a function of the NRD for the correlated vertical 3-D shape pair in Figure 6. The arrows indicate the respective median NRDs for the four stimuli containing boundary disparities. Most data points are located below the diagonal, but a substantial number of points lie near or even above the diagonal. The decorrelated and the solid shape evoked on average lower response rates than the correlated 3-D shape (Fig. 4A). Therefore, the NRD for the correlated vertical 3-D shape (0.76) was significantly larger than for the decorrelated (0.31), solid (0.48), and rim stimulus (0.16; Wilcoxon matched pairs test; p < 0.05). Although large differences in the degree of selectivity for the different 3-D boundaries can be observed for single neurons, the clouds of dots largely overlap (Wilcoxon matched pairs tests; NS), demonstrating the equivalence of all boundary stimuli tested at the population level. Overall, the data clearly demonstrate that at least for a subpopulation of neurons in TEs, curved boundaries in depth can be sufficient for 3-D shape selectivity.
Comparison of selectivity for correlated vertical and boundary stimuli. The NRDs for the decorrelated 3-D shape (triangles), the solid shape (asterisks), and the 3-D rim (squares) are plotted as a function of the NRD for the correlated vertical 3-D shape, for all boundary neurons (N = 37). The arrows on thevertical axis indicate the median NRDs for each boundary stimulus, and the arrow on the horizontal axis indicates the median NRD for the correlated vertical 3-D shape.
Selectivity for 3-D surfaces
We wanted to determine to what extent surface disparities are sufficient for 3-D shape selectivity. In other words, can we observe response differences between concave and convex stereoscopic surfaces? Figure 7A illustrates a neuron selective for a surface curved in depth. Whereas the vertical 3-D shapes with boundaries in depth elicited only weak responses, the neuron fired strongly and selectively to the restricted surface (post hoc LSD test; p < 0.005). Note that this response difference—as for all 104 neurons tested—could not be explained by a selectivity to the monocular images and was preserved over different positions in depth. Even the response to the large surface differed significantly, although to a lesser degree, between concave and convex surfaces. The neuron in Figure 7Bdisplayed a similar pattern of activity, except that the response difference for the large surface was equal to that for the restricted surface. The correlated vertical 3-D shape evoked a weak but significant response difference (post hoc LSD test;p < 0.005). Figure 3B shows an example of a neuron selective for both correlated vertical and restricted surface stimulus. These neurons demonstrate clearly that disparity variations along a surface can be sufficient for 3-D shape selectivity in TEs. Moreover, for the neuron in Figure 7B the curvature in the central area of the large surface stimulus was sufficient for 3-D shape selectivity, which suggests that the monocular 2-D contours of the stimulus were not critical.
Selectivity for 3-D surfaces. A,Example neuron showing strong selectivity for the restricted surface and a weaker selectivity for the large surface stimulus.B, Example neuron showing equally strong selectivity for the restricted and the large surface stimulus, together with a weak selectivity for the vertical correlated 3-D shape. Same conventions as in Figure 3. The scale bars on the right indicate 65 spikes/sec in A and 75 spikes/sec inB.
Overall, 73 neurons in our sample (70%) showed significant response differences for the restricted surface (median SI, 0.77). Twenty-six of the neurons selective for the restricted surface (36%) were also selective for the large surface (median SI, 0.83). Only six neurons showed significant selectivity for the large surface (median SI, 0.69) but not for the restricted surface (median SI, 0.49). Thus, the number of surface-selective neurons totaled 79. Table 2 shows the relative numbers of neurons selective for the surface and for the correlated vertical 3-D shape, as well as the relative numbers of boundary neurons for each of these cell classes. For those neurons selective for both the vertical correlated 3-D shape and the surface, but not for any of the boundary stimuli (N = 12), the presence of a curved surface was both sufficient and necessary for 3-D shape selectivity. Conversely, 13 neurons showed boundary selectivity but no selectivity for any of the 3-D surfaces. However, the restricted surface was curved along both vertical and horizontal axes, and all boundary stimuli were curved only along the vertical axis. Because these 13 neurons may not be selective for doubly curved surfaces, it cannot be concluded that the boundary in depth was necessary for 3-D shape selectivity in those neurons.
Relative numbers of neurons selective for the correlated vertical 3-D shape compared with the number of surface-selective neurons
Figure 8A shows the normalized population PSTH for the population of surface-selective neurons. As was the case for the 3-D boundaries (Fig. 4), the selectivity for the restricted surface was substantial and emerged rapidly after stimulus onset. The large surface elicited much smaller response differences, on average, than did the restricted surface. The eye position traces for the surface stimuli were very similar to those of the vertical correlated 3-D shape (data not shown). Figure8B plots the NRDs for the restricted and large surface, making a distinction between neurons that were selective for the large surface (circles) and neurons that were not (squares). Most data points are located below the diagonal, but a sizable proportion of the neurons (14 of 78; 18%) lies at or above the diagonal, indicating that in some cases, the monocular contours were not necessary for 3-D shape selectivity. On average, however, the response differences were significantly higher for the restricted surface (median NRD, 0.75) than for the large surface (median NRD, 0.2; Wilcoxon matched pairs test;p < 0.001).
Surface selectivity of the population.A, Normalized population PSTH for all surface-selective neurons (N = 79). The average response is plotted to the preferred (black) and nonpreferred (gray) restricted surface (full line) and large surface (dotted line).B, Scatterplot of the NRD for the large surface stimulus as a function of the NRD for the restricted surface stimulus, plotted separately for neurons selective (circles) or not selective (squares) for the large surface. Thearrows indicate the median NRDs for large and restricted surface.
We determined to what extent the selectivity for the restricted surface could also be observed using different dot sizes or a different dot density in 34 of 73 neurons selective for the restricted surface (dot size test). Twenty-six of those neurons (76%) were selective for at least one of the dot sizes at the 25% dot density, and 14 neurons (41%) preferred the same 3-D shape for all three dot sizes in the test (data not shown). Thus, the selectivity for a 3-D surface can be invariant for the texture pattern carrying the surface information.
Mixed selectivity
The neuron in Figure 9 displayed robust selectivity for the correlated vertical 3-D shape, as well as a large response difference for the decorrelated and the solid shape. The restricted surface, however, also evoked a significant response difference (post hoc LSD test; p < 0.001). Because both a curved boundary and a curved surface were sufficient for 3-D shape selectivity, this neuron represents an example of a mixed selectivity for 3-D boundaries and 3-D surfaces. Overall, 24 neurons in our sample (Table 2) showed a mixed selectivity for both boundary stimuli and either the restricted or the large surface.
Mixed selectivity for 3-D boundaries and surfaces. Example neuron showing robust selectivity for the correlated vertical, decorrelated and solid 3-D shape, as well as for the restricted surface. Same conventions as in Figure 3. The vertical calibration bar on the right indicates 150 spikes/sec.
Selectivity for horizontal compared with vertical 3-D shapes
The second goal of the present study was to determine whether TEs neurons are selective for second-order disparities along the horizontal axis. We tested all 104 3-D shape-selective neurons with a concave and a convex horizontal 3-D shape, in which both surface and boundary were curved in depth along the horizontal axis.
Figure 10A shows four example neurons illustrating all possible combinations of vertical and horizontal selectivity. The first neuron on the left (a) is selective for vertical but not for horizontal 3-D shapes, the second (b) is selective for both directions of curvature, the third (c) was selective for horizontal but not for vertical. Finally, column d shows a neuron for which the selectivity was inverted for horizontal (concave) compared with vertical (convex). Table 3 shows the relative numbers of neurons selective for vertical and horizontal 3-D shape. Note that because the horizontal 3-D shape was not included in the search test, the numbers of neurons mentioned do not necessarily provide a reliable estimation of the actual proportions of neurons in TEs. A total of 29 neurons responded selectively to the horizontal 3-D shape pair. Of 16 neurons selective for the vertical and the horizontal 3-D shape, nine were selective in the same direction (as in Fig. 10A,column b), and seven showed the inverted selectivity (Fig.10A, column d). The NRD for the vertical 3-D shape pair is plotted against the NRD for the horizontal 3-D shape pair in Figure 10B, for all neurons selective for either the vertical or the horizontal 3-D shape pair (N= 74). Most neurons lie either below the diagonal (a andd) or in the top left quadrant (c). Thirty neurons in our sample showed no significant response differences for either of the two directions of curvature alone (Table 3), but were selective for the restricted surface. The latter neurons seemed to require the presence of two disparity gradients in the stimulus.
Selectivity for vertical and horizontal 3-D shapes of the population. A, Four example neurons showing selectivity for the vertical but not for the horizontal 3-D shape (a), for both vertical and horizontal 3-D shape (b), for horizontal but not for vertical 3-D shape (c), and for vertical but the inverted selectivity for horizontal (d). Same conventions as in Figure 3. B, Scatterplot of the NRD for the horizontal 3-D shape plotted as a function of the NRD for the vertical correlated 3-D shape, separately for the neurons selective (circles) or not selective (squares) for the horizontal 3-D shape. The letters indicate the subsets illustrated in A.
Relative numbers of neurons selective for the correlated vertical 3-D shape compared with the number of neurons selective for the horizontal 3-D shape
We tested whether TEs neurons can signal the direction of curvature for their preferred 3-D structure, e.g., can signal whether a convex 3-D shape is vertical or horizontal. We compared the largest response of the neuron (either to the vertical or to the horizontal 3-D shape) to the response evoked by the same disparity variation along the orthogonal axis. The large majority of the neurons (53 of 74; 72%) showed significant differences between the responses to vertical and horizontal 3-D shapes (t test; p < 0.05). Note that 58 neurons were selective for one direction of curvature but not for the orthogonal direction (Table 3) and that most of these neurons differentiated between the two directions of curvature.
Figure 11 shows the normalized population PSTH for all neurons selective for the horizontal 3-D shape pair (N = 29). As for the boundary and the surface-selective neurons, the response was already selective in its initial part. The median SI for the horizontal 3-D shape in this population of neurons equaled 0.74, which is comparable to the SI for the correlated vertical 3-D shape (0.75) and for the restricted surface stimulus (0.77). The bottom graph plots the mean eye position difference for concave and convex horizontal 3-D shape for those neurons for which binocular eye movements were recorded (N = 7). A small vergence response was detected, but the neuronal selectivity was already present before the change in eye position could have any effect. Taken together, our results clearly show that the basic selectivity for horizontal 3-D shapes is present in TEs.
Vertical and horizontal selectivity.A,B, Normalized population PSTHs for preferred (black) and nonpreferred (gray) horizontal 3-D shape, for all neurons selective for the horizontal 3-D shape (N = 29;A), and for those horizontal neurons for which binocular eye movements were recorded (N = 7;B). C, The mean difference in horizontal eye position for concave (black) and convex (gray) horizontal 3-D shape. The vertical gray line is a reference at 200 msec after stimulus onset. Thevertical scale bar indicates 1°.
To determine whether the disparity variation along the horizontal boundary was sufficient for 3-D shape selectivity, we tested seven neurons with correlated and decorrelated horizontal 3-D shapes, as well as the monocular presentations of these stimuli. Only one cell showed a significant difference in response to the decorrelated 3-D shape pair (post hoc LSD test; p < 0.025). Moreover, the selectivity for the correlated 3-D shapes never reflected a selectivity for the monocular images per se (data not shown). Thus, the selectivity for the horizontal 3-D shape is based mainly on a selectivity for the disparity variation along the surface of the shape.
DISCUSSION
We studied the 3-D shape selectivity of TEs neurons by presenting 3-D shapes in which either the boundary or the surface were curved in depth, together with 3-D shapes in which both boundary and surface were curved in depth. The results revealed a substantial selectivity for 3-D boundaries in a subpopulation of TEs neurons. In the same recording area, neurons could also be selective for stimuli in which the disparity varied only within the surface of the shape (surface selectivity). Surface- and boundary selectivity could even be observed within single neurons (mixed selectivity). Additionally, we demonstrated that TEs neurons can also be selective for horizontal 3-D shapes and can code the direction of curvature (vertical or horizontal).
Zero-order disparity (position-in-depth) selectivity based on the boundary has been demonstrated in striate and extrastriate visual areas (Burkhalter and Van Essen, 1986; Felleman and Van Essen, 1987; Poggio et al., 1988; Roy et al., 1992), including the inferior temporal cortex (Uka et al., 2000). The stimuli in these studies were solid figure stereograms (either bars or shapes) that contained disparity only along their boundaries. Boundary-based first-order disparity selectivity (i.e., linear disparity variations) has been shown in the caudal intraparietal sulcus (Taira et al., 2000). Some neurons in this area were selective for the orientation of a surface in depth when the disparity varied along only the boundary of a square plate, whereas other neurons responded selectively when the disparity varied over the surface in a random dot stereogram. Our data provide the first evidence for a selectivity for boundaries curved in depth. Because the overwhelming majority of TEs neurons selective for concave and convex 3-D shapes respond to second-order disparity variations (Janssen et al., 2000b), our data demonstrate selectivity for boundary-based second-order disparities. Because it is difficult to present zero- and first-order stimuli that contain disparity only along their surface (i.e., without the disparity-defined boundary present in a random dot stereogram), our data also provide the first evidence for a pure surface-based disparity selectivity.
IT neurons respond selectively to 2-D shapes (Gross et al., 1972;Desimone et al., 1984; Tanaka et al., 1991), even if different cues define the same contours (Sary et al., 1993; Tanaka et al., 2001) or when these contours are partially occluded (Kovacs et al., 1995). It is widely accepted that the 2-D contours of an object are encoded in the firing of small populations of IT neurons (Tanaka, 1996). Our data complement this view in the 3-D domain by showing that a substantial part of the 3-D shape selectivity in TEs is based on the boundary curvature of the stimulus. On the other hand, IT neurons respond selectively to color and texture (Komatsu et al., 1992; Komatsu and Ideura, 1993), which are typical surface characteristics. Likewise, our data clearly demonstrate that a large proportion of TEs neurons is influenced mainly by the disparity curvature along the surface. A surface curved in depth was sufficient to elicit 3-D shape selectivity in a large proportion of the neurons. Moreover, even boundary neurons were frequently influenced by the surface: decorrelating the dots between the left and right eye gave a significant effect in the majority of the boundary neurons, and more than half of the boundary neurons were selective for both 3-D boundaries and surfaces. Because our 3-D surfaces contained both a vertical and a horizontal disparity variation, whereas the boundary stimuli were curved only along the vertical axis, the proportion of mixed neurons is likely to be an underestimation of the real incidence of mixed selectivity in TEs. Thus, rather than being encoded by separate populations of neurons, the neural representations of surface and boundary are intimately related at the single cell level. Note that for 2-D stimuli, surface and boundary provide independent sources of information that can be randomly combined, whereas in our 3-D stimuli, both surface and boundary signaled the same 3-D structure.
The large surface was included in the test to determine the extent to which the monocular contours of the stimulus were necessary for 3-D shape selectivity. A sizable proportion of the neurons responded as selectively to the large surface as to the restricted surface, indicating that the monocular contours were not critical. This conclusion was further confirmed by the response behavior in the search test, in that some neurons showed similar 3-D shape-selectivity for every 2-D shape in the test. This subpopulation of neurons therefore exhibits strong 3-D shape-selectivity but weak or absent 2-D shape-selectivity. Such neurons could encode differences in 3-D structure within planar surfaces, e.g., a groove or protrusion on a wall. Most TEs neurons, however, responded less strongly to the large surface than to the restricted surface. In the search test, these neurons displayed at least some selectivity for 2-D shape, which, on average, was not significantly different from lateral TE (Janssen et al., 2000a). The presence of a larger background region frequently inhibits the responses of IT neurons to luminance-defined 2-D shapes (Missal et al., 1997). Therefore, neurons that failed to respond selectively to the large surface may have been unable to segment the disparity-defined 3-D surface from the large textured background.
In most real-world objects, boundary and surface have the same 3-D structure, but some objects are only sparsely textured. The recognition of such objects could be facilitated by extracting disparity variations along their boundaries. Ramachandran et al. (1973) and Mayhew and Frisby (1976) showed that depth can be derived from the contours of texture-defined patterns in which the surface dots of the right- and left-eye images are uncorrelated. This manipulation is the zero-order analog of our decorrelated 3-D shape. Wilcox et al. (2000)showed that human observers are remarkably accurate in detecting disparity curvature in sparsely textured surfaces, even at short exposure durations, and Vreven et al. (2001) demonstrated that human subjects can extract depth from textureless curved surfaces, which is similar to our solid 3-D shape. Consistent with these psychophysical observations, TEs neurons showed substantial selectivity for 3-D boundaries emerging rapidly after stimulus onset.
Koenderink (1990) has proposed that local surface patches can be fully characterized by their curvedness, i.e., the degree of curvature, and by their shape index, which is related to the sign of the principal curvatures, one of which runs along the vertical axis and one running along the horizontal axis. A sphere, for example, has positive curvature in all directions and a shape index of 1, whereas a cylinder, with no curvature along its axis and maximum curvature in the orthogonal direction, has a shape index of 0.5. Shape index and curvedness define a parametric shape space in which every possible type of local curvature can be situated. Koenderink's shape space represents a potentially useful metric for studying the neural coding of 3-D objects or object parts. Many TEs neurons were selective for either vertical or horizontal disparity variations, and a substantial proportion of those neurons were able to signal the direction of curvature in the stimulus (vertical or horizontal). A large subpopulation of neurons responded selectively only to the restricted surface, in which the curvature had the same sign in both directions. The 3-D preference for vertical seemed to be independent of the preference for horizontal, in that some neurons showed the inverted selectivity for the horizontal compared to the vertical 3-D shape. We may speculate that these neurons actually preferred a saddle shape, which is convex along one axis and concave along the orthogonal axis. Hence, although no attempt was made to explore the shape space systematically, our results suggest that TEs neurons exhibit the basic selectivity necessary to encode surfaces according to their shape index.
We have shown that both boundaries and surfaces curved in depth are represented by TEs neurons, and that these neurons can signal the direction of curvature along a surface. Future research will systematically investigate the coding of 3-D surfaces in a parametric shape space.
Footnotes
This work was supported by the Fonds voor Wetenschappelijk Onderzoek (FWO) Grant G.0142.00, GOA 2000/11, and Geneeskundige Stichting Koningin Elisabeth. P.J. is a research fellow of the FWO. We thank M. Depaep, P. Kayenbergh, G. Meulemans, and G. Vanparrys for technical assistance, S. Raiguel for critical reading of this manuscript, and the Division of Radiology of the KU Leuven Medical School for help with the CT and MRI scans.
Correspondence should be addressed to Rufin Vogels, Laboratorium voor Neuro-en Psychofysiologie, Herestraat 49, B-3000 Leuven, Belgium. E-mail: Rufin.Vogels{at}med.kuleuven.ac.be.
