Abstract
The perception of motion provides valuable interpolations of the visual scene. This fundamental capacity of the visual system is evident in apparent rotation: by presenting only two images of an object rotated in space, a vivid illusion of a smooth apparent motion in three dimensions can be induced. The unseen interpolated rotation views are filled in by the visual system. In the present study, we identified the cortical network responsible for this filling-in process. We argue that cross talk between areas of the ventral and dorsal visual pathways promote the illusion of smooth apparent rotation. Most interestingly, the network represents the unseen object views. Using functional magnetic resonance adaptation, we are able to show that the cortical network selectively adapts to the illusory object views. Our findings provide strong evidence for cortical representations of three-dimensional rotating objects that are view invariant with respect to the rotation path. Furthermore, our results confirm psychophysical investigations that unseen interpolated rotation views can be primed by apparent motion. By applying functional magnetic resonance adaptation, we show for the first time cortical adaptation to unseen objects. Together, our neuroimaging study advances the understanding of the cortical mechanisms mediating the influence of motion on object processing.
Introduction
In our natural environment, we frequently encounter objects in different rotated views. The perception of a rotating object might either be attributable to the object moving or to the observer moving. In both cases, different views of that object are revealed, whereas others become occluded. Kourtzi and Shiffrar investigated how different views of an object might be linked together by motion either in two dimensions (Kourtzi and Shiffrar, 1997) or in three dimensions (Kourtzi and Shiffrar, 1999). In the latter study, the authors made use of a visual illusion called “apparent rotation” (Shepard and Judd, 1976). If two views of the same three-dimensional (3-D) object are presented successively, they produce an illusion of rigid rotation: one object moving back and forth. Kourtzi and Shiffrar (1999) found that an apparent-rotation sequence of an object rotating in depth primed unseen views of that object. Most importantly, only views of that object were primed that lay on the rotation path of the object (interpolated positions). Views that lay outside the rotation trajectory (extrapolated positions) were not primed (Kourtzi and Shiffrar, 1997). These results were replicated in a recent study by McNamara et al. (2006). These studies provide evidence that apparent motion facilitates viewpoint-invariant representations of objects within the path of rotation.
The fact that motion primes unseen (interpolated) rotation positions suggests that the human visual brain represents these interpolated views. The goal of this study was to identify brain regions that are involved in the cortical representation of interpolated rotation positions. Using functional magnetic resonance imaging (fMRI), we aimed at localizing the motion-specific, viewpoint-invariant representations of rotating 3-D objects. To find brain regions that represent unseen rotation positions, we made use of cortical adaptation effects that are observable using the fMRI-adaptation paradigm (Buckner et al., 1998; Tootell et al., 1998; Grill-Spector et al., 1999) (for review, see Grill-Spector et al., 2006; Krekelberg et al., 2006).
We conducted three fMRI-adaptation experiments using an apparent-rotation sequence as the adapting stimulus that was followed by test stimuli that were rotated to different positions. We found regions of the occipito-temporal cortex [functionally defined as the lateral occipital complex (LOC)] (Malach et al., 1995; Kourtzi and Kanwisher, 2000), the transverse occipital sulcus (TOS), the lateral occipital sulcus (LOS), and the superior parietal cortex (SPC) that demonstrated adaptation of activation for stimuli that lay on the rotation path compared with stimuli that lay off the path. A fourth fMRI-adaptation experiment confirmed that these adaptation effects are truly based on the perceived motion and do not occur with static stimuli. Thus, we were able to identify a cortical network representing view-invariant representations of 3-D objects rotating in depth that are motion specific. These findings advance our understanding of complex visual illusions and the cortical mechanisms mediating the influence of motion on object processing.
Materials and Methods
Subjects.
Twenty-nine healthy subjects participated in this study (5 in experiment 1, 10 in experiment 2, 8 in experiment 3, and 6 in experiment 4). Subjects were students recruited from the University of Tübingen (experiment 1) and the University of Frankfurt (experiments 2–4). All subjects had normal or corrected-to-normal vision. The data from one subject had to be excluded because of inadequate slice positioning (experiment 1). Because of excessive head motion, the data of three subjects (one from experiment 2, two from experiment 3) had to be excluded. Before participating in this study, all volunteers gave their informed written consent to the procedure in accordance with institutional guidelines and the Helsinki declaration (www.wma.net/e/ethicsunit/helsinki.htm).
Stimuli.
Five different 3-D objects were created using the BrainVoyager 2000 software package (Brain Innovation, Maastricht, The Netherlands). Stimuli were comparable to those designed by Shepard and Metzler (1971). Each object consisted of 10 solid cubes attached face to face to form a rigid arm-like structure with three to four right-angled joints. Each object was asymmetrical around any possible axis. The objects were chosen to be relatively unfamiliar and meaningless in overall shape. They were presented as grayscale images in front of a black background. Five different perspective projections were generated for each object by 30° rotation steps around the vertical axis. These five objects, respectively their perspective projections, formed a first sample of stimuli. To create a second sample, each object of the first sample was rotated by 180° around the horizontal axis (see Fig. 1 for examples).
From this sample, additional stimuli were created for experiment 4. By superimposing two different perspective projections of one stimulus, double stimuli were created (see Fig. 1a, experiment 4 for examples). The overlapping area of the two stimuli was made transparent (10%) so that the two stimuli were still perceived as individual objects.
Visual stimulation.
Visual stimuli were delivered under computer control to a high-luminance liquid crystal display projector. The image was projected via a mirror onto a frosted screen that was positioned at the head end of the scanner (experiment 1) or directly projected on a screen that was fixed to the head coil (experiments 2 and 3). Subjects viewed both screens through a tilted mirror that was mounted onto the head coil. In experiment 4, stimuli were delivered via a video goggle system (VisuaStim Digital Glasses; Resonance Technology, Northridge, CA). Stimuli subtended 500 × 500 pixels, which resulted in a visual angle of approximately 8°.
Each experimental run began with a fixation point presented for 16 s, then followed an experimental block consisting of 109 trials 4 s each (90 experimental trials, plus 18 fixation trials, plus one extra trial; see below). At the end of each run, a fixation point was shown for 8 s. In summary, one experimental run lasted 7 min, 40 s (460 s). Subjects performed four to five experimental runs within one scanning session. Each experimental run had 18 trials per experimental condition.
Experiments 1–3.
An experimental trial began with the presentation of the apparent-rotation sequence (see Fig. 1a, experiments 1–3). The first frame was shown for 100 ms, followed by an interstimulus interval lasting 100 ms. Then, the second frame was presented for 100 ms. After the apparent-rotation sequence, a blank screen was displayed for 1000 ms, followed by the test stimulus, which was shown for 100 ms. The test stimulus was a repetition of the second frame (condition, “repeated”), a stimulus rotated to a position halfway between the two apparent-rotation frames (“interpolated”), a stimulus extrapolated in the direction of motion (“continuous extrapolated”), a stimulus extrapolated in the reverse direction of the path of motion (“reverse extrapolated”), or a novel stimulus that was equivalent to the interpolated stimulus of the corresponding object of the second sample of stimuli (“novel”). See Figure 1b for examples of the different test stimuli. After the presentation of the test stimulus, the screen went black again for 2600 ms (intertrial interval); then a new trial began. The direction of the apparent-rotation sequence was balanced across trials. The order of the presentation of the experimental trials was counterbalanced across runs per subject [i.e., trials from each condition (fixation included) were preceded (one trial back) equally often by trials from each other condition]. Because of this strategy, the first experimental trial (which had no history) had to be repeated at the end of the experimental block (extra trial). Taking all five experimental runs together, a full counterbalancing was achieved.
Experiment 4.
The timing of an experimental trial as well as the experimental conditions was the same in experiment 4 compared with experiments 1–3. Only frame 1 and frame 2 differed. Instead of an apparent-rotation sequence, subjects were presented two double stimuli (see Fig. 1a, experiment 4). The new frame 1 was a superposition of the original frame 1 onto frame 2, and the new frame 2 was a superposition of the original frame 2 onto frame 1. This presentation abolished the perception of motion between frames 1 and 2; subjects reported to see the stimuli as flickering.
Behavioral task.
In experiment 1, subjects had to indicate via button press whether the test stimulus depicted the same object compared with the object of the apparent-rotation sequence or a different one [object identification task (OIT)]. In experiment 2, subjects had to decide whether the test stimulus was rotated in the same direction of motion as the apparent-rotation sequence or in the other direction [direction of motion task (DMT)]. In experiment 4, subjects had to indicate whether the test stimulus depicted a part of the double stimulus or not. In all three experiments, subjects were instructed to maintain fixation on the red fixation dot, which was visible at all times during the entire experiment (see Fig. 1c). In experiment 3, subjects performed an attention-demanding task on a randomly generated stream of letters and digits (with a size of 0.8° visual angle) that was presented at 2 Hz at the center of the screen. They had to press a button whenever they detected a digit [attention control task (ACT)] (see Fig. 1c).
Data acquisition.
Blood oxygenation level-dependent (BOLD) fMRI was performed on 3T Siemens Trio (experiments 1–3) and Allegra (experiment 4) scanners (both from Siemens, Erlangen, Germany) equipped with birdcage head coils at two different locations (experiment 1, University Clinics, Tübingen; experiments 2–4, Brain Imaging Center, Frankfurt). Scanning procedures were kept similar, and sequence parameters differed only slightly between experiments. A gradient-recalled echo-planar imaging sequence was used with the following parameters: number of slices, 11; repetition time (TR), 1000 ms; echo time (TE), 40 ms; flip angle (FA), 60°; field of view (FOV), 210; slice thickness, 5 mm; in-plane resolution, 3 × 3 mm2; gap thickness, 1 mm (experiment 1). The slices were oriented to reach a total coverage of the occipital and temporal lobes (see Fig. 2). In experiments 2–4, the following parameters differed compared with experiment 1: number of slices, 16; TE, 30 ms; slice thickness, 6 mm. These parameters yielded a total coverage of the entire brain. Functional images were acquired in four to five experimental runs in a single session. Each run comprised the acquisition of 460 volumes. Stimulus presentation was synchronized with the fMRI sequence at the beginning of each run. Each scanning session included the acquisition of a high-resolution magnetization-prepared rapid-acquisition gradient echo sequence for coregistration and anatomical localization of functional data [TR, 2000 ms; TE, 4.38 ms; FA, 15°; FOV, 240; voxel size, 1 × 1 × 1 mm3 (experiment 1); TR, 1240 ms; TE, 2.6 ms; voxel size, 1 × 1 × 2 mm3 (experiments 2–4)].
Data preprocessing.
Data analysis and visualization was performed using the BrainVoyager QX software package (Brain Innovation). The first four volumes of each event-related run were discarded to preclude T1 saturation effects. Preprocessing of the functional data included (1) 3-D motion correction, (2) linear trend removal and temporal high-pass filtering at 0.01 Hz, and (3) slice scan-time correction with sinc interpolation. For each subject, the functional and structural 3-D data sets were transformed into Talairach coordinate space (Talairach and Tournoux, 1988).
The recorded high-resolution anatomy of one subject of experiment 1 and of all subjects of experiments 2 and 3 were used for surface reconstruction, which included gray/white matter segmentation based on intensity values. The cortical surfaces were slightly smoothed, inflated, and flattened (using a manual cut through the calcarine sulcus).
For experiments 2 and 3, a spatial correspondence mapping between subjects' brains was performed using the BrainVoyager QX cortex-based intersubject alignment tool. It has been shown that a cortical correspondence mapping substantially improves statistical analysis across subjects by reducing anatomical variability (Fischl et al., 1999; van Atteveldt et al., 2004; Goebel et al., 2006; Meienbrock et al., 2007). The procedure involved the reconstruction of the gray/white matter boundary of each individual hemisphere. The cortical surfaces were morphed into a spherical representation, and the hemispheres were then aligned based on the curvature information regarding the gyral/sulcal folding pattern. The target of the morphing procedure was a dynamical group average of all included hemispheres. This approach avoids the selection of a specific target hemisphere (Goebel et al., 2006). The correspondence mapping between the individual hemispheres was then used for the alignment of the functional data.
Data analysis.
Our rapid event-related fMRI study used closely spaced trials, leading to a substantial overlap in the resulting hemodynamic responses. Nevertheless, under the assumption of linearity, the underlying hemodynamic responses can be assessed by deconvolution (Dale and Buckner, 1997). A deconvolution analysis estimates the hemodynamic response function for each trial on the basis of a general linear model (GLM). Twenty predictors were defined to cover the temporal extent of a typical hemodynamic response. Because of the hemodynamic lag in the BOLD fMRI response, differences between conditions (as well as the peak in overall response) are expected to occur at a lag of several seconds after stimulus onset (Boynton et al., 1996; Cohen, 1997). On the basis of the deconvolved fMRI signal, we identified the peak points at lags 5–7 after trial onset. To localize regions in the brain that show a significantly lower activation to the repeated stimulus in contrast to the novel stimulus, a deconvolution-based GLM was computed at this peak point (contrast rep < nov; “basic adaptation effect”). The GLM included percent signal change normalization [i.e., the time course of a voxel or region of interest (ROI) is normalized in a way that the mean signal value will be transformed to a value of 100 and the individual values fluctuate around the mean as a percent signal deviations]. Thus, the reported GLM parameter estimates (β weights) directly provide an estimate of the actual percent signal change. Additionally, we provide averaged parameter estimates across peak points 5–7 s after trial onset and trials and subjects per condition as bar plots.
Multisubject statistical maps were thresholded using the false discovery rate (FDR) (Genovese et al., 2002), and q was set to 0.05 (experiments 1–3). For experiment 4, we performed a cluster-size thresholding to correct for multiple comparisons (Forman et al., 1995). We used a plug-in for Brain Voyager QX from F. Esposito (Brain Innovation, Maastricht, The Netherlands) that makes use of a Monte Carlo simulation (1000 iterations). Thus, the multisubject statistical map for experiment 4 is thresholded at p < 0.05, corrected for multiple comparisons.
To reveal an adaptation effect for the critical, interpolated condition, we looked at the detailed response profiles by individual ROI-based GLM analyses of the predefined regions. On each ROI, we performed five different pairwise comparisons: (1) contrasting interpolated versus novel (int < nov); (2) interpolated versus continuous extrapolated (int < conex); (3) interpolated versus reverse extrapolated (int < revex); (4) continuous extrapolated versus novel (conex < nov); and (5) reverse extrapolated versus novel (revex < nov). For ROI-based analyses, the data were normalized to percent signal change and corrected for serial correlation (Bullmore et al., 1996). We report t statistics and corresponding p values. Note that the first contrast (int < nov) is not independent of the contrast that defines the ROI (rep < nov) and is therefore only of descriptive use. Contrasts 2 and 3, however, are statistically independent of the ROI-defining contrast. For each ROI, we also extracted the event-related deconvolution time course of signal intensity (see Figs. 2⇓⇓–5) and provide GLM parameter estimates (see Figs. 3, 5; supplemental Figs. 1, 2, available at www.jneurosci.org as supplemental material).
Analysis for the comparison of experiments 2 and 3.
To reveal similarities and/or differences caused by the different tasks, we wanted to directly compare the results of experiment 2 (DMT) with results obtained in experiment 3 (ACT). We therefore matched the number of subjects from experiment 2 by randomly selecting six subjects out of the total population of nine. We then applied the same statistical analysis as in experiments 1 and 2 for the subpopulation of experiment 2 and for experiment 3 including cortex-based intersubject alignment and GLM group analyses on the peak points of the averaged time course (contrast repeated vs novel, rep < nov). On a low threshold of p < 0.05 (uncorrected), we investigated the response profiles of the overlapping regions of the functional activation maps for experiments 2 and 3, conducting ROI-based analyses for each experiment, respectively (DMT, ACT). As in experiments 1 and 2, we computed five different contrasts for each ROI and each experiment: (1) interpolated versus novel (int < nov); (2) interpolated versus continuous extrapolated (int < conex); (3) interpolated versus reverse extrapolated (int < revex); (4) continuous extrapolated versus novel (conex < nov); and (5) reverse extrapolated versus novel (revex < nov). Again, for ROI-based analyses, the data were normalized to percent signal change and corrected for serial correlation. We report t statistics and corresponding p values. For each ROI and experiment, we also extracted the event-related deconvolution time course of signal intensity (see Fig. 4) and provide GLM parameter estimates (see supplemental Fig. 2, available at www.jneurosci.org as supplemental material).
Eye movements.
We measured movements of the right eye of three subjects inside the magnet by pupil tracking under infrared illumination (MReyetracking system in conjunction with VisuaStim Digital Glasses, Resonance Technology; ViewPoint software, Arrington Research, Scottsdale, AR). The experimental setup was the same as in experiment 2; subjects were performing the DMT. Eye traces were analyzed using Matlab (The MathWorks, Natick, MA) and Statistica (StatSoft, Tulsa, OK). Processing involved linear trend removal of the horizontal and vertical eye-movement traces and detection of saccades and eye blinks exceeding 1.5° of visual angle and a duration of >150 ms. Eye movements that exceeded these limits were classified into blinks or saccades depending on a signal parameter that detected eye-lid closure (signal loss). The eye movements were assigned to the different stimulation conditions, and nonparametric statistics were performed to analyze whether there was a systematic change of eye-movement parameters between experimental conditions. The average number of saccades/blinks was 0.15/0.7 per trial. There were no significant differences between conditions in the number of saccades (χ2 = 6.515; df = 4; p = 0.164) or blinks (χ2 = 8.133; df = 4; p = 0.087). Furthermore, we plotted individual eye positions for each condition and participant (two-dimensional fixation-density plots) (supplemental Fig. 3, available at www.jneurosci.org as supplemental material). The brain imaging data were analyzed the same way as in experiments 1–4 up to the level where statistical contrasts were computed. Instead of first computing a basic adaptation contrast and performing ROI-based analyses to reveal the specific adaptation effects for the interpolated condition afterward, we used a conjunction analysis of the contrast rep < nov and the balanced contrast int < conex and revex. This procedure yielded two regions in the parietal cortex that demonstrated the specific adaptation profile. Supplemental Figure 4 (available at www.jneurosci.org as supplemental material) displays the contrast map and corresponding deconvolved signal time courses. In summary, we replicated the specific fMRI-adaptation effects in this group of three subjects for which we have recorded simultaneously eye-movement data. Thus, this control experiment is evidence that the observed adaptation effects in experiments 1 and 2 are not attributable to eye movements.
Results
Experiment 1
In the present study, we used an apparent-rotation sequence as the adapting stimulus that was followed by one of five (static) test stimuli. Figure 1 shows examples of the stimuli and a description of the experimental procedure and conditions (see Materials and Methods).
Experimental procedure. a, Schematic illustration of the experimental procedure demonstrating the timing of the experimental trial. An experimental trial began with the presentation of the apparent-rotation sequence. The first frame was shown for 100 ms, followed by an interstimulus interval lasting 100 ms (ISI 1). Then, the second frame was presented for 100 ms. After the apparent-rotation sequence, a blank screen was displayed for 1000 ms (ISI 2), followed by the test stimulus (Test), which was shown for 100 ms. After the presentation of the test stimulus, the screen went black again for 2600 ms [intertrial interval (ITI)]. b, Perspective projections of one object from the stimulus set used in the experiment. Two stimuli constituted the apparent rotation sequence (frames 1 and 2). The type of condition is defined by the test stimulus. A test stimulus was a repetition of the second frame (repeated; orange), a stimulus rotated to an interpolated position (interpolated; red), a stimulus extrapolated in the direction of motion (continuous extrapolated; blue), a stimulus extrapolated in the reverse direction of the path of motion (reverse extrapolated; turquoise), or a novel stimulus, which was equivalent to the interpolated stimulus of the corresponding object of the second sample of stimuli (novel; green). Stimuli were created using BrainVoyager 2000 (Brain Innovation). c, Schematic illustration of the different tasks for the three experiments. In experiments 1, 2, and 4 (Exp. 1,2,4), subjects were instructed to maintain fixation on a red fixation point that was present during the entire trial, while performing different tasks on the objects. In experiment 1, subjects had to indicate via button press whether the test stimulus depicted the same object compared with the object of the apparent-rotation sequence or a different one (OIT). In experiment 2, subjects had to decide whether the test stimulus was rotated in the same direction of motion as the apparent-rotation sequence or in the other direction (DMT). In experiment 4, subjects had to indicate whether the test stimulus depicted a part of the double stimulus or not. In experiment 3 (Exp. 3), subjects performed an attention-demanding task on a randomly generated stream of letters and digits that was presented at 2 Hz at the center of the screen. They had to press a button whenever they detected a digit (ACT).
The basic adaptation effect is defined by a decrease in fMRI signal to the test stimulus, which was identical to the second frame of the apparent-rotation sequence (repeated) in contrast to the test stimulus that is different from the frames of the motion sequence. Critically, a reduction in fMRI signal to the test stimulus that was rotated to an interpolated (extrapolated) position would indicate that this brain region represents unseen interpolated (extrapolated) rotation views.
The GLM group analysis (contrast rep < nov) revealed prominent regions bilaterally that demonstrate significantly less [q(FDR) = 0.05] activation for the repeated compared with the novel stimulus. Figure 2 shows the statistical map superimposed on inflated (a) and on flattened (b) surfaces of the reconstructed cortex of both hemispheres of one subject. By their anatomical location and Talairach coordinates (Talairach and Tournoux, 1988), these regions were identified as corresponding to subregions of the occipito-temporal cortex (LOC) (Malach et al., 1995; Kourtzi and Kanwisher, 2000), to subregions of the LOS and the TOS, to subregions of the inferior occipital gyrus (IOG), to subparts of the POS (parieto-occipital sulcus), to the insular cortex (insula), as well as to a region of the right VLPFC (ventrolateral prefrontal cortex).
fMRI results for experiment 1. a, Contrast map averaged across subjects showing regions responding significantly less to the repeated stimulus compared with the novel stimulus [contrast rep < nov; basic adaptation effect; q(FDR) = 0.05] The statistical map is superimposed on inflated cortical surfaces of the left and right hemisphere of one subject. Areas shaded in light gray were not covered because of slice positioning (see Materials and Methods). The sulci are coded in darker gray than the gyri. b, The same statistical map now superimposed on flattened cortical surfaces of the left and right hemisphere of the same subject. Major ROIs are circled and numbered. A white circle/number indicates that this region demonstrates significant adaptation effects (ROI-based analysis; p < 0.05) for stimuli that lay on the path of motion (repeated, interpolated) in contrast to stimuli that lay off the path of motion (novel, continuous extrapolated, reverse extrapolated). c, Event-related deconvolved BOLD fMRI responses (GLM parameter estimates averaged across trials and subjects for all voxels in each ROI) reported against time for each of the experimental conditions: orange, repeated; red, interpolated; blue, continuous extrapolated; turquoise, reverse extrapolated; green, novel. The ROI numbers correspond to b. Time point 1 is trial onset. Error bars indicate SEM. POS, Parieto-occipital sulcus; Insula, insular cortex; VLPFC, ventral lateral prefrontal cortex; ant, anterior; post, posterior; inf, inferior; sup, superior.
We investigated the functional response profiles of those ROIs that were defined by the basic adaptation effect (rep < nov) in a second stage of analysis. To clarify the amount of adaptation for the unseen interpolated rotation positions with respect to the control conditions (novel, continuous extrapolated, reverse extrapolated), we performed the following contrasts: interpolated versus novel (int < nov), interpolated versus continuous extrapolated (int < conex), and interpolated versus reverse extrapolated (int < revex). The response profiles of the respective regions were deconvolved in time and are shown as event-related time courses (Fig. 2c). GLM parameter estimates averaged across peak points are provided in supplemental Figure 1 (available at www.jneurosci.org as supplemental material).
We found a significant adaptation effect (p < 0.05) for the interpolated condition compared with the novel condition (int < novel) in 13 of 17 predefined ROIs (Table 1). More interesting, eight of these regions also demonstrated significant adaptation effects for the interpolated condition in contrast to the extrapolated conditions (int < conex and int < revex): LOC, LOS, LOS/TOS, and TOS of the left hemisphere as well as LOC anterior, IOG anterior, LOS/TOS, and insula inferior of the right hemisphere (Table 1).
Statistical analysis for experiment 1
To test whether these eight areas selectively demonstrate adaptation of activation only for the stimuli on the rotation trajectory of the object (repeated, interpolated) in contrast to the stimuli outside the trajectory (continuous extrapolated, reverse extrapolated), we also computed the contrasts continuous extrapolated versus novel (conex < nov) and reverse extrapolated versus novel (revex < nov). Only the subregion of the inferior insula of the right hemisphere demonstrated a significant (p < 0.05) adaptation effect for the continuous extrapolated stimulus compared with the novel stimulus (but no adaptation effect for the reverse extrapolated stimulus compared with the novel stimulus) (supplemental Table 1, available at www.jneurosci.org as supplemental material). The seven regions that show adaptation of activation for the repeated and interpolated but not for the continuous extrapolated and the reverse extrapolated conditions in contrast to the novel condition are marked with white circles in Figure 2b.
Together, these results reveal two dominant regions in either hemisphere, one within the ventral visual processing stream (LOC) and one within the dorsal stream (LOS/TOS), and a region in the right IOG that show adaptation of activation for stimuli that lay on the rotation path of the object (interpolated, repeated) compared with stimuli that lay off the path of motion of the object (continuous extrapolated, reverse extrapolated, novel).
Experiment 2
To confirm our findings and to control for possible alternative explanations, we conducted experiments 2 and 3. Visual stimulation, experimental setting, and scanning procedures were highly similar for experiments 2 and 3. The experiments differed only in the behavioral task subjects had to perform during scanning. Several studies (Eger et al., 2004; Murray and Wojciulik, 2004; Vuilleumier et al., 2005; Yi and Chun, 2005) have emphasized that attention affects adaptation of activation in the LOC. Because subjects performed an object identification task in experiment 1, we wanted to replicate an object-dependent task in experiment 2 that requires attending to motion features. Thus, we instructed our subjects to indicate whether the test stimulus was rotated in the same direction of motion as the apparent-rotation sequence or in the other direction (DMT). In experiment 3, in contrast, subjects had to perform a demanding task on a randomly generated stream of letters and digits that was presented at the fixation point (ACT). Furthermore, in experiments 2 and 3 (in contrast to experiment 1), we slightly varied scanning sequence parameters such that we achieved a total coverage of the brain (see Materials and Methods). In summary, we wanted to replicate our findings of experiment 1 in experiment 2. Furthermore, we controlled for task-specific effects (experiment 1 vs experiment 2) and attention-related effects (experiment 2 vs experiment 3).
To improve the spatial matching between individual subject's brains, we performed a cortex-based intersubject alignment of the anatomical and functional data in experiments 2 and 3 (see Materials and Methods). It has been shown that a cortical correspondence mapping substantially improves statistical analysis across subjects by reducing anatomical variability (Fischl et al., 1999; van Atteveldt et al., 2004; Goebel et al., 2006; Meienbrock et al., 2007).
We first computed a GLM analysis on the peak points of the averaged time courses (similar as in experiment 1) of all subjects using the contrast repeated versus novel (rep < nov). Significantly less activation [q(FDR) = 0.05] for the repeated stimulus compared with the novel stimulus (basic adaptation effect) was found again in subregions of the LOS/TOS in both hemispheres and in subregions of the LOC, a region of the posterior fusiform gyrus (pFG), a region of the SPC, a region in the postcentral sulcus, and a region in the SPFC (superior prefrontal cortex) in the right hemisphere. Figure 3 shows the statistical map superimposed on inflated (a) and flattened (b) surfaces of the reconstructed cortex of both hemispheres of one subject.
fMRI results for experiment 2. a, Contrast map averaged across subjects (using the cortex-based intersubject alignment approach; see Materials and Methods) showing regions responding significantly less to the repeated stimulus compared with the novel stimulus [contrast rep < nov; basic adaptation effect; q(FDR) = 0.05]. The statistical map is superimposed on inflated cortical surfaces of the left and right hemisphere of one subject. The sulci are coded in darker gray than the gyri. b, The same statistical map now superimposed on flattened cortical surfaces of the left and right hemisphere of the same subject. Major ROIs are circled and numbered. A white circle/number indicates that this region demonstrates significant adaptation effects (ROI-based analysis; p < 0.05) for stimuli that lay on the path of motion (repeated, interpolated) in contrast to stimuli that lay off the path of motion (novel, continuous extrapolated, reverse extrapolated). c, Event-related deconvolved BOLD fMRI responses (GLM parameter estimates, averaged across trials and subjects for all voxels in each ROI) reported against time for each of the experimental conditions and BOLD fMRI responses (GLM parameter estimates averaged across peak points 5–7 s after trial onset and trials and subjects for all voxels in each ROI) for each of the experimental conditions: orange, repeated; red, interpolated; blue, continuous extrapolated; turquoise, reverse extrapolated; green, novel. The ROI numbers correspond to b. Time point 1 is trial onset. Error bars indicate SEM. postCS, Postcentral sulcus; SPFC, superior prefrontal cortex.
In congruence with experiment 1, we also investigated the response profiles of those preselected regions of experiment 2 in an ROI-based analysis. We found adaptation of activation for the unseen interpolated position in contrast to the control conditions (novel, continuous extrapolated, reverse extrapolated) in five of eight ROIs: left TOS, right pFG, LOC, LOS/TOS, and SPC (Table 2). Furthermore, none of those areas showed an adaptation effect for the extrapolated conditions compared with the novel condition (p > 0.05). Figure 3c shows event-related time courses of the deconvolved MR signal and GLM parameter estimates averaged across peak points.
Statistical analysis for experiment 2
In summary, the results of experiment 2 nicely confirm the findings from experiment 1: subregions of LOC and LOS/TOS demonstrate a clear adaptation effect for the stimuli on the object's rotation trajectory in contrast to stimuli outside the motion trajectory. Because of the increased brain coverage in experiment 2 compared with experiment 1, an additional region of the SPC was also found to exhibit the specified adaptation effects. Apart from this highly overlapping cortical network, the involvement of different frontal regions seems to be related to the two different tasks applied in experiments 1 and 2. The OIT task elicited activity in the ventral prefrontal cortex, whereas the DMT activated the dorsal prefrontal cortex. This finding is in line with other investigations demonstrating that the ventral prefrontal cortex is involved in nonspatial tasks, whereas the dorsal prefrontal cortex is more involved in spatial tasks (Courtney et al., 1998; Munk et al., 2002).
Experiment 3
We conducted a third experiment to test for attention effects. While keeping stimuli the same as in experiments 1 and 2, subjects were instructed to perform an attention-demanding task on the fixation point in experiment 3 (ACT). To reveal similarities and/or differences caused by the different tasks, we wanted to directly compare the results of experiment 2 (DMT) with results obtained in experiment 3 (ACT). We therefore matched the number of subjects of both experiments by randomly selecting six subjects of the total population of nine for experiment 2. We then applied the same statistical analysis as in experiment 1 for the subpopulation of experiment 2 and for experiment 3 including cortex-based intersubject alignment and GLM group analyses on the peak points of the averaged time course (contrast repeated vs novel, rep < nov).
Figure 4 shows the statistical maps for both tasks (orange, DMT; purple, ACT) overlaid on inflated cortical surfaces of one subject. The functional activation maps are defined by the contrast repeated versus novel (rep < nov) and are shown at two different thresholds, q(FDR) = 0.05 (Fig. 4a) and p < 0.05 (uncorrected) (Fig. 4b). The statistical map for the subpopulation of experiment 2 reveals the same pattern of activity as for the total population of experiment 2; however, the statistical map for experiment 3 looks quite different: on a high threshold [q(FDR) = 0.05], only a small region in the right intraparietal sulcus demonstrates significantly less activity for the repeated compared with the novel condition (Fig. 4a). Lowering the threshold to p < 0.05 (uncorrected) reveals a patchy pattern of activity in mainly occipital, parietal, and temporal regions (Fig. 4a). On the lower threshold, both contrast maps overlap at four distinct regions: a region in the left LOS (1), a region in the left junction of the intraparietal and the transverse occipital sulcus (2), a region in the right transverse occipital sulcus (3), and a region in the SPC (4).
Comparison of fMRI results for experiments 2 and 3. a, Contrast maps averaged across subjects (using the cortex-based intersubject alignment approach; see Materials and Methods) for the object task (DMT; orange) and the center task (ACT; purple) showing regions responding significantly less [q(FDR) < 0.05] to the repeated stimulus compared with the novel stimulus (contrast rep < nov). The statistical maps are superimposed on inflated cortical surfaces of the left and right hemisphere of one subject. The sulci are coded in darker gray than the gyri. b, The same contrast maps as in a on a lower threshold [p < 0.05, uncorrected (uncorr.)]. Overlapping regions are marked with white lines and are numbered. c, Event-related deconvolved BOLD fMRI responses (GLM parameter estimates, averaged across trials and subjects for all voxels in each ROI) reported against time for each of the experimental conditions: orange, repeated; red, interpolated; blue, continuous extrapolated (cont. extrapolated); turquoise, reverse extrapolated (rev. extrapolated); green, novel. Always, two MR-signal time courses refer to the same ROI in the two different experiments. Time point 1 is trial onset. Error bars indicate SEM.
We also investigated the response profiles of those overlapping regions that were defined by the more liberal threshold by conducting ROI-based analyses for each experiment, respectively (DMT, ACT). Figure 4c displays event-related time courses of the deconvolved MR signal of the overlapping regions for each experiment (orange frame, DMT; purple frame, ACT). GLM parameter estimates averaged across peak points are provided in supplemental Figure 2 (available at www.jneurosci.org as supplemental material). For the DMT experiment, three of four ROIs demonstrated a significant adaptation effect (p < 0.05) for the unseen interpolated stimulus in contrast to the novel stimulus (Table 3). For the ACT experiment, however, none of the regions exhibited an adaptation effect for the unseen interpolated stimulus in contrast to the novel stimulus (Table 3). Three of four ROIs demonstrated a significant adaptation effect (p < 0.05) for the interpolated stimulus in contrast to the continuous extrapolated stimulus in the DMT experiment, whereas none of the regions showed this effect for the ACT experiment (Table 3). Furthermore, for the DMT experiment, two regions exhibited adaptation of activation for the interpolated stimulus in contrast to the reverse extrapolated stimulus, but no region did so in the ACT experiment (Table 3).
Statistical analysis for experiment 3
These results suggest that attention plays a major role for the integration of an apparent-rotation sequence. If attention is drawn away from the rotating objects to a demanding task on the fixation point, the cortical network normally involved in representing interpolated rotation positions does not show this specificity any longer.
Experiment 4
Experiment 4 was designed to confirm that the observed adaptation effects (experiments 1 and 2) were actually based on the perception of apparent motion and not simply because of conventional adaptation effects based on image similarity or overlapping tuning functions for specific rotation angles. We therefore manipulated the original visual presentation: instead of presenting an apparent-rotation sequence as the adapting stimulus, we presented two double stimuli. These stimuli were created by superimposing the original first frame on top of the original second frame of the apparent-rotation sequence and vice versa (Fig. 1a, experiment 4) (see Materials and Methods for a more detailed description of the procedure). The presentation of the two double stimuli no longer elicited the perception of apparent motion, but subjects reported to see two flickering objects.
We again computed a multisubject GLM analysis on the peak points of the averaged time courses (as in experiments 1–3) using the contrast repeated versus novel (rep < nov). An adaptation effect (p < 0.05, corrected) for the repeated stimulus compared with the novel stimulus (basic adaptation effect) was found in a region of the right LOC and in regions of the SPC. Figure 5a displays the statistical map superimposed on a sagittal and a coronal slice of one subject. As in experiments 1–3, we performed additional analyses on the so-defined ROIs (supplemental Table 2, available at www.jneurosci.org as supplemental material). None of the five different pairwise comparisons (int < nov, int < conex, int < revex, conex < nov, revex < nov) was significant for the region in the right LOC (p > 0.05). The two ROIs in the parietal cortex revealed a significant adaptation effect (p < 0.05) for the interpolated stimulus in contrast to the novel stimulus and for the continuous extrapolated stimulus in contrast to the novel stimulus. The region in the right parietal cortex additionally showed a significant adaptation effect (p < 0.05) also for the reverse extrapolated stimulus in contrast to the novel stimulus. However, most interesting, the ROIs in the parietal cortex also did not show an adaptation effect for the interpolated stimulus in contrast to either the continuous or the reverse extrapolated stimulus (p > 0.05). Figure 5b shows event-related time courses of the deconvolved MR signal and GLM parameter estimates averaged across peak points for the three ROIs.
fMRI results for experiment 4. a, Contrast map averaged across subjects showing regions responding significantly less to the repeated stimulus compared with the novel stimulus [contrast rep < nov; basic adaptation effect; p < 0.05, corrected (corr.)]. The statistical map is superimposed on a sagittal and a coronal slice of one subject. L, Left; R, right. b, Event-related deconvolved BOLD fMRI responses (GLM parameter estimates, averaged across trials and subjects for all voxels in each ROI) reported against time for each of the experimental conditions and BOLD fMRI responses (GLM parameter estimates averaged across peak points 5–7 s after trial onset and trials and subjects for all voxels in each ROI) for each of the experimental conditions: orange, repeated; red, interpolated; blue, continuous extrapolated (cont. extrap.); turquoise, reverse extrapolated (rev. extrap.); green, novel. The ROI numbers correspond to a. Time point 1 is trial onset. Error bars indicate SEM.
Thus, the elimination of the motion perception had a major impact on the previously described network. This findings support our conclusion from experiments 1 and 2 that the observed adaptation effects to the interpolated stimulus were actually based on the perceived apparent rotation.
Eye-movement control
In an additional control study, we recorded fMRI data for three subjects that performed the DMT and simultaneously recorded the eye position. We replicated the main fMRI-adaptation effects as presented in supplemental Figure 4 (available at www.jneurosci.org as supplemental material). Moreover, the analysis of eye-movement data (see Materials and Methods and supplemental Fig. 3, available at www.jneurosci.org as supplemental material) confirmed that the observed fMRI-adaptation effects cannot be explained by differences in eye movements between experimental conditions.
Discussion
The goal of our study was to investigate the effect of motion on the representation of 3-D objects. In apparent rotation, different views of an object are linked by (apparent) motion, thereby creating the illusion of a smooth rotational object motion. In the present fMRI study, we wanted to localize cortical regions that show motion-specific, view-invariant representations of rotating 3-D objects. We found a cortical network comprising areas of the ventral and dorsal processing streams (Ungerleider and Mishkin, 1982) of both hemispheres that showed adaptation to stimuli that lay on the rotation trajectory of the object, but no adaptation to stimuli that lay off that path. In particular, regions in the LOC and the TOS/LOS in both hemispheres and regions of the right IOG (rIOG) and of the SPC of only the right hemisphere showed the specified adaptation profiles. Thus, these areas seem to encode those views of a rotating 3-D object that are linked by motion. Our findings thus support psychophysical evidence (Kourtzi and Shiffrar, 1997, 1999; McNamara et al., 2006) showing that linking different views of a 3-D object through motion does not lead to a complete view-invariant representation of the rotating object but to a representation that is restricted to the rotation trajectory.
A highly similar cortical network has been found investigating the phenomenon of structure from motion (SFM). In SFM, moving dots evoke the perception of a 3-D object in the absence of any other visual object cue (Wallach and O'Connell, 1953; Treue et al., 1991). The cortical network representing SFM comprises areas in the dorsal visual pathway, namely the human motion complex (hMT+/V5), the intraparietal sulcus, and other parts of the parietal cortex and areas in the ventral visual pathway, predominately the LOC (Kriegeskorte et al., 2003; Murray et al., 2003). Thus, both motion- and object-selective regions are involved in the representation of SFM. SFM and apparent rotation are highly related because, in both cases, the perception of an object is generated through (apparent) motion. Thus, it is not surprising that a similar cortical network underlies both phenomena. We think that the interaction of dorsal and ventral areas in the visual processing hierarchy forms the basis for the perception of apparent rotation. But, in contrast to SFM, in which motion per se leads to the perception of an object, in apparent rotation the influence of motion is more subtle. In apparent rotation, motion processing influences one feature of a rotating object, namely its viewpoint.
Attention- and task-related effects
The focus of attention has an effect on the cortical network involved in representing the interpolated rotation positions. We observed several changes within the reported network when the focus of attention is drawn away from the objects (experiment 3). First, the level of cortical activation within the reported network is strongly reduced. Moreover, the specific effect of adaptation toward the interpolated position vanishes (i.e., above the overall reduced power, there is a specific effect on the differential activation pattern). Although we were still able to find areas that showed the basic adaptation effect, these areas no longer displayed adaptation to the interpolated position. This is in line with previous studies showing that it is essential for adaptation effects to occur that attention is directed to the objects under study (Eger et al., 2004; Murray and Wojciulik, 2004; Vuilleumier et al., 2005; Yi and Chun, 2005). If in our study the objects are not attended, the neuronal assemblies might not involve in the representation of the interpolated positions.
Despite the global influence attention has on adaptation effects, we are confident that the differential adaptation profile for the experimental conditions is not attributable to attention effects. First, observers were not able to anticipate the condition of the upcoming trial, because our event-related design used randomized presentation of trials from all conditions. Furthermore, we used two different object-related tasks leading to essentially similar results. This observation is in line with a study by Murray and Wojciulik (2004) demonstrating that, as long as attention is focused on the objects, the differential adaptation profile of a cortical region representing the attended objects is not affected by the type of task performed.
Motion specificity
In experiment 4, we eliminated the motion illusion, and this manipulation had two effects on the previously described cortical network representing the interpolated stimulus. First, only subparts of the original network (LOC and SPC) now still displayed the basic adaptation effect (adaptation of activation for the repeated stimulus in contrast to the novel stimulus). Second and most interesting, those regions no longer displayed the specific adaptation effects to the interpolated stimulus in contrast to the extrapolated stimuli. LOC only showed adaptation to the repeated stimulus and no adaptation to interpolated or extrapolated stimuli, whereas regions in the SPC displayed a pronounced adaptation effect to the repeated stimulus and also some adaptation for the interpolated as well as the extrapolated stimuli (all in contrast to the novel stimulus). We interpret these adaptation profiles as demonstrations of (static) viewpoint effects on the representations of objects in temporal and parietal regions. This result is in agreement with several other studies investigating the effect of viewpoint on static object representation (Grill-Spector et al., 1999; Gauthier et al., 2002; James et al., 2002; Vuilleumier et al., 2002). Most of the studies found viewpoint-independent representations for rotation angles up to 60° in ventral visual areas such as the LOC (Grill-Spector et al., 1999; James et al., 2002; Vuilleumier et al., 2002) (but see Gauthier et al., 2002). In contrast, object representation in the parietal cortex has been shown to be viewpoint dependent (James et al., 2002; Vuilleumier et al., 2002).
In the present study, we evaluated the effects of viewpoint on the representation of 3-D objects both during static presentation and during motion presentation. Thus, we are able to extend the previous findings by stating that the same regions showing static viewpoint effects and additional regions in the dorsal pathway (LOS/TOS, rIOG) display viewpoint effects that are motion specific (i.e., as soon as objects are seen in motion, those regions additionally code interpolated rotation positions). We did not observe a complete viewpoint-invariant representation of the object, but the invariance was restricted to views of the object that lay on its rotation trajectory. Thus, we conclude that a cortical network of regions in the temporal and parietal cortices displays viewpoint effects that are motion specific (Stone, 1999).
In conclusion, our study provides additional insights into the cortical representation of 3-D objects rotating in depth. More precisely, our four fMRI-adaptation experiments show that a cortical network of areas in the ventral and dorsal stream of the visual processing hierarchy subserves the on-line integration of motion-specific and view-specific features of rotating objects. Furthermore, our study expands the fMRI-adaptation method to the study of dynamical stimuli. To our knowledge, this is the first study combining a motion sequence and a static stimulus in a cross-adaptation approach to assess functional properties of brain regions processing dynamic stimuli. Adaptation experiments so far used either two motion sequences (Huettel et al., 2004; Krekelberg et al., 2005) or two static images (Grill-Spector et al., 1999; Kourtzi and Kanwisher, 2000, 2001; James et al., 2002; Vuilleumier et al., 2002; Kourtzi et al., 2003a,b; Valyear et al., 2006).
Footnotes
-
This work was supported by the German Research Foundation [Grants TH 812/1-1 (experiment 1) and MU 2358/1-2 (experiment 3 and 4)] and the Federal Ministry of Education and Research [Grant BMBF 01 GO 0508 (experiments 2–4)]. We declare that we have no competing financial interest. We are very grateful to Christian Altmann and Elisabeth Huberle for help with programming and data acquisition and to Nikos Logothetis for support. We thank Heike Ringeling for help with data analysis, Matthias Dhum for help with the eye tracking procedure, and Amanda Kaas and Oliver Doehrmann for helpful comments on this manuscript.
- Correspondence should be addressed to Sarah Weigelt, Department of Neurophysiology, Max Planck Institute for Brain Research, Deutschordenstrasse 46, D-60528 Frankfurt am Main, Germany. weigelt{at}mpih-frankfurt.mpg.de
