Temporal Processing Capacity in High-Level Visual Cortex Is Domain Specific

Experiment 1: fMRI temporal processing capacity experiment.

Subjects viewed gray-level stimuli, which consisted of two different types of five different domains: characters, bodies, faces, places, and objects (Fig. 1A, example stimuli in first row). Characters consisted of pseudowords (Glezer et al., 2009) and numbers. Bodies consisted of either whole bodies with obscured heads or limbs. Faces were either those of adults or children. Places were either indoor corridors or houses. Finally, objects were either cars or guitars. The view, size, and retinal position of the images varied. Each item was overlaid on a 10.5° phase-scrambled background generated from a randomly selected image from the entire set to minimize low-level differences across categories.

We introduce a novel paradigm in which the number of stimuli (information) in a trial is constant (eight stimuli per trial; Fig. 1B) and the presentation rate, and consequently trial duration, varies across conditions. Prior studies (Mukamel et al., 2004; McKeeff et al., 2007; Gentile and Rossion, 2014) used constant block duration, confounding the number of stimuli with rate: blocks with faster rates had more stimuli compared with blocks with slower rates. Consequently, with this design, it is impossible to disentangle two competing effects on fMRI responses: stimulus rate versus the number of stimuli. In contrast, our new design measures the effect of rate alone on cortical responses. In each trial, stimuli of one category were shown in rapid serial visual presentation (RSVP) at a single rate of 1, 2, 4, or 8 items per second (Hertz). Stimulus trials were counterbalanced with blank baseline trials. In each run, stimuli were shown at one rate. The order of runs at different rates was counterbalanced across subjects. The total duration of visual presentation at a particular rate was 11 min, which equated the statistical power across rates. The same set of 1440 images was shown across all rates, but images did not repeat within a rate.

Subjects fixated on a central dot and pressed a button when a phase-scrambled oddball image appeared randomly in a trial. Trials contained 0, 1, or 2 phase-scrambled images with probability of 0.25, 0.5, or 0.25, respectively.

Experiment 2: control study.

The design of the second experiment was identical to the main experiment, except that we modified the stimulus set (Fig. 1A, example stimuli in second row) to simultaneously control for multiple low-level differences between stimuli of different categories including their (1) contrast, (2) luminance, (3) similarity to other stimuli of the same category, (4) visual field coverage, and (5) spatial frequency power distributions. In doing so, we also considered the impact of image manipulations on high-level factors and made efforts to equate stimuli of different categories in terms of (6) familiarity and (7) memorability. Below, we detail the results from the quantification of these seven effects from Experiment 2.

1. Michelson Contrast: Calculated as the ratio between the difference and sum of the maximum and minimum pixel intensities, was matched across all images by scaling the grayscale values of each image such that 1% of pixels is saturated at the lowest and highest intensities, respectively. Further image processing by the SHINE toolbox (Willenbockel et al., 2010) and spatial filtering introduced a small amount of variability in contrast across images, but there is no significant difference in contrast level among images of different categories (no main effect of category, F_(4,1435) = 0.73, p > 0.05; Fig. 1C, Contrast).

2. Luminance: We used the SHINE toolbox (Willenbockel et al., 2010) to match histograms of grayscale values across all stimuli such that the mean luminance of images from different categories does not differ (no main effect of category, F_(4,1435) = 0.97, p > 0.05; Fig. 1C, Luminance).

3. Similarity: To estimate visual similarity among images of a category, we measured the mean Euclidean distance between normalized grayscale values (scaled to be between 0 and 1) of each stimulus and every other stimulus of the same category (Grill-Spector et al., 1999). The distance between a pair of images was calculated using the following formula in which p represents the pth pixel in an image, n represents the total number of pixels in an image, and I_p and I′_p represent the pth pixel in two different images: Similarity (S) was defined as follows: S = 1 − D. Thus, pairwise similarity values range from 0 (images with inverted intensity difference at each pixel) to 1 (identical images). Although there are numerically significant differences in mean similarity across categories (main effect of category, F_(4,101529) = 788.52, p < 0.001), the range of similarity values is similar across categories (Fig. 1C, Similarity), and differences in the mean values are not perceptually relevant because the size of the effect (0.0029, standard deviation (SD) of category means in normalized grayscale values) is below the resolution of the display (0.0039, minimum pixelwise distance resolved by the display in normalized values, corresponding to the change in one gray-level value of a possible 255 increments). Boxplots in Figure 1C, Similarity, show distributions of similarities for all possible pairwise comparisons in each category.

4. Visual field coverage: While we varied the view, size, and position of stimuli used in Experiment 1 to maximize variation within a category and reduce variation across categories, we further refined the cropping and placement of stimuli in the visual field in Experiment 2. This refinement further reduced biases related to intrinsic regularities in the shape and aspect ratio of different categories. Figure 1C, Pixel Intensity, shows the mean luminance of each pixel location, averaged across all images in a category.

5. Spatial frequency: Was controlled by filtering the images with a low-pass Gaussian filter with a visual angle cutoff frequency of 8 cycles/degree. After filtering, the mean spatial frequency distribution across images of each category was matched. Figure 1C, Spatial Frequency, shows mean proportion of total power in each spatial frequency, averaged across all images in a category.

6. Familiarity: To diminish familiarity effects across categories, all stimuli were selected to be equally unfamiliar to participants. None of the face or place stimuli were personally familiar to our participants, and pronounceable pseudowords and uncommon numbers were used as character stimuli instead of more familiar English words and frequently encountered numbers.

7. Memorability: We tested whether the memorability of stimuli belonging to different categories was matched by comparing recognition memory for stimuli of various categories presented at 1 Hz (see below, Behavioral testing). Performance did not significantly differ across categories at 1 Hz (no main effect of category, F_(3,31) = 0.79, p > 0.05).

Functional localizer.

All subjects participated in four runs of an independent functional localizer experiment (6 min per run) collected on a different day using 16 s blocks, 1 Hz presentation, and an oddball task (Weiner and Grill-Spector, 2010, 2011).

Data acquisition.

Subjects were scanned on a GE 3 tesla Signa scanner at the Center for Cognitive and Neurobiological Imaging at Stanford University using a custom-built, phase-array 32-channel head coil. We acquired 34 slices covering occipitotemporal cortex (resolution: 2.4 × 2.4 × 2.4 mm; one-shot T2*-sensitive gradient echo acquisition sequence: FOV = 192 mm, TE = 30 ms, TR = 2000 ms, and flip angle = 77°). A whole-brain, anatomical volume was acquired (T1-weighted BRAVO pulse sequence; resolution: 1 × 1 × 1 mm, TI = 450 ms, flip angle = 12°, 1 NEX, FOV = 240 mm).

Data analysis.

Data were analyzed with MATLAB using the mrVista toolbox (http://github.com/vistalab) as in our prior publications (Weiner and Grill-Spector, 2010, 2011).

Functional ROIs were defined in individual subjects from the functional localizer using anatomical and functional criteria (Weiner and Grill-Spector, 2010, 2012; Grill-Spector and Weiner, 2014). A common threshold (t > 4, voxel level) was used to define all regions across all subjects, and subsequent analyses were conducted in ROIs identifiable in at least 9 of the 12 subjects (ROIs from a representative subject are shown in Fig. 3A).

Character-selective ROIs (characters > others) included a bilateral region in the inferior occipital sulcus (IOS, N = 12), a bilateral region in the posterior occipital temporal sulcus (pOTS; N = 12), corresponding to the visual word form area (Cohen et al., 2000; Ben-Shachar et al., 2011), and a more anterior patch often lateralized to the left hemisphere localized to mid OTS (mOTS; N = 9). In our prior studies, we use anatomical nomenclature with the corresponding selectivity to refer to face-, place-, and body-selective ROIs (e.g., mFus-faces). Here, we extend this nomenclature to character-selective regions. Thus, we use the label pOTS-characters for visual word form area 1 (VWFA-1) and mOTS-characters for VWFA-2.

Body-selective ROIs (bodies > others) were found bilaterally in the lateral occipital sulcus (LOS; N = 12), inferior temporal gyrus (ITG; N = 12), and middle temporal gyrus (MTG; N = 12) as in Weiner and Grill-Spector (2011), collectively corresponding to the extrastriate body area (Downing et al., 2001). An additional body-selective region was defined in the OTS (N = 9), referred to elsewhere as the fusiform body area (Peelen and Downing, 2005; Schwarzlose et al., 2005).

Face-selective ROIs (faces > others) were observed on the inferior occipital gyrus (IOG; N = 11), corresponding to the occipital face area (Gauthier et al., 2000), and on the posterior (pFus; N = 12) and mid (mFus; N = 12) lateral fusiform gyrus, collectively corresponding to the fusiform face area (Kanwisher et al., 1997).

Place-selective ROIs (places > others) were observed near the transverse occipital sulcus (TOS; N = 12) as described in previous studies (Hasson et al., 2003), collateral sulcus (CoS; N = 12), corresponding to the parahippocampal place area (Epstein and Kanwisher, 1998), and medial parietal/retrosplenial cortex (RSC; N = 10), corresponding to the retrosplenial complex (O'Craven and Kanwisher, 2000).

ROIs were considered in three anatomical sections arranged from posterior to anterior corresponding to putative stages of the visual hierarchy: LOTC (IOS-characters, LOS-bodies, IOG-faces, and TOS-places), posterior VTC (pOTS-characters, ITG-bodies, and pFus-faces), and mid VTC (mOTS-characters, OTS-bodies, mFus-faces, and CoS-places). Two additional regions were located outside these anatomical expanses (MTG-bodies and RSC-places).

Estimating responses to experimental conditions.

We ran a GLM using the experimental conditions to generate a design matrix that was then convolved with the HRF implemented in SPM (http://www.fil.ion.ucl.ac.uk/spm). This procedure was done on the time course of each voxel (units of percentage signal change) and separately for each rate. Response amplitudes (betas) and residual variance of each voxel were estimated from the GLM. The contrast effect size (CES) was measured as the difference in response amplitudes to a preferred category versus all other categories (in units of percentage signal change). Selectivity (t-value) was measured as the ratio of the CES versus its standard error. The standard error was estimated from the residual variance of the GLM without whitening (Worsley, 2001).

To examine the goodness of our model fits across rates, we compared the variance explained by the GLM fits in each voxel for each rate. Across both experiments, this analysis revealed no significant differences in the proportion of variance explained by models fit separately for 1 and 8 Hz data in any ROI (no significant differences, ts < 2.01, ps > 0.05, paired t test for each ROI comparing proportion of variance explained at 1 and 8 Hz; Fig. 2). Goodness of fit of the GLM tended to be highest at 4 Hz in most category-selective ROIs even though these regions were defined using a localizer with stimuli presented at 1 Hz. This fit at 4 Hz was significantly better than in other rates in some ROIs in Experiment 1 (main effect of rate, Fs > 2.99, ps < 0.05, one-way ANOVA on proportion of variance explained by GLM for each ROI; Fig. 2A, thick lines) but not in any ROI in Experiment 2 (no main effect of rate, Fs < 2.49, ps > 0.05; Fig. 2A,B, thin lines). These findings rule out the possibility that our results reflect differences in model fits between fast (8 Hz) and slow (1 Hz) presentation rates or between short (2 s) and long (8 s) trial durations.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Consistent GLM model fits between fast (8 Hz) and slow (1 Hz) presentation rates. The mean proportion of variance explained by the GLM fits in ROIs selective to characters, bodies, faces, and places in Experiment 1 (A) and Experiment 2 (B). Error bars indicate SEM across subjects. Regions in which there was a significant main effect of rate (p < 0.05) are plotted with thick lines. Results of paired t tests comparing the variance explained by 1 and 8 Hz GLMs in individual ROIs are summarized for sets of regions above each plot. These analyses rule out the possibility that any differences that we find across regions reflect differences in model fits between fast (8 Hz) and slow (1 Hz) presentation rates or between short (2 s) and long (8 s) trial durations.

Evaluating the effect of rate on category selectivity.

We measured the effect of rate on category selectivity using two different repeated-measures ANOVAs. Category selectivity was defined as the mean voxel t-value of preference to one category versus others, averaged across voxels in a given ROI. The first ANOVA was conducted to determine whether selectivity varies differentially as a function of rate across regions located in the same putative hierarchical tier. This ANOVA included factors of rate and preferred domain using data from subjects with all regions defined in a particular anatomical section for repeated-measures analyses (anatomical sections—LOTC: IOS-characters, LOS-bodies, IOG-faces, and TOS-places; posterior VTC: pOTS-characters, ITG-bodies, and pFus-faces; mid VTC: mOTS-characters, OTS-bodies, mFus-faces, and CoS-places; Fig. 3B). To determine the reproducibility of domain-specific effects of rate on selectivity after controlling low-level stimulus properties, we directly compared selectivity data from Experiments 1 and 2 for sets of regions in the same anatomical section using a three-way ANOVA with factors of rate, preferred domain, and experiment. The second ANOVA was conducted to determine whether selectivity varies differentially as a function of rate across regions with the same category preference, but in different hierarchical tiers (preferred categories—characters: IOS, pOTS, and mOTS; bodies: LOS, ITG, MTG, and OTS; faces: IOG, pFus, and mFus; places: TOS, CoS, and RSC; Fig. 4).

To gain a finer-grained estimate of the presentation rate producing maximal selectivity in a given region within the range of rates measured in the present study, we next fitted third-order polynomial functions to the temporal frequency tuning curve of each ROI in individual subjects as in previous studies (McKeeff et al., 2007; Gauthier et al., 2012). We then identified the rate corresponding to the local peak of the function within the 1–8 Hz range in which we presented our stimuli (Table 1, Selectivity). We compared the distributions of these local maxima across regions in the same anatomical section and across regions with the same category preference using a series of one-way ANOVAs and data from all subjects.

Evaluating the effect of rate on response amplitude.

Selectivity is affected by (1) differences in response amplitudes across conditions and (2) the noise level. Consequently, we measured the effect of rate on response amplitudes to preferred and non-preferred categories in each ROI and the effect of rate on the CES or the difference in amplitude between preferred and non-preferred stimuli. We first determined whether the effect of presentation rate on response amplitude differs for preferred and non-preferred stimuli in sets of regions preferring the same category using a series of three-way repeated-measures ANOVAs with factors of rate, preference, and ROI separately for Experiment 1 (Fig. 5) and Experiment 2 (Fig. 7). We then tested whether this interaction significantly differed between experiments using a series of four-way ANOVAs with factors of rate, preference, ROI, and experiment. Likewise, we determined the consistency of rate modulations for preferred stimuli across experiments using a three-way ANOVA with factors of rate, ROI, and experiment. For Experiment 2, we also examined if the effect of rate on response amplitudes in a given ROI varied across each individual stimulus domain using a two-way repeated-measures ANOVA with factors of rate and domain for each ROI (Fig. 6).

The same curve-fitting procedures and statistical analyses used for selectivity data were also applied to amplitude data to estimate the rate eliciting peak responses to the preferred category (Table 1, Preferred) as well as peak CES (Table 1, CES) for each ROI. To further quantify these effects, we compared these additional metrics of temporal capacity across regions processing different domains in the same hierarchical tier and across regions processing the same domain in different hierarchical tiers using a series of one-way ANOVAs. Results using amplitudes to the preferred category or CES (Table 1, CES) were similar (no significant differences, t₍₁₂₎ = 0.50, p > 0.05, paired t test comparing estimated peak rates derived from amplitude to the preferred category and CES computed at the subject level and then averaged across subjects for each ROI in Experiment 1). Therefore, we focus our analysis on response amplitudes to the preferred category in the main text, but present all data in Table 1.

Finally, to assess the level of consistency between different neural measures of temporal capacity within and across experiments, we compared the capacity estimates derived from various metrics (peak amplitude to the preferred category, peak CES, and peak selectivity) in Experiments 1 and 2 among sets of regions located in the same anatomical section using a series of three-way ANOVAs with factors of preferred domain, experiment, and metric. Complementary analyses were then performed for sets of regions with the same category preference using three-way ANOVAs with factors of ROI, experiment, and metric.