Predicting the Partition of Behavioral Variability in Speed Perception with Naturalistic Stimuli

Measuring natural stimuli

A fundamental problem of perception is that multiple proximal stimuli can arise from the same distal cause. This stimulus variability is an important source of uncertainty that limits human and ideal speed discrimination performance. To measure natural stimulus variability, we photographed a large number of natural scenes (Burge and Geisler, 2011, 2015), and then drifted those photographs at known speeds behind a 1° aperture, approximately the size of foveal receptive fields in early visual cortex (Gattass et al., 1981, 1988). This procedure generates motion signals that are equivalent to those obtained by rotating the eye during smooth tracking of a target (Spering et al., 2005; Osborne et al., 2007) (Fig. 2A). The sampled set of stimuli approximates, but almost certainly underestimates, the variability present in the natural stimulus ensemble; looming and discontinuous motions, for example, are not represented in our training set (Schrater et al., 2001; Nitzany and Victor, 2014). Thus, the forthcoming estimates of the impact of natural stimulus variability on ideal and human performance are likely to underestimate the impact of stimulus variability on human performance in natural viewing.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Naturalistic image movies and preprocessing. A, Naturalistic image movies were obtained by drifting photographs of natural scenes at known speeds behind 1° apertures for 250 ms. Rotating the eye in its socket (e.g., tracking an object) creates the same pattern of motion in the stationary background. Optical properties of the eye and the temporal integration of the photoreceptors were also modeled. B, Full space-time image movies (Ixyt) and vertically filtered space-time image movies (Ixt). Moving images can be represented as oriented signals in space-time. C, Vertically oriented receptive fields respond identically to full space-time movies and vertically filtered movies.

Movies drifted leftward or rightward with speeds ranging between 0.25 to 8.0°/s. Movies were presented for 250 ms, the approximate duration of a typical human fixation. The sampling procedure yielded tens of thousands of unique stimuli (i.e., image movies) at dozens of unique speeds. Image movies were then filtered so that only vertical orientations were present; that is, the stimuli were vertically averaged (i.e., xt) versions of full space-time (i.e., xyt) movies (Fig. 2B). Vertical averaging reduces stimulus complexity, but the resulting stimuli are still substantially more realistic than classic motion stimuli, such as drifting sinewaves. Furthermore, vertically oriented receptive fields respond identically to vertically averaged and original movies (Fig. 2C). Thus, in an individual orientation column, the filtered movies should generate the same response statistics as the full space-time movies (Burge and Geisler, 2015; Jaini and Burge, 2017). Finally, the contrasts of the vertically averaged stimuli were fixed to the modal contrast in natural scenes (see Discussion). Thus, our stimuli represent a compromise between simple and real-world stimuli, allowing us to run experiments with more natural stimuli without sacrificing quantitative rigor and interpretability. Our analysis should be generalizable to full space-time movies with more realistic forms of motion.

Measuring early noise

All measurement devices are corrupted by measurement noise. The human visual system is no exception. Early measurement noise occurs at the level of the retinal image and places a fundamental limit on how well targets can be detected. Possible sources of early noise include the Poisson variability of light itself and the stochastic nature of the photoreceptor and ganglion cell responses (Hecht et al., 1942). The ideal observer for speed discrimination should be constrained by the same early noise as the human observer if it is to provide an accurate indication of the theoretically achievable human performance limits (Fig. 1A).

Human observers performed a target detection task using the equivalent input noise paradigm (Burgess et al., 1981; Pelli, 1985). The task was to detect a known stationary target embedded in dynamic Gaussian white noise. On each trial, human observers viewed two stimuli in rapid succession and tried to identify the stimulus containing the target (Fig. 3A,B). The time course of stimulus presentation was identical to the forthcoming speed discrimination experiment. Figure 3C shows psychometric functions for target detection in 1 human observer as a function of target contrast. Each function corresponds to a different noise contrast. Detection thresholds, which are the target contrasts required to identify the target interval 76% of the time (i.e., d′ of 1.0 in a 2IFC task), are shown for two different targets (3.0 and 4.5 cpd) in Figure 3D. Consistent with previous studies, contrast power at threshold increases linearly with pixel noise (Burgess et al., 1981; Pelli, 1985). Figure 3E shows the same data plotted on logarithmic axes, a common convention in the literature. There are two critical points on this function. The first is its value when pixel noise equals zero, where detection performance is limited only by internal noise. The second is at double the contrast power of the first point: the so-called “knee” of the function, where the pixel noise equals the internal noise. This level of pixel noise is known as the equivalent input noise. The knee of the function, and thus the estimate of equivalent input noise, is robust to whether or not the observer is using a detector (e.g., receptive field) that is optimal for detecting the target.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Measuring early noise with a target detection experiment. A, Stimulus construction. On each interval, the stimulus was either a stationary target Gabor stimulus or a middle gray field corrupted by dynamic noise. B, On each trial, the task was to report which of two intervals contained the target stimulus. C, Psychometric functions from 1 human observer (S1) for detecting a 3 cpd target, in noise having different RMS contrasts (0.00, 0.05, 0.10, 0.20). D, Threshold target contrast power for the same human observer. Thresholds increase linearly with noise contrast power. Error bars indicate 95% bootstrapped CIs; many error bars are smaller than the symbols. E, Target contrast power at detection threshold plotted on a log-log axis (same data as D) for all three observers. Arrows indicate the estimate of equivalent input noise.

The equivalent input noise was estimated separately for each target type and human observer. Estimates were consistent across target types and were thus averaged. Noise estimates for the first, second, and third human observers are 2.5%, 2.3%, and 2.9%, respectively (Fig. 3E). These values are in line with previous reports (Burgess et al., 1981; Pelli, 1985; Williams, 1985).

The estimates of equivalent input noise may reflect the exact amount of early measurement noise alone (Pelli, 1991). The estimates of equivalent input noise may also reflect the combined effect of early measurement noise and noise arising at later processing (e.g., decision) stages. Regardless of which possibility is correct, the target detection experiment provides an upper bound on the amount of early noise in the human visual system. The ideal observer used in the main text is limited by early noise at this upper bound. Because the upper bound is small, early noise only weakly impacts ideal observer performance (see below).

Ideal observer

An ideal observer performs a task optimally, making the best possible use of the available information given stimulus variability and specified biological constraints. In addition to natural stimulus variability and early noise (Figs. 2, 3), we model the optics of the eye (Wyszecki and Stiles, 1982; Thibos et al., 1992), the temporal integration of photoreceptors (Schneeweis and Schnapf, 1995), and the linear filtering (Hubel and Wiesel, 1962) and response normalization (Albrecht and Geisler, 1991; Heeger, 1992; Carandini and Heeger, 2011) of cortical receptive fields. These are all well-established features of early visual processing and determine the information available for subsequent processing.

Assuming that the relevant factors have been accurately modeled, ideal observers provide principled benchmarks against which to compare human performance. Given the information available to a particular stage of processing, ideal observers allow the researcher to ask whether subsequent processing stages use that information as well as possible. Humans often track the pattern but fail to achieve the absolute limits of ideal performance. As a consequence, ideal observers often serve as principled starting points for determining additional unknown factors that cause humans to fall short of theoretically achievable performance limits.

Developing an ideal observer with natural stimuli is challenging because it is unclear a priori which stimulus features are most useful for the task. We find the optimal receptive fields for speed estimation using a recently developed Bayesian statistical learning method called Accuracy Maximization Analysis (Geisler et al., 2009; Burge and Jaini, 2017; Jaini and Burge, 2017). Given a stimulus set, the method learns the receptive fields that encode the most useful stimulus features for the task (Fig. 4A). Once the optimal features are determined, the next step is to determine how to optimally pool and decode the responses R = [R₁, R₂, …, R_n] of those receptive fields where n is the total number of receptive fields. Eight receptive fields capture essentially all of the useful stimulus information; additional receptive fields provide negligible improvements in performance (Burge and Geisler, 2015).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Ideal observer receptive fields (RFs), response distributions, computations, and estimates. A, Optimal space-time RFs for speed estimation given the naturalistic stimulus set and biological constraints. B, RF response distributions for RFs 1 and 2, conditioned on the speed of the image movie (colors). Each symbol represents the joint response to an individual movie. The variability of responses for each speed (color) is due to natural stimulus variability; that is, it is the nuisance stimulus variability in the feature space defined by the optimal RFs. C, Computations of a hypothetical neuron implementing optimal encoding and pooling. Each noisy, contrast-normalized stimulus is processed by the optimal RFs. The responses of these RFs are pooled in a weighted quadratic sum. The weights are determined by the response covariance in B corresponding to the neuron's preferred speed. The response of this hypothetical neuron represents the likelihood that a given stimulus had its preferred speed. The optimal pooling rules thus represent an LNLN (linear, nonlinear, etc.) cascade. D, Speed-tuning curves of hypothetical neurons implementing optimal encoding and pooling, whose responses represent the likelihood of each speed given a stimulus. The speed-tuning curve R̄^L (X_u) is the average likelihood across stimuli at each of many different speeds. Shaded regions represent ± SD CIs on response. This response variability is due to natural stimulus variability. E, An arbitrary stimulus creates a population response R^L over hypothetical speed-tuned neurons. Optimal decoding yields the optimal estimate. F, Ideal observer estimates. The optimal estimate is read out from the population of hypothetical speed-tuned neurons in E, and is equivalent to reading out the posterior probability distribution p(X|R) over speed. The variance of ideal observer speed estimates (histogram) is dominated by stimulus-driven variance.

The optimal pooling rules are specified by the joint statistics relating the latent variable and the receptive field responses (Bishop, 2006; Jaini and Burge, 2017). With appropriate response normalization, the responses across stimuli for each speed are conditionally Gaussian (Lyu and Simoncelli, 2009; Burge and Geisler, 2015; Sebastian et al., 2017; Iyer and Burge, 2019) (Fig. 4B). To obtain the likelihood of a particular speed, the Gaussian response statistics require that the receptive field responses to a given stimulus be pooled via weighted quadratic summation (Fig. 4C). The computations for computing the likelihood thus instantiate an enhanced version of the motion-energy model, indicating that energy-model-like computations are the normative computations supporting speed estimation with natural stimuli (Adelson and Bergen, 1985; Jaini and Burge, 2017). The speed tuning curves of hypothetical neurons implementing these computations are approximately log-Gaussian, similar to the approximately log-Gaussian speed tuning curves of neurons in area MT (Nover et al., 2005) (Fig. 4D). Finally, an appropriate readout of the population response of these hypothetical neurons is equivalent to decoding the optimal estimate from the posterior probability distribution p(X|R) over speed (Fig. 4E,F). If a 0, 1 cost function is assumed, the latent variable value corresponding to the maximum of the posterior is the optimal estimate. We have previously verified that reasonable changes to the prior and cost function do not appreciably alter the optimal receptive fields, pooling rules, or performance (Burge and Jaini, 2017). This approach provides a recipe for how to construct neurons that are highly invariant to nuisance stimulus variability and tightly tuned to speed. It also provides a normative justification, grounded in natural scene statistics, for descriptive models proposed to account for response properties of neurons in cortex (Adelson and Bergen, 1985; Simoncelli and Heeger, 1998; Perrone and Thiele, 2001; Nover et al., 2005; Rust et al., 2006; Jaini and Burge, 2017).

The factors thus far described in the paper — stimulus variability and early noise, biological constraints, and the optimal computations (encoding, pooling, decoding) — all impact ideal performance in our task. Given a particular stimulus set, the only factor subject to some uncertainty is the precise amount of early noise. However, within the bound set by the detection experiment (Fig. 3), different amounts of early noise have only a minor effect on ideal performance (see below). Thus, estimates of ideal performance are set overwhelmingly by stimulus variability.

Measuring efficiency

The ideal observer benchmarks how well humans use the stimulus information available for the task. Efficiency quantifies how human sensitivity d′_human compares with ideal observer sensitivity d′_ideal and is given by the following: where σ_human² is the total variance of the human decision variable, σ_ideal² is the total variance of the ideal decision variable, and σ_E² is the stimulus-driven component of the ideal decision variable. The third approximate equality in Equation 9 assumes that stimulus-driven variability equals ideal observer variability because the impact of early noise is bounded to be small (compare Fig. 3).

To measure human sensitivity, we ran a 2IFC speed discrimination experiment. On each trial, human observers viewed two moving stimuli in rapid succession, and indicated which stimulus was moving more quickly (Fig. 5A). This design is similar to classic psychophysical experiments with one critical difference. Rather than presenting the same (or very similar) stimuli in each condition hundreds of times, we present hundreds of unique stimuli one time each. This stimulus variability jointly limits human and ideal performance. Human sensitivity is computed using standard expressions from signal detection theory d_human^′ = , where PC_human is the proportion of times that the comparison is chosen in a given condition in a 2IFC experiment and Φ⁻¹(·) is the inverse cumulative normal. (This expression is correct assuming the observer uses the optimal criterion, an assumption that is justified by the data.)

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Measuring speed discrimination. A, The task in a 2IFC experiment was to report the interval containing the faster of two naturalistic image movies. Unlike classic psychophysical studies, which present the same stimuli hundreds of times, the current study presents hundreds of unique stimuli one time each. This design injects naturalistic stimulus variability into the experiment. Human responses are assumed to be based on samples from decision variable distributions (inset). B, Ideal observer estimates across hundreds of standard (red) and comparison movies (white) at one standard speed (3°/s) and four comparison speeds. C, Human versus ideal observer sensitivity for all standard and comparison speeds. Shaded regions mark regions of plot where humans are less efficient than ideal but are still performing the task. For all conditions, humans are less sensitive than the ideal observer by a single-scale factor: efficiency: d_human^′ = . Negative d′ values correspond to conditions in which the comparison was slower than the standard. D, Psychometric functions for 1 human observer (symbols) at five standard speeds. The degraded ideal observer (solid curves) matches the efficiency of the human observer (one parameter fit to human data). E, Human speed discrimination thresholds (d′ = 1.0) as a function of standard speed for 3 human observers (symbols) on a semilog plot. The pattern of human thresholds matches ideal observer thresholds (solid curve). Vertically shifting the ideal observer thresholds by an amount set by each human's efficiency (arrows) shows degraded observer performance (solid curves, one free parameter fit per human).

To measure ideal sensitivity, we ran the ideal observer in a simulated experiment with the same stimuli as the human. (The ideal observer was trained on different stimuli than the human and ideal observers were tested on.) Ideal sensitivity (i.e., d′) was computed directly from the distributions of ideal observer speed estimates in each condition (Fig. 5B). Human and ideal sensitivities across all speeds are linearly related (Fig. 5C). Rearranging Equation 9 shows that human sensitivity d_human^′ = equals the ideal observer sensitivity degraded (scaled) by the square root of the efficiency. Thus, a single free parameter (efficiency) relates the pattern of human and ideal sensitivities for all conditions. The efficiencies of the first, second, and third human observers are 0.43, 0.41, and 0.17, respectively.

Transforming the sensitivity data back into percent comparison chosen shows that the details of the degraded ideal nicely account for the human psychometric functions (Fig. 5D). The psychometric functions can be summarized by the speed discrimination thresholds (d′ = 1.0; 76% correct in a 2IFC task). The pattern of human and ideal thresholds match; the proportional increases of the human and ideal threshold functions with speed are the same (Fig. 5E). These results quantify human uncertainty σ²_human and show that an ideal observer analysis of naturalistic stimuli predicts the pattern of human speed discrimination performance, and replicate our own previously published findings (Burge and Geisler, 2015).

Together, the ideal observer and speed discrimination experiment reveal the degree of human inefficiency (i.e., how far human performance falls short of the theoretical ideal). But they cannot determine the sources of this inefficiency. Humans could be inefficient because of late noise (i.e., stochastic internal sources of variability arising after early noise). Humans could also be inefficient because of fixed suboptimal computations. If inefficiency is due exclusively to late noise, stimulus variability must equally limit human and ideal observer performance. If human inefficiency is partly due to suboptimal computations, stimulus variability will cause more stimulus-driven variability in the human than in the ideal. How can human behavioral variability be partitioned to determine the sources of inefficiency in speed perception? To do so, additional experimental tools are required.

Predicting and measuring decision variable correlation

A double-pass experiment, when paired with ideal observer analysis, can determine why human performance falls short of the theoretical ideal. In a double-pass experiment (Burgess and Colborne, 1988; Gold et al., 1999; Li et al., 2006), each human observer responds to each of a large number of unique trials (the first pass), and then performs the entire experiment again (the second pass). Double-pass experiments can “unpack” each point on the psychometric function (Fig. 6A,B), providing far more information about the factors driving and limiting human performance than standard single-pass experiments. The correlation in the human decision variable across passes (decision variable correlation) is key for identifying the factors that limit performance and determine efficiency (Burgess and Colborne, 1988; Sebastian and Geisler, 2018).

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

Decision variable correlation and response repeatability in a double-pass experiment. A, Psychometric data from the first human observer and cumulative Gaussian fit plotted as proportion comparison chosen versus d′ for the standard speed of 1°/s (same data as in Fig. 5D). B, Schematic for visualizing decision variable correlation across passes when standard and comparison speeds are identical (e.g., both equal 1°/s). Samples correspond to individual double-pass trials (small circles). The value of each sample indicates the difference between the estimated speeds of the comparison and standard stimuli on each trial. Decision variable values corresponding to response agreements and disagreements fall in white and gray quadrants, respectively. Decision variable distributions with the decision variable correlation predicted by efficiency (solid ellipse) and by the null model with a decision variable correlation of zero (dashed ellipse). Decision variable correlation depends on the relative importance of correlated and uncorrelated factors across passes. Stimuli are correlated on each repeated trial of a double-pass experiment; internal noise is not. Criteria on each pass (vertical and horizontal lines, respectively) are assumed to be optimal and at zero. C, Predicted response counts (bars) for each response type (−−, −+, +−, ++) across passes (100 trials per condition) given the decision variable correlation shown in B. D, Proportion of trials on which responses agreed across both passes of the double-pass experiment as a function of proportion comparison chosen for 1 human observer. Agreement data (symbols) and prediction (solid curve) assuming that efficiency predicts decision variable correlation (i.e., that all human inefficiency is due to late noise). The null prediction assumes that the decision variable correlation across passes is zero (dashed curve). The agreement data are predicted directly from the efficiency of the human observer (zero free parameters). Error bars indicate 68% bootstrapped CIs on human agreement. Shaded regions represent 68% CIs from 10,000 Monte Carlo simulations of the predicted agreement data assuming 100 trials per condition.

The power of this experimental design is that it enables behavioral variability to be partitioned into correlated and uncorrelated factors. Factors that are correlated across passes (e.g., the stimuli) increase the correlation of the decision variable across passes. Factors that are uncorrelated across passes (e.g., internal noise) decrease decision variable correlation. If the variance of the human decision variable is dictated only by stimulus-driven variability, decision variable correlation will equal 1.0. If the variance of the human decision variable is dictated only by internal noise, decision variable correlation will equal 0.0. If both stimulus-driven variability and internal noise play a role, the correlation will have an intermediate value.

Decision variable correlation, like the decision variable itself, cannot be measured directly using standard psychophysical methods. Rather, it must be inferred from the repeatability of responses across passes in each condition. The higher the decision variable correlation, the greater the proportion of times responses agree (i.e., repeat) in a given condition (Fig. 6B,C).

In each condition, we used the pattern of response agreement to estimate decision variable correlation (Fig. 6B,C), and then plotted agreement against the proportion of times the human observer (symbols) chose the comparison stimulus as faster (Fig. 6D). Human response agreement implies a decision variable correlation that is significantly different from zero. For the seven conditions shown in Figure 6D (i.e., all comparison speeds at the 1°/s standard speed), the maximum likelihood fit of decision variable correlation across the seven comparison levels is 0.43. Thus, 43% of the total variance in the human decision variable is due to factors that are correlated across repeated presentations of the same trials.

How should the estimate of decision variable correlation be interpreted? Human decision variable correlation across passes is given by the following: where σ_E² is the variance of the speed estimates due to external (i.e., stimulus) factors, σ_I² is the variance due to internal factors (e.g., noise), and σ_human² is the total variance of the human speed estimates. Decision variable correlation is driven by stimulus variation because the stimuli are perfectly correlated across passes.

The estimated decision variable correlation is strikingly similar to the efficiency measured for each observer. Although the exact relationship between decision variable correlation and efficiency depends on the source of human inefficiency, the fact that they are similar is no accident. Under the hypothesis that all human inefficiency is due to noise (i.e., stochastic internal factors that are uncorrelated with the stimuli), stimulus variability must impact human and ideal observers identically: the stimulus-driven variance in the human speed estimates (σ_E² in Eq. 10) will equal the stimulus-driven variance in the ideal observer speed estimates (σ_E² in Eq. 9). Plugging Equation 9 into Equation 10 shows that, under the stated hypothesis, human decision variable correlation equals efficiency as follows: This mathematical relationship has important consequences. It means that the estimate of human efficiency from the speed discrimination experiment (Fig. 5C) provides a zero-free parameter prediction of human decision variable correlation in the double-pass experiment (Fig. 6). The behavioral data confirm this prediction. Human efficiency in the discrimination experiment quantitatively predicts human response agreement in the double-pass experiment (Fig. 6D, symbols vs solid curve). The implication of this result is striking. It suggests that natural stimulus variability equally limits human and ideal observers and thus that the source of human inefficiency is due near-exclusively to late noise. Human speed discrimination is therefore optimal, except for the impact of late internal noise.

These results generalize across all conditions and human observers. Figure 7A shows measured response agreement versus proportion comparison chosen for the first human observer in each of the five standard speed conditions. Figure 7B plots measured response agreement against efficiency-predicted agreement, summarizing the agreement data for each human observer across all standard speeds; prediction uncertainty given the number of double-pass trials in each condition is shown as 95% CIs (shaded regions). The decision variable correlations that best account for the response repeatability across all conditions of the first, second, and third human observers are 0.45, 0.43, and 0.18, respectively. For the first 2 observers, stimulus-driven variance and noise variance have approximately the same magnitude. For all observers, the data are consistent with the hypothesis that decision variable correlation equals efficiency (solid curves), and data are not consistent with the null model in which decision variable correlation equals zero (dashed curves). Figure 7C plots decision variable correlation against efficiency for each human observer. Efficiency tightly predicts decision variable correlation for all three human observers, with zero additional free parameters.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Predicted versus measured response agreement and decision variable correlation. A, Proportion response agreement versus proportion comparison chosen for all five standard speeds (1–5°/s) for the first human observer. Human data (symbols) and predictions (curves) are shown using the same conventions as in Figure 6D. B, Measured versus predicted response agreement for all conditions and all human observers (symbols). Human agreement equals efficiency-predicted agreement for all three human observers (solid line). Shaded regions represent 95% CIs on the prediction from 1000 Monte Carlo simulations. Efficiency-predicted agreement for the null model, which assumes decision variable correlation is zero, is also shown (dashed curve). C, Decision variable correlation versus efficiency for each human observer (symbols). Human efficiency, measured in first pass of the speed discrimination experiment, tightly predicts human decision variable correlation in the double-pass experiment with zero free parameters. Error bars indicate 95% bootstrapped CIs on human efficiency and on human decision variable correlation. Shaded regions represent the expected relationship between efficiency and decision variable correlation if humans use fixed suboptimal computations (i.e., suboptimal receptive fields). Red brackets indicate uncertainty about the precise value of efficiency due to uncertainty about the precise amount of early noise (Fig. 3). Solid and dashed black lines indicate the best-fit regression lines, corresponding to receptive field correlations of 0.97 and 0.92, respectively.

These results must be interpreted with some caution. Uncertainty about the amount of early measurement noise can cause uncertainty about human efficiency (Eq. 8) and thus about the predicted decision variable correlation (Eq. 11). We simulated ideal observers with different amounts of early noise and computed efficiency for each human observer (Fig. 8A). Fortunately, the detection experiment establishes an upper bound on the amount of early noise for each human observer (compare Fig. 3), thereby constraining the uncertainty about the predicted decision variable correlation (Fig. 8B, red brackets). Because the upper bound on early noise is low, the maximum and minimum possible efficiencies differ by ∼10%, depending on whether early noise at the upper or lower bound is assumed (Fig. 8A,B, red brackets). The measured decision variable correlation values (Fig. 8C) are in line with the predictions. Thus, uncertainty about the amount of early noise has only a minor impact on the interpretation of our results.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Early noise, efficiency, and predicted decision variable correlation. A, Efficiency in speed discrimination for each human observer (symbols) changes as a function of the amount of early noise modeled in the ideal observer. If early noise is negligible, efficiency is given by η = σ_E²/σ_human² (Eq. 9). If early noise is non-negligible, efficiency is given by η = (σ_E² + σ_{I, early}²)/σ_human² (Eq. 8). Red brackets and shaded regions indicate the minimum and maximum human efficiencies, given the bound on early noise established by the detection experiment (compare Fig. 3). B, Predicted decision variable correlation for each human observer given the uncertainty about human efficiency. The maximum (solid line) and minimum (dashed line) predicted decision variable correlations correspond to ideal observers having the maximum and minimum amount of early noise. The predicted decision variable correlations differ by ∼10% at maximum. C, Measured decision variable correlation for each human observer. Error bars indicate 95% bootstrapped CIs.

In the best performing observers, natural stimulus variability accounts for nearly half of all behavioral variability, despite the fact that the naturalistic stimulus set used to probe speed discrimination performance almost certainly underestimates the importance of stimulus variability in natural viewing (see Discussion). External variability therefore shapes the optimal computations, dictates the pattern of human performance, and predicts the partition of behavioral variability (i.e., the relative importance of external and internal sources of variability). These findings motivate continued efforts to model and characterize how natural stimulus variability impacts neural and perceptual performance in natural tasks.

Suboptimal computations

Human efficiency equals human decision variable correlation (Figs. 7C, 8B,C). To confidently conclude from this result that human inefficiency is almost entirely due to noise (i.e., stochastic internal sources of variability), we must rule out the possibility that suboptimal computations can produce the same result. How do fixed suboptimal computations impact the relationship between efficiency and decision variable correlation? To answer this question, one must determine how suboptimal computations impact the stimulus-driven component of the decision variable. To do so, we analyzed the estimates of a degraded observer that suboptimally encodes stimulus features (Burgess et al., 1981; Dosher and Lu, 1998; Neri and Levi, 2006; Sebastian and Geisler, 2018). If the wrong features are encoded, informative features may be missed, irrelevant features may be processed, and the variance of the stimulus-driven component of the decision variable may be increased relative to the ideal.

To create suboptimal feature encoders (i.e., suboptimal receptive fields), we corrupted the optimal receptive fields with fixed samples of Gaussian white noise (Fig. 9A). Receptive field correlation (i.e., cosine similarity) quantifies the degree of suboptimality ρ_f = f_opt^Tf_subopt/(‖f_opt‖ ‖f_subopt‖) where f_opt and f_subopt are the optimal and suboptimal receptive fields, respectively. Compared with the responses of the optimal receptive fields, the responses of these suboptimal receptive fields segregate less well as a function of speed (Fig. 9B). We generated degraded observers with suboptimal receptive fields having different receptive field correlations and examined estimation performance (Fig. 9C). We found that the stimulus-driven variance ασ_E² of the degraded observer estimates is a scaled version of the ideal stimulus-driven variance; the scale factor α = 1/ρ_f² is equal to the squared inverse of receptive field correlation (Fig. 9D). Thus, suboptimal receptive fields systematically increase the variance of the stimulus-driven component of the decision variable.

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

Relationship between suboptimal receptive fields and stimulus-driven variability in degraded observers. A, Optimal receptive fields (top; also see Fig. 4A) and suboptimal receptive fields from the degraded observer (bottom); only the first two receptive fields of each observer are shown. To obtain a suboptimal receptive field with a particular receptive field correlation ρ_f, we added fixed samples of Gaussian white noise to the corresponding optimal receptive field. The variance of the corrupting noise is given by σ_corrupt² = ((1/ρ_f²) − 1)/N, where N is the number of pixels defining each receptive field. B, Impact of suboptimal receptive fields on the conditional response distributions p(R|X). As the receptive fields become more suboptimal, the response distributions (colored ellipses) more poorly distinguish different values of the latent variable (colors). C, Effect of suboptimal receptive fields on degraded observer speed estimates for movies drifting at one speed (3°/s). As receptive field correlation decreases, the stimulus-driven variance of the estimates increases because informative stimulus features are not encoded and uninformative features are. D, The proportional increase of stimulus-driven SD for degraded versus the ideal observer estimates, assuming that the degraded observer has no late internal noise. Symbols plot the mean result from 100 Monte Carlo simulations. The stimulus-driven variance of the speed estimates increases with the squared inverse of receptive field correlation. E, Relationship between decision variable correlation and efficiency for degraded observers with different combinations of fixed suboptimal computations (i.e., receptive field correlations; gray levels) and internal noise. Points indicate mean decision variable correlation and mean efficiency from 100 Monte Carlo simulations of each degraded observer.

If humans are well modeled by a degraded observer with both suboptimal receptive fields and late noise, the total variance of the human estimates is given by σ_human² = ασ_E² + σ_I². Replacing terms in Equations 9 and 10 and performing some simple algebra show that the relationship between efficiency and decision variable correlation is given by the following: Thus, with suboptimal computations (i.e., receptive fields), decision variable correlation will be systematically larger than efficiency by the inverse square of receptive field correlation. (When receptive field correlation equals 1.0, Eq. 12 reduces to Eq. 11.) For example, if receptive field correlation is 0.5, decision variable correlation is 4× higher than efficiency. We verified the relationship between decision variable correlation and efficiency by simulating degraded observers with different levels of suboptimal computations and late noise (Fig. 9E). As predicted by Equation 12, the more suboptimal the computations (i.e., receptive field correlations), the more decision variable correlation exceeds efficiency. We reanalyzed our results in the context of Equation 12, comparing the behavioral data with the predictions of various degraded observer models. For all three observers, decision variable correlation is larger than efficiency by ∼5%, corresponding to a receptive field correlation of 0.97 (Fig. 7C). (These numbers assume an ideal observer with early noise set to the upper bound established by the detection experiment; Fig. 3). If no early noise is assumed, then decision variable correlation exceeds efficiency by 15%, corresponding to a receptive field correlation of 0.92 (Fig. 7C). Thus, no more than 15% of human inefficiency can be attributed to fixed suboptimal computations.

The simulations just described only consider the potential impact of fixed suboptimal computations that are linear. We cannot definitively rule out nonlinear suboptimal computations that leave stimulus-driven variability unchanged while selectively amplifying the impact of early noise, making amplified early noise indistinguishable from late noise. However, such computations are highly unlikely, given current knowledge of early visual processing. More importantly, suboptimal computations that selectively amplify early noise will not alter the predicted relationships between efficiency and decision variable correlation.

Thus, our results imply that the deterministic computations performed by the human visual system in speed estimation are very nearly optimal. Although natural stimulus (i.e., nuisance) variability is a major and unavoidable factor that limits performance in natural viewing, its impact is minimized as much as possible by the computations performed by the visual system.

Stimulus variability and behavioral variability

In this paper, we have shown that natural stimulus variability limits behavioral performance and drives response repeatability. Thus, reducing stimulus variability should increase sensitivity (i.e., improve behavioral performance) but decrease response repeatability. To test this prediction, we ran a new speed discrimination experiment using drifting random-phase sinewave gratings (Fig. 10). A stimulus set composed of drifting sinewaves has less variability than the set of naturalistic stimuli used in the main experiment. As predicted, with sinewave stimuli human sensitivity improves (Fig. 10A), responses become less repeatable (Fig. 10B), and decision variable correlation is lower (Fig. 10C). Interestingly, reducing stimulus variability affects decision variable correlation in the third human observer less than it does in the first two. This is the expected pattern of results given that the third observer (S3) had low decision variable correlation with naturalistic stimuli and was thus already dominated by internal noise (Figs. 7C, 8C). However, not all of the results were quite as expected. We anticipated that decision variable correlation would equal 0.0 for all three human observers with sinewave stimuli. But decision variable correlation exceeded 0.0 for each observer. What accounts for this discrepancy? We have ruled out commonly considered trial order effects (e.g., feedback-based effects) as the cause (Laming, 1979), but we are unsure of the cause. Whatever the case, with reduced stimulus variability, internal noise, which is uncorrelated across stimulus repeats, becomes the dominant source of variability limiting performance in all human observers.

• Download figure
• Open in new tab
• Download powerpoint

Figure 10.

Effects of reducing stimulus variability. A, Speed discrimination psychometric functions for the first human observer with naturalistic stimuli (black curve) and drifting sinewave stimuli (gray curve) for a 1°/s standard speed. Sinewave stimuli can be discriminated more precisely. B, Proportion response agreement versus proportion comparison chosen for naturalistic stimuli (black) and artificial stimuli (gray) for the same human observer. C, Decision variable correlation with artificial stimuli versus decision variable correlation with naturalistic stimuli for each human observer (symbols). Error bars indicate 95% bootstrapped CIs. Decision variable correlation is consistently lower when artificial stimuli are used.