Humans Perceive Binocular Rivalry and Fusion in a Tristable Dynamic State

Observers.

Observers were 15 undergraduate students (10 female, 5 male) from Stanford University who received credit in an introductory psychology course for participation. All had normal or corrected-to-normal visual acuity, and exhibited normal binocular perception thresholds (50 s of arc or less at 16 inches as determined by a RANDOT test (Stereo Optical; Fawcett and Birch, 2000). All observers were naive to the purpose of the experiment. Prior written informed consent was provided according to a protocol approved by the Institutional Review Board of Stanford University.

Apparatus.

Stimuli were generated using MATLAB (The MathWorks) and MGL (Gardner et al., 2018). They were presented through a half-silvered pellicle haploscope (Planar, model SD2620W) that allowed for the images from two computer screens to be overlaid and seen exclusively by one eye each through polarized glasses. Observers sat in a darkened room one meter from the monitors, with perpendicular lines of sight to both. Monitors had a resolution of 1920 × 1200, a refresh rate of 60 Hz, and subtended 57.6° × 37.9° of visual angle. A keyboard resting on their lap was used to record responses.

Stimuli.

Single Gabor patches were presented separately to each eye, fusing into a vertically-oriented percept or engaging in rivalry depending on their orientation difference. They had a carrier spatial frequency of 5 cycles per degree and the Gaussian window had a standard deviation of 0.14° (see Fig. 1A). The patches were kept small in an effort to avoid incomplete rivalry, as with larger stimuli sub regions of a scene can be perceived through different eyes at the same time (Wilson et al., 2001; Kang et al., 2009). Psychophysical studies have shown that smaller stimuli are more likely to result in periods of complete dominance (Blake et al., 1992).

We provided a number of features to help observers maintain fixation on the Gabors with both eyes. A black fixation point marked the center of each patch, whereas three black circles from 0.5° to 0.7° in radius were shown in both eyes as a strongly-fusible reference frame. In each trial, two additional dots flanked the patches. These served as references for the true vertical or horizontal axes. All of the above features were present in both eyes. Nonius “v's” also emanated from the fixation point, pointing upwards in one eye and downwards in the other so that they formed a binocular “x” during proper vergence to further aid in maintaining fixation. The rest of the display was set to mean luminance (see Fig. 1A).

Experimental design.

Observers were shown pairs of monocular Gabor patches with orientation disparities ranging from 0° to 90°, and were asked to continuously report their perceived orientations over 1 min trials. In the main experiments, each stimulus was symmetrical about the vertical axis. Observers were told to hold down the left arrow key if the stimulus appeared rotated counter-clockwise from vertical and the right arrow key if it appeared rotated clockwise - tracking periods of perceptual dominance. The up arrow key was used to indicate a perfectly vertical orientation. This would indicate fusion, as for small disparities the stimuli would be seen as a single vertical Gabor tilted in depth about the horizontal axis. This tilt was not mentioned to observers, given that a fused Gabor should appear perfectly aligned with the reference dots and be reported as vertical. All buttons were to be released if the stimulus appeared patchy or of uncertain orientation, until it resolved into one of these three percepts. Observers responded with one of the three given keys 92.86% of the time, suggesting that patchy or mixed percepts only made up a small proportion of our results. After being given these instructions, observers were shown an example stimulus with 30° of disparity and asked to practice their responses to ensure they understood the task. They were encouraged to ask any questions they had about the procedure. After this practice period, each stimulus was shown for 1 min with the next stimulus pending a “ready” button press that allowed for a break between trials. Nine levels of orientation disparity (0, 10, 20, 25, 30, 35, 40, 50, and 90) were shown for 2 1 min trials each, swapped between eyes to control for ocular dominance. Shuffled, this produced an 18 min trial block. Sample trial records for three subjects at 0°, 30°, and 90° can be seen in Figure 1B.

To test whether fusion and rivalry could coexist in a tristable state without the involvement of stereopsis, another block of trials was run using a horizontal reference axis. All stimuli were the same, but rotated 90°. Gabor patches with small orientation disparities about the horizontal axis (i.e., vertical disparities) are seen as a single fused horizontal stimulus (Kertesz and Jones, 1970) but do not elicit a perception of depth (Cumming et al., 1991). This allowed fusion to be elicited in the absence of any depth cues from stereopsis. Observers were given the same instructions as before, but with the up arrow key now indicating a perfectly horizontal stimulus.

To better characterize the dynamics of tristable stimuli, each observer was given six more one-minute trials with the “most tristable” stimulus disparity from the vertical trial block, whose orientation we will refer to as that subject's tristable point. To determine the tristable point, the difference between the total duration of fusion and the mean of the two rivalrous durations was calculated for each disparity. The stimulus disparity with the smallest difference for each observer was chosen as their tristable point and presented three times with crossed- and un-crossed disparities in a 6 min trial block.

To confidently treat vertical reports as incidences of fusion, it was important to ensure that observers were not reporting all slightly-oriented stimuli as vertical. Each observer was given a simple orientation discrimination task at the beginning of the session to confirm their ability to distinguish small orientations. Observers were asked to classify a series of binocularly matched Gabor patches, otherwise identical to those in the main experiment, as rotated clockwise or counter-clockwise from vertical. Each was shown for 350 ms, then replaced by a random noise mask within the same Gaussian envelope to prevent observers from using iconic memory to compare subsequent stimuli (Sperling, 1960). This mask remained until the right or left arrow keys were used to indicate clockwise and counterclockwise judgments after each presentation. Forty trials were presented with orientations chosen according to a 3-down 1-up staircase beginning at 1.5° and advancing by steps of 0.25°. Our observers showed an average discrimination threshold (calculated by taking the mean of all of their performance reversals) of 2.65° of orientation, with a standard deviation of 1.00° and a total range of 1.28° to 4.36°. These results are consistent with previous work finding that orientation discrimination thresholds in humans generally lie at 2° of orientation or less (Heeley and Timney, 1988; Paradiso and Carney, 1988; Heeley and Buchanan-Smith, 1990). As the smallest nonzero orientations in our task were 10°, we conclude that vertical reports outside of the 0° condition indicate truly fused percepts rather than imperceptibly rotated monocular images.

Model simulations.

We tested a range of models where fusion and rivalry were separate bistable systems, each model simulated by sampling from the actual distributions of durations our observers reported. We also tested a single-process model that was a variant of an existing model of rivalry (Wilson, 2003). Fusion durations were taken from the eight total minutes of data recorded at the tristable point. Interfusion durations were sampled from the same data, where the total time elapsed between two periods of fusion was considered to be an interfusion period. The final periods in all trials were excluded from sampling as their durations were artificially truncated by the trial's ending. Rivalry durations were sampled from the 40°, 50°, and 90° disparity stimuli in both horizontal and vertical reference trial blocks. This was based on evidence that rivalry rates do not vary with orientation disparity (Wade, 1974), and so those trials would be a good representation of “underlying” rivalry at the tristable point. Some conflicting studies (Kitterle and Thomas, 1980; O'Shea, 1998) have found a decrease in rivalry alternations over time at smaller orientation disparities. However, “rivalry alternations over time” is not a direct measurement of percept duration and the observed decrease could actually result from the insertion of fusion periods—a possibility acknowledged by the authors. We used these duration distributions to simulate various possible ways the rivalry process could behave during periods of fusion.

Two-process stopping model.

One possibility is that the process of rivalry stops when fusion is perceived, and then continues on from where it left off after fusion subsides, what we call the “stopping model.” To simulate the stopping model, rivalry was first simulated by taking alternating samples from the left- and right-eye dominance duration distributions until they spanned at least 8 60 s periods. The same process was used to produce a series of fusion and interfusion durations. Then the fusion periods were placed between the rivalry periods, using the interfusion durations to space them. This effectively resulted in a rivalry process that was intermittently frozen during periods of fusion.

To relax this rather strict assumption that the rivalry process was completely stopped during fusion, we also built models in which rivalry was allowed to slow down during fusion. We tested 1.5× and 3× “slowdown” models. These models were produced in the same way as the stopping model, only with some portion of the rivalry series that occurred after the fusion insertion point removed to simulate that the rivalry process was continuing during fusion. The amount removed was related to the length of the fusion duration such that the ratio of fusion duration to rivalry removal was either 1.5× or 3× (e.g., 1 s of rivalry was skipped after a 3 s period of fusion for the 3× slowdown model).

Two-process continuation model.

The process of rivalry could also continue at a normal rate during perceived fusion, a model we term the “continuation model.” The same process as above was used to simulate this possibility, but the fusion was overlaid instead of interleaved into the rivalry. This can be thought of as a 1× slowdown model, where for each period of fusion an equally long period of rivalry was cut from the rivalry series. This simulated rivalry as a totally separate process carrying on unaffected underneath fused percepts.

Two-process interruption model.

The rivalry process might alternatively be completely disrupted during fusion, so that rivalrous percepts seen after periods of fusion are akin to those seen following new stimulus onsets. Psychophysical experiments have shown that following interruptions, rivalrous percepts tends to return to their prior states (Leopold et al., 2002; Pearson and Brascamp, 2008). Most relevantly for our paradigm, interruptions of rivalrous stimuli by fusible ones have demonstrated similar effects (Kanai et al., 2007). To capture these effects, we used a simple model previously developed to account for the early dynamics of rivalrous perception that lead to perceptual stabilization across interruptions (Noest et al., 2007). This model predicts the course of rivalry across a series of simulated “on” and “off” periods. It uses two variables per eye to do so: a “local field” value H representing the stimulus-related membrane potential of responding neurons, and a value A, which implements adaptation through shunting-style gain control. These variables were updated at each time step according to the following equations: In our case, H_i represents the response of neurons to the counter-clockwise-rotated stimulus, whose strength is X_i. The gain control term −(1+A_i) H_i serves to lower the response to the stimulus over time, with A_i approaching a sigmoidal transformation of H_i. βA_i is a critical term for breaking symmetry and producing the percept stabilization effects upon each initial presentation of the stimuli. Finally, −γS[H_j] is the cross-inhibition from the neurons responding to the competing stimulus. Aside from τ_A, we used all of the same parameter values as in the original paper: X = 1, α = 5, τ_H = , γ = , β = , as well as the same sigmoid function S(z > 0) = ; S(z ≤ 0) = 0. We set the initial H values to 0 and the initial A values to be low and asymmetric as suggested in the paper (A_i = 0.5, A_j = 0.45). The original model used τ_A = 1s, and specified that any arbitrarily faster time constants for H gave similar results (τ_H << 1). We kept τ_H = 20 ms, but changed τ_A for each subject to fit the mean rivalrous percept durations produced by the model to those the subjects perceived in the [40 50 90] disparity conditions. This ensured that the model would simulate a similar underlying rate of rivalry to each subject's observed rate. A grid search revealed that the model's mean durations depended linearly on the value of τ_A as follows: mean duration = τ_A *. 55 + 507 ms. The resulting τ_A values ranged from 4.30 to 10.19 s, with a mean of 7.02 s. The mean rivalry durations produced by the model without interruptions then had a correlation of 0.996 with the rivalry durations reported by the subjects in the [40 50 90] rivalry conditions.

To compare this model's performance with our data, we used the fusion and interfusion durations reported by our subjects as stimulus off and on durations respectively. We simulated 8 min for each subject, with a simulation time step of 1 ms. To convert the results to a button press format, time steps where H_i > H_j were considered clockwise reports, H_j > H_i were considered counterclockwise reports, and any points where both values fell below 0.4 were considered vertical (fused) reports. Bootstrapping was achieved by choosing 8 of the subject's 8 1 min trials to use as input each time.

Single-process model.

Rather than being separate bistable processes, fusion and rivalry might exist within a single process that can alternate between all three states. We simulated such a single-process model by expanding a portion of an existing model that could produce binocular rivalry (Wilson, 2003) to have three competing units. The original model contained left- and right-eye excitatory units that each activated inhibitory neurons to suppress the other, and self-adapted over time. We added a third unit to this model, connected to each of the original units with the same parameters and interactions to preserve symmetry. Each unit (left, right, and fused) thus had three variables which changed over time - an excitatory strength E_V, inhibitory strength I_V, and adaptation term H_V. All three values were initially set to zero and then updated according to the following equations (shown here for the “left” unit): E_Vleft was the excitatory activity of the unit representing a counter-clockwise-rotated percept when viewing the tristable stimulus, whose input strength V_left was set to 10 for all values of t. The two other units representing a clockwise or fused percept had equivalent variables, E_Vright and E_fused. Each unit received inhibitory input from its two neighbors with strength g = 0.45. The dynamics of the excitatory activity had a time constant τ_E = 20 ms. The asymptotic value of each unit's excitatory activity was described by a Naka–Rushton-like equation; that is, it rose sigmoidally to a maximum firing rate as a function of the difference between the input and inhibition from the other two units. This difference was half-rectified such that if the inhibition exceeded the input the difference was considered zero, i.e., for the left unit [V_left(t) − gI_Vright − gI_Vfused]₊, as seen in Equation 3. Each unit suppressed the other two with an inhibitory activity I_V that approached its excitatory activity E_V with a time constant of τ_I = 11 ms (i.e., I_Vleft approaches E_Vleft, as seen in Eq. 4). Our model also included a noise component in the inhibitory strength equation, which caused simulated percept durations to become log-normally distributed (Fox and Herrmann, 1967; Lehky, 1988). The noise values were drawn from a Gaussian distribution with mean 0 and standard deviation 400 (Eq. 5). Finally, the strength of slow self-adaptation approached h = 0.47 of the excitatory activity with a time constant τ_H. This time constant was fit so that the model produced mean durations similar to those of each subject at tristability. In a grid search, the model was found to produce mean duration times approximately 1.066 times the value of τ_H. This relationship was used to determine τ_H for each subject. The resulting τ_H values ranged from 2.94 to 5.92 s with a mean of 4.17 s. Simulated durations had a correlation of 0.94 with observed durations. Apart from these fitted τ_H values, the addition of noise and inhibition from two units rather than one, all equations and parameters were preserved from the Wilson model.

The single-process model was simulated for 8 min for each subject with a simulation step of 1 ms. To extract simulated button-press records, the unit with the highest excitatory value at a given time step was considered to be perceptually dominant. Durations shorter than 150 ms were discarded to account for subjects' limited response speeds. The eight-minute simulated response record for each observer was then processed in the same way as the psychophysical data for comparison (see Fig. 9).

Code accessibility.

The custom code used to simulate our models is available upon request from the corresponding author.

Duration distribution fitting.

To better assess the duration distributions of rivalry and fusion percepts, times from multiple trials were combined. Duration distributions for rivalry were taken from the 40°, 50°, and 90° disparity trials in both horizontal and vertical reference trial blocks. Those of fusion were taken from the 25°, 30°, and 35° trials of the vertical trials. In all cases, the final period of each trial was omitted due to its truncation by the trial's end. Log-normal distributions were fit by finding the mean and standard deviation which maximized the log-likelihood of the z-scored data. The Nelder–Mead method was used to search over parameters, and bootstrapping was used to get 95% confidence interval (CI) estimates.

Statistical analysis.

Kolmogorov–Smirnov (KS) tests were used when comparing the duration distributions of percepts. These tests determined whether observed values could be considered statistically distinct from each other (when comparing triads around fused and monocular percepts), or from simulated values sampled from a fitted distribution (when testing for log normal fits).

Autocorrelation of perceptual durations.

To measure the degree to which each perceptual duration predicted future durations, we computed the correlations between the duration of each percept and their neighbors from one to 10 periods later. Each observer's perceptual durations were shifted from between zero and 10 periods and correlated with themselves. The values were normalized so that the autocorrelation values (for shifts of zero) equaled one. The results for all observers were then averaged to produce the final plots (see Figs. 5; 9C).

Observer exclusion criteria.

We removed four observers from our analysis on the basis of unreliable fusion or rivalry reports. Specifically, two observers (S02 and S08; see Fig. 3) had an average rivalry duration for 90° disparity trials, which was >1.5 times the interquartile range (IQR) above the mean of all observers (>∼6 s). These observers sometimes reported seeing one orientation for nearly whole trials, and so were considered to be outliers in their perception of rivalry. Two observers (S05 and S06; see Fig. 3) saw stimuli with 0° or 10° of disparity as non-vertical (unfused) an average of >1.5 IQR more than the rest of the observers (>∼17/240 s total). As stimuli with 10° of disparity should appear mostly fused and 0° have no possibility of a rotated appearance, these observers who reported significant non-fusion were considered outliers in their perception of fusion.