Going with the Flow: The Neural Mechanisms Underlying Illusions of Complex-Flow Motion

Psychophysical measurement of the Pinna–Brelstaff illusion in human and macaque

Previous studies of the Pinna–Brelstaff figure (Bayerl and Neumann, 2002; Gurnsey and Pagé, 2006; Pan et al., 2016) have focused on the perception of illusory rotation elicited by real expansion and contraction of the stimulus (Fig. 1B, left). By comparison, the perception of illusory radial expansion or contraction elicited by rotation of the figure has rarely been addressed (Fig. 1B, right). Here we examined both of these illusory effects in nine human subjects and two rhesus monkeys. We created Pinna–Brelstaff figures that varied only in the relative orientation of the local Gabor patches (+45° and −45°), producing illusions of opposite directions (Fig. 1C; Gurnsey et al., 2002). To confirm that the variation of the Pinna–Brelstaff figures generates complex-flow motion perception for both rotary and radial illusions, and to map the correspondence between the relative orientation of the Gabor and the perceptual outcome, we first ran a 3AFC detection task (Fig. 1D, top; Kingdom and Prins, 2016) on human subjects. They were asked to report whether they did or did not perceive illusory motion and, if they did, in which direction. None of our subjects reported illusory motion under any kind of real motion for the 0° control stimulus (Fig. 2A, middle column). By comparison, each of the ±45° tilted Gabor stimuli generated opposite illusory flow motion perceptions (Fig. 2A, left and right columns). For example, real expansion of the Pinna–Brelstaff figures resulted in illusory CCW rotation for the +45° condition and illusory CW rotation for the −45° condition, whereas real contraction elicited CW and CCW illusory rotations for the +45° and −45° conditions, respectively. Likewise, real CW rotation of the Pinna–Brelstaff figures resulted in illusory contraction for the +45° condition and illusory expansion for the −45° condition (and again this pattern reversed for real CCW rotation of the figures). These results are summarized in Figure 2B.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Illustration of the Pinna–Brelstaff illusion, stimulus parameters, and the 3AFC/2AFC psychophysical paradigm. A, Classical Pinna–Brelstaff rotary illusion. B, Real radial (left) and rotary (right) motion of Pinna–Brelstaff figure produce illusory rotary and radial motions, respectively. C, Illustration of Pinna–Brelstaff figures used in this study, whose local Gabor orientation varied between +45°, 0°, and −45°. D, Schematic illustration of psychophysical 3AFC detection tasks used for human subjects (top) and 2AFC discrimination tasks used for both human subjects and macaque monkeys (bottom).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Illusory flow motion percepts during different physical manipulations of Pinna–Brelstaff figures. A, The results of the 3AFC detection tasks to Pinna Illusory complex-flow motions in human subjects. Each row represents a physical manipulation of Pinna–Brelstaff figures, and each column represents a stimulus condition (+45°, 0°, −45°). The x-axis delineates the three motion choices. The illusory motion patterns are illustrated by orange icons on the x-axis, and No illusion was marked as “No.” The y-axis shows the proportion (percentage) for each perceptual choice. During real radial motion condition, the three options were CCW illusory rotation, No illusion, and CW illusory rotation. For the real rotary motion condition, the three options were contraction illusion, no illusion, and expansion illusion. B, Summary of the perceived illusory complex-flow motions under different kinds of physical motion manipulations for each stimulus condition (+45°, 0°, −45°).

Based on the subjects' perceptual responses to the three stimulus conditions (+45°, 0°, −45°; Fig. 2B), we then quantified the magnitude of the illusory motion, using a 2AFC Class A direction-nulling discrimination procedure (Brindley, 1970; Kingdom and Prins, 2016; Fig. 1D, bottom; see Materials and Methods). In this task, subjects were instructed to report the perceived direction of rotary or radial motion of a given stimulus figure as a function of the rotary or radial speed physically superimposed onto the illusory stimulus motion. For example, in Figure 3A, the radial expansion stimulus had a fixed radial speed of 5°/s (linear speed), which elicited clear illusory rotation. The speeds of the real rotary motion superimposed onto the illusory rotation are plotted on the abscissa (positive, CW rotation; negative, CCW rotation). The illusory rotation induced by the real expansion stimulus was either strengthened or reduced by the addition of real nulling rotation, leading to a rightward or leftward shift of the psychometric functions. The shifts were dependent on the angle of the tilted Gabor orientation (Fig. 3A, magenta and green curves). We were interested in the speed at which the illusory CW and CCW rotations were cancelled (nulled) by the superimposed real rotation and could no longer be discriminated from each other (50% on the ordinate). We used a modified Bayesian adaptive staircase procedure (Kontsevich and Tyler, 1999; Prins, 2013) to optimally estimate this speed, and the corresponding point on the abscissa is called the PSE. This is a measure of the strength of the induced illusory rotation.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

The results of 2AFC psychophysical experiments on human subjects and macaque monkeys. A, B, Examples of psychometric functions obtained from human subject LF when tested for illusory rotary and radial motion with the real expansion and CW rotation tasks. Green, black, and magenta represent −45°, 0°, and +45° tilted Gabors. Positive and negative values on the x-axis represent the speed of different types of flow motion patterns as indicated underneath, while the circles show the response frequency as a function of physical speed, and the circle sizes represent the repeat times of that speed condition through using the staircase method (see Materials and Methods). C, Box plots show the distributions of individual Δ PSE values from all nine subjects. Real expansion condition on the left and real CW rotation condition on the right. D, E, Examples of psychometric functions from human subject LF, testing perception of illusory rotation using real contraction, and illusory radial motion using real CCW rotation. Same conventions as in A and B. F, Box plots show the distribution of individual Δ PSE values across all nine subjects. Real contraction condition on the left and real CCW rotation condition on the right. G, H, Examples of psychometric functions from a single day of testing in monkey WJ using the same physical manipulations of the same Pinna–Brelstaff figures as those in A and B. I, Box plots show the distributions of Δ PSE values over 7 d from two monkeys, WJ and DX. J, K, Examples of psychometric functions for monkey WJ obtained from a single day with the same physical manipulation of the same Pinna–Brelstaff figures as those in A and B. L, Box plots show the distribution of Δ PSE values over 7 d from two monkeys. Asterisks denote statistical significance at **p < 0.01; ***p < 0.001.

The PSE values were similarly determined for the rotary stimulus (Fig. 3B) eliciting illusory radial expansion and contraction. Here, the real rotary speed was fixed at 30°/s of angular speed (corresponding to a mean linear speed of ∼5.5°/s). Real nulling radial motion (positive, expansion; negative, contraction) was then superimposed onto the illusory radial motion.

We quantitatively define the strength of the illusory effects by plotting the PSE differences between these conditions (Δ PSE). The mean Δ PSE for the ±45° conditions calculated across all nine human observers differ significantly from the control values for both induced illusory rotary and radial motion (Fig. 3C, left: F_(2,24) = 59.46; p = 4.65 × 10⁻⁵ for −45°; p = 3.17 × 10⁻⁵ for +45°; Fig. 3C, right: F_(2,24) = 76.45; p = 1.81 × 10⁻⁵ for −45°; p = 2.31 × 10⁻⁶ for +45°; Tukey-Kramer post hoc corrected one-way ANOVA).

Figure 3D–F shows similar results for contraction and CCW rotation tasks (Fig. 3F, left: F_(2,24) = 61.01; p = 2.77 × 10⁻⁷ for −45°; p = 4.7 × 10⁻⁴ for +45°; Fig. 3F, right: F_(2,24) = 37.5; p = 0.0013 for −45°; p = 3.31 × 10⁻⁴ for +45°; Tukey-Kramer post hoc corrected one-way ANOVA). These results demonstrate that the Pinna–Brelstaff figures produce robust and, more importantly, predictable illusory effects, consistent with the results of previous psychophysical studies (Pinna and Brelstaff, 2000; Bayerl and Neumann, 2002; Gurnsey et al., 2002; Gurnsey and Pagé, 2006).

We performed similar psychophysical tasks in two monkeys (Fig. 3G–L). The animals were trained to report the motion directions of the stimulus (i.e., CW vs CCW rotation; Fig. 3G) and expansion versus contraction (Fig. 3H). Note that the monkeys reported only what they perceived as they did not know which trials contained the illusory motion conditions. The slopes of psychometric functions derived from the monkeys are shallower but exhibit similar shifts as those obtained from the human observers (see examples in Fig. 3G,H). Across sessions, the mean Δ PSE between ±45° and 0° conditions in both rotary and radial motion discrimination tasks is highly significant in each animal (Fig. 3I, left: F_(2,39) = 74.88; −45°: monkey WJ, p = 5.43 × 10⁻⁹; monkey DX, p = 8.1 × 10⁻⁵; +45°: monkey WJ, p = 2.82 × 10⁻⁶; monkey DX, p = 4.76 × 10⁻⁵; Fig. 3I, right: F_(2,39) = 84.77; −45°: monkey WJ, p = 1.03 × 10⁻⁶; monkey DX, p = 7.41 × 10⁻⁷; +45°: monkey WJ, p = 1.24 × 10⁻⁷; monkey DX, p = 9.82 × 10⁻⁵; Tukey-Kramer post hoc corrected one-way ANOVA). Figure 3J–L illustrates analogous results for the contraction and CCW rotation tasks (Fig. 3L, left: F_(2,39) = 85.58; −45°: monkey WJ, p = 0.0027; monkey DX, p = 2.57 × 10⁻⁴; +45°: monkey WJ, p = 2.65 × 10⁻⁴; monkey DX, p = 2.06 × 10⁻⁴; Fig. 3L, right: F_(2,39) = 32.04; −45°: monkey WJ, p = 2.87 × 10⁻⁷; monkey DX, p = 0.0049; +45°: monkey WJ, p = 1.98 × 10⁻⁵; monkey DX, p = 1.49 × 10⁻⁴; Tukey–Kramer post hoc corrected one-way ANOVA). In summary, these results show that, similar to human observers, nonhuman primates do perceive illusory motions in the Pinna–Brelstaff figures.

Neural responses of area MSTd to Pinna–Brelstaff figures

We next asked how neurons in areas MSTd and MT respond to Pinna–Brelstaff figures. We recorded from 312 well isolated single units in the same macaques that had performed the psychophysical measurements (MSTd, N = 192; MT, N = 120). MSTd and MT were identified both by anatomical reconstruction of the MRI scans (Fig. 4A) and the retinotopic organization of their neurons (Fig. 4B,C). In general, our MSTd neurons exhibited large RFs (mean ± SEM diameter, 30.85 ± 0.5°), whose visuotopic extent ranged from a hemifield to the whole visual display (Fig. 4B). The sizes of the RFs were uncorrelated with retinal eccentricity (r = −0.013, p = 0.87, Spearman rank correlation). In contrast, MT units usually had relatively smaller RFs that were confined to the contralateral visual field (Fig. 4C). Their RF sizes significantly increased as a function of eccentricity (r = 0.85, p = 1.75 × 10⁻³⁴, Spearman rank correlation). These results are consistent with previous electrophysiological studies of these visual areas (Gattass and Gross, 1981; Van Essen et al., 1981; Desimone and Ungerleider, 1986; Tanaka et al., 1986).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

The MRI scanning results and receptive field properties of MT and MSTd. A, Top row, MRI scanning results of one macaque monkey. The locations of chamber and electrode are shown in coronal and horizontal planes, and the penetrating guided grid is shown in the aerial view. Bottom row, Illustration of recording sites in the same hemisphere of the same monkey. Blue and orange dots, Electrode tip positions in MT and MSTd areas, respectively. B, Top, screen-scaled receptive field sizes and locations of MSTd neurons (gray rounded rectangles, RFs of neurons in the right hemisphere; black rounded rectangles, RFs of neurons in the left hemisphere). MST neurons have big receptive fields, which spread both ipsilaterally and contralaterally. Bottom, The relationship between retinal eccentricities and RF diameters. Diameters were plotted as a function of eccentricity. The RFs of MSTd neurons are nonretinotopically organized. C, Same conventions as in B, MT neurons have relatively smaller receptive fields, which are contralaterally distributed, and their RFs are retinotopically organized.

The first row on the top of Figure 5A illustrates the responses of an MSTd neuron to the 0° Pinna–Brelstaff control figure under two different complex-flow motion patterns, demonstrating that it favored real CCW rotation (Fig. 5A, third row, responses to real expansion/contraction). We also tested this neuron across three conditions (+45°, 0°, and −45°) using expanding and contracting Pinna–Brelstaff figures. The neural responses are illustrated by the PSTHs plots (Fig. 5A). Although neither of these radial motions were the preferred motion pattern for this neuron, the +45° and −45°conditions exhibited significantly enhanced responses (left magenta in the second row, p = 0; right green in the last row, p = 9.75 × 10⁻²¹; Wilcoxon signed-rank test), when compared with the 0° condition. Such facilitation of the neural response is consistent with the perceptual results that illusory CCW rotation is produced by real expansion of the +45°and contraction of the −45° Pinna–Brelstaff figure (Fig. 2B).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5.

Results of single-unit recordings from area MSTd in response to physical manipulation of the Pinna–Brelstaff figures. A, Top row, The PSTHs and raster plots show the responses of a real CCW rotation-sensitive MSTd neuron to real rotary motion patterns tested with the 0° control condition. Horizontal black dashed lines, Spontaneous responses. The lower three rows: The PSTHs and raster plots show the same MSTd unit responding to real expansion (left) and contraction (right) of the Pinna–Brelstaff figures. Orange icons show the illusory motion predicted from the psychophysics (Fig. 2B), which corresponds to the facilitative responses of the neuron. B, A real contraction-sensitive MSTd neuron responding to the 0° control condition with radial motion patterns as well as real CW (left) and CCW (right) rotations of the Pinna–Brelstaff figures. Same conventions as in A. C, D, Log axis scatter plots of the distributions of facilitative (cases, N = 264) and suppressive (cases, N = 264) responses of MSTd neurons across all Pinna–Brelstaff stimuli. E, Log axis scatter plot showing the relationship between the strength of facilitative response (Δ facilitative response) and the strength of suppressive response (Δ suppressive response, positive values show more suppression). White circles represent cases without both significant facilitation and suppression; gray circles represent cases with either significant facilitation or suppression; black circles represent cases with both significant facilitation and suppression.

Figure 5B shows another MSTd neuron that prefers real contraction, as measured with the 0° Pinna–Brelstaff control figure. In the null direction (expansion), this neuron exhibits significant suppression (p = 0, Wilcoxon signed-rank test). Testing the ±45° conditions using real CW and CCW rotation elicited significantly enhanced responses (Fig. 5B, left, magenta: p = 0; right, green: p = 4.77 × 10⁻³⁰; Wilcoxon signed-rank test), when compared with the 0° condition. In addition, real CCW rotation of the +45° figure (Fig. 5B, right, magenta) and CW rotation of the −45° figure (Fig. 5B, left, green) produced significant suppression (−45°, p = 0; +45°, p = 1.01 × 10⁻³³; Wilcoxon signed-rank test). As expected, the facilitated and suppressed responses of this neuron are also consistent with the perceptual results of illusory contraction and expansion generated by the Pinna–Brelstaff figures (Fig. 2B).

We summarized the population results across MSTd units preferring rotary and radial motion (N = 132; Fig. 5C,D). Firing rates in response to the physical manipulation of the Pinna–Brelstaff figures were plotted for the ±45° condition (on the ordinate) against the control 0° condition (on the abscissa). Based on the observation in Figure 2, neurons whose preferred real motion patterns matched the illusory motion pattern were plotted in Figure 5C, while those matching the anti-preferred motion patterns of the neurons were plotted in Figure 5D. Each unit had two response cases for both preferred and antipreferred directions. For example, real expansion of the +45° condition and contraction of the −45° condition each contributes one case of predicted facilitation for the example unit in Figure 5A, and vice versa for suppression. Therefore, N represents cases, not the number of neurons. In Figure 5C, we found that the overall average response was significantly enhanced by 63.6% for both the ±45° conditions (cases: N = 264, Z = 13.27, p = 3.47 × 10⁻⁴⁰, Wilcoxon signed-rank test). Specifically, nearly half of the cases showed significant facilitation (43.94%, 116 of 264 cases; permutation test, p < 0.01; Fig. 5C, filled symbols). In Figure 5D, the overall average response was significantly reduced by 25.4% (cases: N = 264, Z = −9.76, p = 1.67 × 10⁻²², Wilcoxon signed-rank test). Of those, 28.41% cases showed significant suppression (75 of 264 cases; permutation test, p < 0.01; Fig. 5D, filled symbols). We also compared the strength of facilitation and suppression for each neuron (Fig. 5E). The strength for both was calculated as the absolute difference between illusory conditions and the control condition (presented as Δ facilitative response and Δ suppressive response). The cases were divided into the following three groups depending on their significances of facilitation and suppression effects: cases without significant facilitation and suppression (cases: N = 114, 43.2%); cases with either significant facilitation or suppression (cases: N = 104, 39.4%); and cases with both significant facilitation and suppression (cases: N = 46, 17.4%). As expected, the number of significant cases increases as the Δ response strength for facilitation or suppression rises. We found that neurons with stronger facilitation to its preferred illusory motion pattern tend to have stronger suppression to its antipreferred illusory motion pattern, exhibiting a weak but significant positive linear correlation (r = 0.17, p = 0.0046, Spearman rank correlation). In addition, the strength for facilitation is significantly larger than for suppression (cases: N = 264, Z = 7.96, p = 1.67 × 10⁻¹⁵, Wilcoxon signed-rank test).

In summary, the facilitation and suppression in MSTd was consistent with psychophysical predictions (Fig. 2B), demonstrating comparable neural selectivity between an illusory complex-flow motion pattern (e.g., illusory CCW rotation) and its physical counterpart (e.g., real CCW rotation). We therefore conclude that a subgroup of MSTd neurons represent illusory complex-flow motions (rotary or radial) elicited by viewing the Pinna–Brelstaff stimulus, suggesting that a fraction of MSTd neurons may contribute directly to the perception of illusory motion.

Tuning properties of MSTd neurons to Pinna–Brelstaff figures

The above neural responses were examined for only one motion axis (i.e., radial or rotary motion). To assess the neural responses mediating illusory flow motions for a more complete stimulus range, including spiral motions (Graziano et al., 1994), we next plotted the data using a two-dimensional polar coordinate system as defined in previous studies (Graziano et al., 1994; Heuer and Britten, 2004; Xu et al., 2014). In these plots, the real radial (expansion and contraction) and rotary (CW and CCW) motions are represented on the horizontal and vertical meridians, respectively. In addition, real spiral motions, which are derived from the weighted combination of radial and rotary motions, are distributed with equal intervals between the two main axes (Fig. 6A). The gray dots in Figure 6A illustrate the distribution of the recorded MSTd neurons in their preferred real complex-flow motion patterns.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6.

MSTd polar-plot tuning for complex-flow motion patterns. A, Canonical complex-flow motion patterns represented in polar coordinates, and the MSTd neuron population distribution of the preferred complex-flow motion pattern plotted against the normalized vector-summed firing. B, An MSTd neuron sensitive to CCW rotation. Left, Qualitatively predicted response tuning curves to physical manipulation of the Pinna–Brelstaff figures (green, −45°; gray, 0°; magenta, +45°). Middle, Neural responses to the Pinna–Brelstaff figures (error bars show the mean firing rate ± SEM), arrows represent the vector sums calculated from the tuning curve, and they agree with the predictions. Right, Response to moving random dot field (error bars show the mean firing rate ± SEM). C, A real expansion-sensitive MSTd neuron in response to the physical manipulation of the Pinna–Brelstaff figures. Same conventions as in B. D, An MSTd neuron sensitive to real spiral 2 motion in response to the physical manipulation of the Pinna–Brelstaff figures. Same conventions as in B.

For a hypothetical neuron sensitive to illusory complex-flow motion, the tuning curves for the ±45° conditions can be predicted to change when compared with the 0° control. For example, Figure 6B shows the polar tuning curves for an MSTd neuron that prefers real CCW rotation, as defined by the 0° control condition (Fig. 6B, middle, gray curve). For a CCW rotation-selective neuron, real expansion of the +45° condition should increase the neuronal response since this stimulus elicits illusory CCW rotation (Fig. 2B). This is reflected in Figure 6B (left, top right quadrant), where real expansion or expansion containing stimuli like spiral 1 motion (Fig. 6A) results in a rightward shift (firing rate increase) of the magenta curve (+45° condition) compared with the gray curve (0° control). In contrast, during real contraction of the +45° condition, the neural response should decrease because this stimulus elicits illusory CW rotation (Fig. 2B) that is the antipreferred motion for this neuron. This is reflected in the top left quadrant of the left panel in Figure 6B, where real contraction or contraction-containing stimuli like spiral 2 motion results in a rightward shift (firing rate decrease) of the magenta curve compared with the gray curve. Hence, the overall magenta tuning curve is expected to shift rightward (Fig. 6B, left, magenta vs gray curve). Compared with the +45° condition, the −45° condition elicits opposite illusory flow-motion perception under the same physical motion manipulation (Fig. 2B), so the facilitation and suppression effects are reversed and the tuning curve is expected to shift leftward (Fig. 6B, left, green vs gray curve). We calculated the polar direction for each condition tuning curve using vector summation, then measured the angular difference between the ±45° and 0° control (Δ preferred angle). Consistent with these predictions, the vector sum of the neuron (Fig. 6B, middle) is significantly shifted rightward by 38.4° for the +45° condition (Fig. 6B, middle, magenta arrow; p = 0, permutation test) and 75° leftward for the −45° condition (Fig. 6B, middle, green arrow; p = 0, permutation test). The right panel in Figure 6B shows the classical motion responses of the neuron to moving dot stimuli. Figure 6C presents tuning curves obtained from a real radial motion (expansion)-sensitive MSTd neuron. Using the same logic as that for the rotary motion preference neuron in Figure 6B, the tuning curves are predicted to shift upward and downward for the +45° and −45° conditions, when compared with the 0° control condition (Fig. 6C, left, green and magenta curves vs gray curve). Consistent with this prediction, the vector sum of this neuron is significantly shifted by 22.6° upward and 16.8° downward, respectively (+45°, p = 0; −45°, p = 0; permutation test).

The shift of the vector sum in the tuning curves is an efficient way to quantify the illusory effects of the Pinna–Brelstaff figures for neurons preferring radial and rotary motions. However, this is not applicable for neurons preferring spiral motion, whose responses can be influenced by both radial and rotary motions (Fig. 6A,D). As a result, their amplitudes instead of vector sums are expected to change (Fig. 6D, left; for a more detailed description, see Materials and Methods). An exemplar neuron preferring spiral motion is shown in Figure 6D (middle), and significant changes in the tuning amplitude (Δ amplitude) were indeed observed for the Pinna–Brelstaff figures (+45°: 12.31, p = 0; −45°: 16.17, p = 0.0002; permutation test).

We next examined the tuning curve changes across the population (rotary, N = 50; radial, N = 82; spiral, N = 60; Fig. 7). The sign of the Δ preferred angle and Δ amplitude was assigned such that positive values represent the Δ changes as predicted from the Pinna–Brelstaff illusory effects (Fig. 2B). Other Δ changes not in line with the prediction were given as negative values. For rotation preferring neurons (Fig. 7A), the overall distribution of Δ preferred angle is significantly shifted to the right (14.64 ± 1.66°, mean ± SEM; Z = 7.17, p = 7.75 × 10⁻¹³; cases, N = 100, Wilcoxon signed-rank test; N represents cases). Within the population, 35% (35 of 100 cases; permutation test, p < 0.01) individual cases exhibited significant shifts. Similarly, for radial motion-preferring neurons (Fig. 7B), the overall distribution is also significantly shifted rightward (19.32 ± 1.49°, mean ± SEM; Z = 10.09, p = 6.36 × 10⁻²⁴; cases, N = 164, Wilcoxon signed-rank test). Again, 61.59% of cases (101 of 164 cases; permutation test, p < 0.01) show significant shifts. Finally, for the subpopulation preferring real spiral motion (Fig. 7C), the mean Δ amplitude is 4.27 ± 0.94 spikes/s, which is significantly larger than zero (Z = 4.88, p = 1.08 × 10⁻⁶; cases, N = 120, Wilcoxon signed-rank test). About 31.67% of individual cases (38 of 120 individual cases; permutation test, p < 0.01) showed a significant change, with the majority being in the predicted direction. In summary, these results demonstrate that across all complex-flow motion axes, distinct subgroups of MSTd units exhibit significant changes in their responses consistent with the Pinna–Brelstaff illusory effects.

• Download figure
• Open in new tab
• Download powerpoint

Figure 7.

Population summary of changes in the polar tuning functions during presentation of Pinna–Brelstaff figures. A, Distribution of Δ preferred angle for rotary motion sensitive MSTd units, arrow shows the population mean (black bars, cases that were significantly different with a permutation test at p < 0.01; white bars, remaining cases). B, Distribution of Δ preferred angle for radial motion-sensitive MSTd units; conventions as in A. C, Distribution of Δ amplitude for spiral motion-sensitive MSTd units; conventions as in A.

Neural responses of area MT to Pinna–Brelstaff figures

The neural origin of a visual illusion is often elusive (Spillmann and Werner, 1990). Area MT is an essential motion center in the primate dorsal visual pathway for processing global motion signals (Movshon et al., 1983; Rust et al., 2006). However, it has previously been shown to be insensitive to complex-flow motions (Tanaka et al., 1986; Smith et al., 2006), but because area MSTd receives major inputs from MT (Maunsell and Van Essen, 1983; Ungerleider and Desimone, 1986), it is still important to ascertain how MT itself responds during the perception of illusory complex-flow motions.

To address this, we examined how MT neurons respond to the global stimuli during presentation of the same Pinna–Brelstaff figures. We recorded from 120 MT units, about half of these MT units exhibited significant responses to stimuli during the presentation of the 0° condition (N = 64 of 120). Among these responsive MT units, about a third of the cases (35.94%, 46 of 128 cases; permutation test, p < 0.01; similar to MSTd, each MT neuron also has two cases, so the case number is twice the number of MT neurons) showed significantly increased responses to the ±45° conditions when compared with the 0° control condition (Fig. 8A). The population average response was enhanced by 25.59% (Z = 5.21, p = 1.93 × 10⁻⁷; cases, N = 128, Wilcoxon signed-rank test). About one-third of the cases (32.81%, 37 of 128 cases; permutation test, p < 0.01) showed significantly reduced responses (Fig. 8B), with an overall reduction of 23.61% (Z = −5.71, p = 1.11 × 10⁻⁸; cases, N = 128, Wilcoxon signed-rank test). Since the origin for both radial and rotary motion are kept at the center of the screen without considering the RF location of MT neurons and their relatively small RF sizes, it is expected that MT neurons respond to local real or illusory motion vectors of the embedded Gabor patches within the Pinna–Brelstaff figures. Despite this difference, MT response properties can still be measured using the same polar plot methodology as used for MSTd. However, for MT neurons the axes represent the local direction of motion of the Gabor elements rather than expansion, contraction, and CW and CCW rotations. We combined the tuned responses distributed on the cardinal axes (Fig. 8C, Δ preferred angle equivalent to radial and rotary motion in MSTd cells), separating them from tuned responses distributed between the cardinal axes (Fig. 8D, Δ amplitude equivalent to spiral motion). The tuning properties indicate that MT neurons can also represent illusory motions, even if they are local.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8.

Results of single-unit recordings of MT in response to the physical manipulation of Pinna–Brelstaff figures. A, B, Log axis scatter plots of distributions of facilitative and suppressive responses across all Pinna–Brelstaff stimuli; cases were determined following the method used for MSTd neurons. C, Population results defining the percentage of a subgroup of MT units with preferred angles distributed around cardinal polar axis (black bars, cases that were significantly different with a permutation test at p < 0.01; white bars, remaining cases). The arrow shows a population mean of Δ preferred angle for both ±45° stimulus cases of 11.14 ± 2.24° (mean ± SEM; Z = 5.98, p = 2.18 × 10⁻⁹; cases, N = 128, Wilcoxon signed-rank test) with 41.41% of cases (53 of 128 cases); permutation test, p < 0.01) showing significant response changes. Cases were determined following the method used for MSTd neurons. D, Population results of Δ amplitude for another group of MT units with preferred angles distributed between cardinal polar axis; conventions are as in C. Mean amplitude change: 4.03 ± 1.31 spikes/s (mean ± SEM; Z = 2.96; p = 0.003; cases, N = 112, Wilcoxon signed-rank test) with 28.57% of cases (32 of 112 cases; permutation test, p < 0.01) showing significant response changes.

To further examine the presupposition that MT neurons respond to the local illusory motions, we performed a masking experiment for subgroups of MT neurons exhibiting significant responses to the global manipulation of the Pinna–Brelstaff figures. In the masked condition, the Gabor stimuli were presented only within the receptive field of the recorded MT neuron (Fig. 9A). This masking operation effectively removed both the global illusory and real complex-flow motions in the Pinna–Brelstaff figures. Nevertheless, the tuning curves of the MT neuron in response to the +45°, 0°, and −45° conditions were almost identical for the unmasked stimuli (Fig. 9B, left) and the masked stimuli (Fig. 9B, right). For the population, we also found that there was no significant difference between masked and unmasked stimuli (Δ preferred angle: Z = 0.8, p = 0.4209, N = 19; Δ amplitude: Z = 0.32, p = 0.5417, N = 13; Wilcoxon signed-rank test; Fig. 9C). To explain the contribution of local motion components in Pinna–Brelstaff figures to the tuning in MT, we can take the example neuron in Figure 9B and plot the tuning to both random dot complex-flow (Fig. 9D, up) and translational (Fig. 9D, down) motion. The preferred tuning direction for translational motion is consistent with the local direction of the complex-flow motion in the receptive field. We can see that the global motion stimuli with CW rotations will generate a local motion component of left–downward motion inside the receptive field of this unit (Fig. 9E, black arrow). But because of the aperture effect, the direction encoded by the MT neuron is biased to the direction perpendicular to the orientation of the gratings (Fig. 9E, white arrow) compared with the unbiased motion direction. The biased direction (left) is closer to the preference of the MT neuron (left–upward motion; Fig. 9D, down), and, as a result, the neuron shows a stronger response. This causes a higher firing rate observed for the +45° condition that is not seen under the 0° control condition (no aperture effect). Consistent with this, the whole tuning curve under the +45° condition is shifted downward, as indicated by the difference in the resultant vectors in the polar plot in Figure 9B. These results demonstrate that MT neurons predominantly encode the local illusory motion signals (presumed to be driven by the aperture effect) embedded in the micropatterns of the Pinna–Brelstaff figures.

• Download figure
• Open in new tab
• Download powerpoint

Figure 9.

The results of masking experiments on MT neurons. A, Unmasked full-field and locally masked Pinna–Brelstaff stimuli used. The red dashed circle in the right panel marks the location and size of the receptive field of an MT neuron. B, Polar plot of the neural responses to unmasked and masked Pinna–Brelstaff stimuli (error bars show the mean ± SEM), showing almost identical response tuning curves for the two conditions. The icons of black arrows around the polar axis present the local linear motion vectors inside the receptive field. C, Left, Scatter plot of Δ preferred angles across MT neurons specifically tested for unmasked and masked conditions. Box plot alongside the scatter plot showing the statistical distribution of Δ preferred angles between unmasked and masked conditions; there were no statistical differences for both Δ parameters between the two conditions. Right, Scatter plot and box plot of Δ amplitudes, using the same conventions as those for Δ preferred angles. D, Up, The response tuning curve of the exemplar neuron to random dot complex-flow motion, giving the strongest response to left–upward local linear motion; under this direction, the global complex-flow motion pattern is contraction. Down, The response tuning curve of the same neuron to translational random dot motion, exhibiting a left–upward (∼150°) preference. E, Illustrated derivation of biased motion direction.

Comparison of MT and MSTd neuronal responses to Pinna–Brelstaff figures

The above results demonstrate that MSTd neurons are able to globally encode illusory complex-flow motion, while MT neurons represent local biased motion. There are known differences in motion preferences and spatial scales of the receptive fields between MT and MSTd, so we next examined whether the sensitivity and selectivity for global illusory motion between these two cortical areas are different or not. For these analyses, we combined the different complex-flow motions together so that we could directly compare real against illusory motion.

First, we directly compare the Δ firing rate differences between MT and MSTd neuronal responses encoding illusory motion. The Δ firing rate was calculated by subtracting the firing rate of 0° control condition from the firing rate of ±45° conditions (only facilitative response cases were compared). In Figure 10A, we found that the Δ firing rates of the MSTd neurons (10.68 ± 0.79 spikes/s, mean ± SEM; cases, N = 264) are significantly larger than those of the MT neurons in response to the same unmasked stimuli (5.23 ± 0.92 spikes/s, mean ± SEM; cases, N = 128; Z = 4.53, p = 5.79 × 10⁻⁶, Wilcoxon rank sum test). As a control, we also compared the Δ firing rate (calculated by subtracting the firing rate of blank stimulus condition from the firing rate of 0° control condition) for real motion responses between the two areas and found that they are comparable (Fig. 10B; MSTd: 43.80 ± 2.59 spikes/s, mean ± SEM; N = 132; MT: 44.69 ± 3.61 spikes/s, mean ± SEM; N = 64; Z = −0.31, p = 0.7565, Wilcoxon rank sum test).

• Download figure
• Open in new tab
• Download powerpoint

Figure 10.

Comparison of both illusory and real motion sensitivity between MSTd and MT. A, Population distributions of Δ firing rate for illusory motion. B, Population distributions of Δ firing rate for real motion. C, The population distributions of AUC for both illusory (orange) and real (black) motion. Asterisks denote statistical significance: ***p < 0.001.

We used ROC analysis (Green and Swets, 1966; Britten et al., 1992, 1996; Celebrini and Newsome, 1994; Price and Born, 2010) to estimate how well an ideal observer (here the neuron) could discriminate real and illusory motions based on the neural activity in both brain areas. Specifically, for MSTd neurons the discriminability refers to the global real or illusory complex-flow motions, and for MT neurons it refers to local linear motion directions that are present within the MT receptive field as a part of the global motion pattern. The AUC quantifies the sensitivity of a neuron to discriminate illusory motion against the 0° control. Individual AUC values were calculated for each illusory motion response case in MSTd and MT, and population distributions are illustrated by the orange boxplots in Figure 10C. The mean AUC value for MSTd neurons (0.79 ± 0.01, mean ± SEM; cases, N = 264) is significantly larger than that for MT neurons (0.65 ± 0.03, mean ± SEM; cases, N = 128; Z = 3.52, p = 4.31 × 10⁻⁴, Wilcoxon rank sum test), demonstrating that MSTd has higher sensitivity in discriminating Pinna–Brelstaff illusory motion. A similar comparison was also performed for real motion, and the results are not significantly different between the two areas (MSTd: 0.89 ± 0.02, mean ± SEM; N = 132; MT: 0.93 ± 0.01, mean ± SEM; N = 64; Z = −1.13, p = 0.2571, Wilcoxon rank sum test). Hence, these results indicate that although both MT and MSTd respond to the Pinna–Brelstaff figures, MSTd exhibits relatively higher sensitivity compared with MT.

Integration time window for global illusory complex–flow motions

As demonstrated previously and in the current study, MT neurons have relatively small RFs compared with MSTd neurons and thus are well suited for processing local motion information within their RFs. The outcome of such local motion processing in MT is subsequently pooled by downstream areas such as MSTd to generate complex-flow motion (Layton and Fajen, 2016; Yu et al., 2018). Processing of visual information along a hierarchy of visual areas results in a distribution of temporal responses (Schmolesky et al., 1998). We therefore performed temporal ROC and peak firing rate analysis for both MSTd and MT units to address whether the integration of motion signals that generate illusory motion is similar to or different from real motion.

We first applied temporal sliding-window ROC analysis and calculated the AUC value across the stimulus duration for both real and illusory motion. We measured the population latency as the first significant AUC response above baseline (see Materials and Methods). In MT, the population latencies for real (Fig. 11A) and illusory (Fig. 11B) motion were 54 and 51 ms, respectively. In addition, we also computed the latency difference at the individual neuron level, and found no significant difference (Fig. 11E, right, orange and black boxes; real motion: 66.97 ± 2.44 ms, mean ± SEM; cases, N = 74; illusory motion: 78.73 ± 4.39 ms, mean ± SEM; cases, N = 44; Z = 1.76; p = 0.0783, Wilcoxon rank sum test). In MSTd, the population latency for real motion is 70 ms (Fig. 11C), shorter than that for illusory motion (85 ms; Fig. 11D). Again at the individual neuron level, the difference is significantly different between these two global motions (Fig. 11E, left, orange and black boxes; real motion: 84.25 ± 1.76 ms, mean ± SEM; cases, N = 110; illusory motion: 107.46 ± 3.98 ms, mean ± SEM; cases, N = 124; Z = 4.75; p = 2.06 × 10⁻⁶, Wilcoxon rank sum test). We further compared the latency difference at the individual neuron level between MT and MST for either real or illusory motion. We found that MSTd neurons take a significantly longer time than MT to reliably discriminate both real and illusory motion (Fig. 11E; real motion: Z = 6.31, p = 2.86 × 10⁻¹⁰; illusory motion: Z = 3.92, p = 8.96 × 10⁻⁵). The temporal ROC result demonstrates that both real and illusory complex-flow motions require a time window to integrate MT local motion signals, allowing MSTd to reliably discriminate global flow-motion patterns. The integration time that enables MSTd to reliably signal global flow motion is ∼16 ms for real motion and ∼34 ms for illusory motion, respectively, when compared with MT. Since the ROC latency in MT for both motion classes are almost the same, an extra 15 ms of integration time is required for illusory motion to be reliably represented in MSTd. The fact that there is no temporal difference between illusory and real motion in MT suggests that the integration mechanisms from its V1 inputs may be similar. We also compared the ROC latencies of MT neurons responding to masked and unmasked stimuli, and we found comparable latency distributions between those two conditions (masked condition: 89.07 ± 7.73 ms, mean ± SEM; cases, N = 15; unmasked condition: 78 ± 9.54 ms, mean ± SEM; cases, N = 11; Z = −1.71; p = 0.0866, Wilcoxon rank sum test). These results further suggest that illusory motion signals are propagated from MT to MSTd.

• Download figure
• Open in new tab
• Download powerpoint

Figure 11.

Temporal ROC analysis of AUC for both MSTd and MT responses. A, Temporal ROC analysis of MT neurons to real motion. Averaged black curve (± SEM) shows the change of AUC value throughout the first 200 ms after stimulus onset. The first time point where the AUC value is significantly higher than the 0.5 threshold (p < 0.01, after Bonferroni correction) is marked by a dashed line and a black arrow on the x-axis. B, Temporal ROC analysis of MSTd neurons to real motion. Same conventions as in A. C, Temporal ROC analysis of MT neurons to illusory motion. Same conventions as in A. D, Temporal ROC analysis of MSTd neurons and illusory motion. Same conventions as in A. E, Population distributions of latencies from MT and MSTd representing both illusory (orange) and real (black) motions. F, Population distributions of latencies from MSTd representing both illusory (orange) and real (black) rotary/radial motion. Asterisks denote statistical significance: **p < 0.01, ***p < 0.001.

Considering the differences between rotary and radial motion across visual space, we further compared MSTd latencies to both motion patterns. Interestingly, we found that MSTd neurons respond much faster to real rotary motion than to real radial motion (Fig. 11F, black boxes; real rotary motion: 76.76 ± 3.65 ms, mean ± SEM; cases, N = 38; real radial motion: 86.12 ± 1.95 ms, mean ± SEM; cases, N = 72; Z = −2.91; p = 0.0037, Wilcoxon rank sum test). However, such a difference was not found for illusory flow motions (Fig. 11F, orange boxes; illusory rotary motion: 100.83 ± 7.57 ms, mean ± SEM; cases, N = 44; illusory radial motion: 109.98 ± 4.69 ms, mean ± SEM; cases, N = 80; Z = −0.72; p = 0.4716, Wilcoxon rank sum test).

Figure 12 presents the population-averaged response curves of a temporal analysis for MSTd and MT neurons using onset and peak firing rate. The response onset latency for MSTd illusory motion (∼71 ms) was substantially longer than that for MT (52 ms), and so is the peak latency (MSTd, 92 ms; MT, 66 ms; Fig. 12A,B, left column); these differences were consistent with both onset (MSTd, 64 ms; MT, 53 ms) and peak (MSTd, 94 ms; MT, 67 ms) latencies of real motion (Fig. 12A,B, right column). Similar to the ROC temporal analysis, we found that in MSTd, illusory motion needs more time for its representation than real motion (Fig. 12A; illusory motion, 71 ms; real motion, 64 ms), while such differences did not exist in MT (Fig. 12B, illusory motion, 52 ms; real motion, 53 ms). We also measured the onset and peak latencies at the individual neuron level, and the averaged onset latency for illusory motion of MSTd neurons was significantly longer than that of MT (Fig. 12C, left, orange boxes; MT, 58.64 ± 4.18 ms, mean ± SEM; cases, N = 57; MSTd: 69.27 ± 2.27 ms, mean ± SEM; cases, N = 144; Z = 4.19; p = 2.85 × 10⁻⁵, Wilcoxon rank sum test), and so is the peak latency (Fig. 12D, orange boxes; MT: 75.47 ± 4.24 ms, mean ± SEM; cases, N = 57; MSTd: 91.65 ± 3.43 ms, mean ± SEM; cases, N = 144; Z = 3.78; p = 1.56 × 10⁻⁴, Wilcoxon rank sum test). These latency differences suggest that MSTd neurons require a time window to globally integrate feedforward inputs of MT local illusory motion. For real motion, as expected, an integration time window is also needed (Fig. 12A,B, right column) when comparing MT and MSTd responses (Fig. 12C,D, black boxes; see statistical summary in the figure legend). We also compared the responses of MSTd neurons between illusory and real motion conditions, and responses of MT neurons between the same motion conditions. We found no statistical difference for the peak latency (Fig. 12D, orange vs black boxes; MSTd: Z = 1.13; p = 0.2573; MT: Z = −0.1; p = 0.9207, Wilcoxon rank sum test). However, for the response onset latency, similar to the temporal ROC statistical analysis, we found that MSTd neurons respond significantly later for illusory than for real motion, while such a difference did not exist in MT (Fig. 12C, orange vs black boxes; MSTd: Z = 3.42; p = 6.32 × 10⁻⁴; MT: Z = 0.95; p = 0.3398, Wilcoxon rank sum test).

• Download figure
• Open in new tab
• Download powerpoint

Figure 12.

Temporal peak firing rate analysis of MSTd and MT neural responses to both real and illusory complex-flow motions. A, Average PSTH responses (boxcar smoothed, ±SEM) of MSTd subgroups representing illusory and real radial motion. Black arrows on the x-axis mark the time at which the response begins and peaks. Left, Result from ±45° Pinna–Brelstaff figures (illusory motion). Right, Result from 0° control Pinna–Brelstaff figure (real motion). B, Average temporal responses of MT neurons to the same stimuli used for MSTd. Same convention as in A. C, Population data for the response onset latencies of individual MT and MSTd neurons to illusory (orange boxes) or real (black boxes) motion (real MSTd: 60.79 ± 3.35 ms, mean ± SEM; cases, N = 117; real MT: 54.07 ± 2.95 ms, mean ± SEM; cases, N = 85; Z = 2.85; p = 0.0044, Wilcoxon rank sum test). D, Population data for the peak latency (real MSTd: 85.32 ± 4.03 ms, mean ± SEM; cases, N = 117; real MT: 72.32 ± 3.06 ms, mean ± SEM; cases, N = 85; Z = 2.7; p = 0.0069, Wilcoxon rank sum test); same conventions as in C. Asterisks denote statistical significance: **p < 0.01, ***p < 0.001.

From the above results, we conclude that a bottom-up integrative mechanism underlies both real and illusory complex-flow motions encoded by MT and MSTd, but the processing of illusory local motion signals requires additional time to reliably represent global illusory motion (summarized in Fig. 13 as an example for illusory rotation). It would be reasonable to expect that MST takes longer to extract a global illusory motion induced by other classes of real flow motion (e.g., illusory rotation, which is induced by real expansion), when compared directly with the same class of real flow motion (e.g., real rotation). Note that we cannot exclude the additional possibility that feedback from higher cortical areas may also contribute to the global integration of complex-flow motion in MSTd (Fig. 13, feedforward accumulation, intracortical computation, and recurrent feedback are illustrated as potential neural mechanisms). These possibilities have been suggested in our recent fMRI study on dynamic network communication accounting for the individual variation of human perception of the same illusory complex-flow motion (Wang et al., 2018). We also cannot exclude the possibility that synchronized population firing of spatially aligned MT neurons could represent both real and illusory complex-flow motions with no latency delays (Fig. 13, middle, MT). However, this requires multiple single-unit recordings simultaneously obtained from these spatially aligned MT neurons and is beyond the scope of the current study to address.

• Download figure
• Open in new tab
• Download powerpoint

Figure 13.

Schematic summary of motion integration from local to global illusory complex-flow motion. Diagram illustrates how an example MSTd CCW rotation-sensitive neuron with a large RF (top) integrates MT local motion inputs (middle) to represent global illusory CCW rotary motion from a physically expanding +45° Pinna–Brelstaff stimulus (bottom). The AUC temporal dynamics of MSTd and MT populations plotted within the MSTd RF (top circle) show the integration time window for illusory motion. The black circles represent MT and MSTd neuron RFs. Small inset on the right of the middle panel illustrates the orthogonal motion signal of MT neurons due to the aperture effect (middle, short orange arrows); such biased illusory motion signals from a group of circularly arranged MT neurons are assumed to be integrated (shown by orange arrows between middle and top panels) by the MSTd neuron, resulting in global illusory CCW rotation (top, long orange arrows and circle).