Abstract
Sensitivity to visual motion is a fundamental property of neurons in the visual cortex and has received wide attention in terms of mathematical models. A key feature of many popular models for cortical motion sensors is the use of pairs of functions that are related by a 90° phase shift. This phase relationship, known as quadrature, is the hallmark of the motion energy model and played an important role in the development of a class of model dubbed elaborated Reichardt detectors. For decades, the literature has supported a link between quadrature and the observation that motion detectors and human observers often prefer a 1/4 cycle displacement of an apparent motion stimulus that consists of a pair of sinusoidal gratings. We show that there is essentially no link between quadrature and this preference. Quadrature is neither necessary nor sufficient for a motion sensor to prefer 1/4 cycle displacement, and motion energy is not maximized for a 1/4 cycle step. Other properties of motion sensors are the key: the opponent subtraction of two oppositely tuned stages that individually have sinusoidal displacement tuning curves. Thus, psychophysical and neurophysiological data revealing a preference at or near 1/4 cycle displacement do not offer specific support for common quadrature or energy-based motion models. Instead, they point to a broader class of model.
Introduction
An influential set of models (Adelson and Bergen, 1985; van Santen and Sperling, 1985; Watson and Ahumada, 1985) for direction-selective (DS) neurons in cat and monkey primary visual cortex (Hubel and Wiesel, 1962, 1968) grew rapidly from the idea that visual motion had a simple representation in the Fourier domain (Fahle and Poggio, 1981; Watson and Ahumada, 1983). From the beginning, pairs of filters with a quadrature phase relationship were seen as critical elements for constructing the oriented spatiotemporal filters underlying the DS property of the models. Quadrature became inextricably linked to this approach following the introduction of the motion energy model (Adelson and Bergen, 1985), which used the sum of squares of the outputs of a quadrature pair of DS filters to transform a modulated, simple-cell-like response into an unmodulated, complex-cell-like response. Quadrature DS models, which were designed to be physiologically plausible, have received extensive electrophysiological support in V1 of cat and monkey (Pollen and Ronner, 1981; Emerson et al., 1992; Emerson, 1997; Emerson and Huang, 1997; De Valois and Cottaris, 1998; De Valois et al., 2000; Rust et al., 2005; Touryan et al., 2005).
One stimulus thought to provide evidence for quadrature models is the two-flash sinusoidal grating apparent motion stimulus. Using this stimulus, Nakayama and Silverman (1985) found that a 1/4 cycle displacement of a grating was optimal for psychophysical direction discrimination, and they interpreted this as being consistent with motion sensors having quadrature filters. This hypothesis was reinforced by Watson (1990), who reported that quadrature models generally predict an optimal displacement of 1/4 cycle in two-frame displays. The hypothesized link between quadrature phase and 1/4 cycle displacement has been picked up and incorporated into the motivation and interpretation of psychophysical, electrophysiological, and theoretical studies (Boulton and Hess, 1990; Baker et al., 1991; Pinkus and Pantle, 1997; Gepshtein et al., 2007).
Nevertheless, we tested the displacement tuning of the motion energy model and found that there was no link between quadrature and 1/4 cycle displacement preference. We verify analytically that quadrature does not lead to a preference for a 1/4 cycle step, and in particular that motion energy is not maximized for a 1/4 cycle step. We also demonstrate that a simple Reichardt model (Reichardt, 1961), which prefers 1/4 cycle step (van Santen and Sperling, 1985), is not a quadrature model, despite its mathematical links to some versions of a motion energy model. Finally, we will show that opponency is the key to the 1/4 cycle preference, and that nearly any linear filter followed by squaring is sufficient to construct an opponent motion model that prefers 1/4 cycle step for a classical two-flash grating stimulus, regardless of the spatial frequency of the grating.
To advance the understanding of the relevant models and facilitate their use in experimental design and data interpretation, we have made them available for interactive use at www.imodel.org under the topic “Opponency, Quadrature and Displacement Tuning for Two-Frame Motion.”
Materials and Methods
Motion energy model.
The basic motion energy (ME) unit, shown in Figure 1A, consists of the sum of squares of the outputs of two linear filters that have a quadrature phase relationship (Adelson and Bergen, 1985). Here, the quadrature filters, Ge and Go (even and odd), are Gabor functions, which are the product of a Gaussian and a sine wave, as follows:
where
and r is the position vector (x, y), and n is the unit vector (cos θ, sin θ) pointing in the preferred direction, θ, which is set to zero for rightward motion. The spatial frequency (SF) and temporal frequency (TF) tuning of the model are determined by the SF and TF of the sinusoid, fr and ft, respectively, and by the SD of the Gaussian in space and time, σr and σt, respectively.
ME unit and RD. A, Icons (top) show x–t cross sections through two Gabor linear filters (Eqs. 1, 2) that have a quadrature phase relationship. The sum of the squared filter outputs is the final ME output (bottom). B, The RD model using the notation of Adelson and Bergen (1985; their Fig. 18). Boxes indicate spatial and temporal linear filters.
The response of the ME unit is given by the sum of the squares of the convolutions (*) of the even and odd linear filters with the visual stimulus, s(x, y, t), as follows:
We need only consider the response at one spatial location, say x = xc, y = yc, the central position in the spatial field; therefore, the simpler notation, ME(t), suffices. The response of an opponent ME model is defined to be the difference between the responses, ME+(t) and ME−(t), of two ME units that differ only by having preferred directions, θ, that are 180° opposite.
Visual stimulus.
The two-flash grating stimulus (Nakayama and Silverman, 1985; Boulton and Hess, 1990; Baker et al., 1991) consists of a sinusoidal grating that is presented briefly at two distinct times, separated by ΔT, on an otherwise mean gray background. The spatial phase of the grating is ϕ on the first flash and ϕ + Δϕ on the second flash. The grating orientation is optimized for the motion detector (neuron or model) being tested. The two flashes occur on distinct video frames when presented on a CRT, and are modeled as δ functions in time, as follows:
where fs is the spatial frequency of the grating, and the grating is orientated parallel to the y-axis.
Results
We first characterize the displacement tuning of the quadrature motion energy unit to show that quadrature does not lead to a particular preferred displacement. Next, we demonstrate that motion models lacking quadrature filters can have a preference for 1/4 cycle displacement. Having shown that quadrature is neither necessary nor sufficient for the 1/4 cycle preference, we outline a simpler set of properties that leads to the 1/4 cycle preference.
Quadrature is not sufficient
We tested a quadrature ME model with the two-flash grating stimulus and found that the displacement tuning curves (Fig. 2) were sinusoidal and that the optimal displacement changed systematically with the TF used to construct the linear filters of the ME model. Repeating the test at a variety of initial phases, we observed that the location of the peak and the sinusoidal shape were phase independent and only the amplitude of response varied with initial phase. This showed that the ME model, despite its quadrature phase relationship, does not generally respond best to 1/4 cycle displacements.
Displacement tuning curves for three ME models constructed from different Gabor filters. Color indicates the TF (see legend) of the Gabor filters. Each response is normalized to fill the range from 0 to 1. ΔT was 10 ms. Vertical dashed lines indicate peaks.
To verify our simulation results, we derived (Appendix 1, available at www.jneurosci.org as supplemental material) that the response of the ME model is a function of phase displacement Δϕ, and found that it had a sinusoidal dependence on Δϕ (Appendix 1, available at www.jneurosci.org as supplemental material, Eq. 8), and that the optimal displacement, Δϕ*, was (Appendix 1, available at www.jneurosci.org as supplemental material, Eq. 9) as follows:
where ft is the TF of the ME model, Δt is the time between flashes of the grating, and a is a scalar that is close to 1 for typical ME models. Given a near 1, the following is true:
For example, when ft = 20 Hz and Δt = 0.10 s, the optimal displacement is 0.2 cyc, which is exactly what is observed in Figure 2 (red line).
From this, we conclude that the quadrature ME model has no particular preference for 1/4 cycle and that quadrature is not sufficient to produce a preference for 1/4 cycle step. Nevertheless, many motion models do in fact prefer a 1/4 cycle step (van Santen and Sperling, 1985; Watson, 1990), and given the literature that has linked this to quadrature, one might reasonably ask whether quadrature is necessary to generate the 1/4 cycle preference in those models, even if it is not sufficient.
Quadrature is not necessary
To demonstrate that quadrature filters are not necessary to produce a preference for 1/4 cycle displacement, we consider a Reichardt detector (RD) that conforms to Figure 1B. Such a model is a member of the class known as elaborated Reichardt detectors (ERDs), some members of which have previously been shown to have a 1/4 cycle displacement preference (van Santen and Sperling, 1985). We consider a very basic RD, which has two spatially offset point detectors and a simple temporal delay. Thus, its linear spatial and temporal filters (Fig. 1B, boxes) are as follows:
This particular ERD satisfies the constraints of van Santen and Sperling (1985; their Appendix A) for having a 1/4 cycle preference, yet the spatial filters, f1(x) and f2(x), and the temporal filters, h1(t) and h2(t), are not quadrature pairs. To be in quadrature, they must differ by a constant 90° phase shift, but instead they differ by a constant spatial (or temporal) displacement, resulting in a phase shift that varies with frequency.
We can also test whether the spatiotemporal, inseparable filters of this model are in quadrature. Adelson and Bergen (1985) showed that an ERD like that in Figure 1B is formally equivalent to an opponent ME model. In their Appendix, they show that a pair of equivalent spatiotemporal ME filters, say q1(x, t) and q2(x, t), can be computed from the separable spatial and temporal filters of the ERD by noting that their outputs are A − B′ and A′ + B (notation of Fig. 1B). Thus, for our simple RD model, the following is true:
The Fourier transforms (Bracewell, 1986) of these filters are as follows:
It is easy to verify that these functions do not have a quadrature relationship: at ωx = ωt = 0, they differ in amplitude (Q1 = 0, Q2 = 2) and therefore are not simply phase-shifted versions of each other.
In summary, the simple RD is not a quadrature model in either of the commonly used senses: its constituent separable filters are not in quadrature, nor are the resulting inseparable filters. Nevertheless, for the two-flash grating stimulus, the RD model exhibits the 1/4 cycle preference.
Given that quadrature is neither necessary nor sufficient for a 1/4 cycle preference, yet this preference is observed in DS neurons (Baker and Cynader, 1986; Baker et al., 1991) and in psychophysics (Nakayama and Silverman, 1985), what property of motion detection is important for achieving the 1/4 cycle phase preference?
Opponency plays a crucial role
A critical feature of the RD model considered above and of the quadrature energy models examined by Watson (1990) is that they are opponent. That is, they consist of a subtraction of subunits that are identical except for being tuned to opposite directions. To see how opponency can create a preference for 1/4 cycle, consider the displacement tuning curve, D(Δϕ), for the 20 Hz ME+ model, which is replotted in Figure 3A (solid black line). An oppositely tuned unit, ME−, has a displacement tuning curve, D(−Δϕ) (Fig. 3A, dotted line), that is the reflection of the original around the vertical axis. The resulting tuning curve for the opponent model, Dopp (Fig. 3A, red line), is as follows:
which is a sine function in Δϕ, and therefore must peak at 1/4 cycle. Here, β is the phase shift of the original nonopponent tuning curve. This gives the following result: Any opponent motion model formed from a subunit that has a sinusoidal tuning curve that is directional (i.e., does not peak at 0 or 1/2 cycle) will prefer 1/4 cycle displacement. In Appendix 2 (available at www.jneurosci.org as supplemental material), we show that a very broad class of model has sinusoidal displacement tuning curves, in particular, any product of the outputs of two linear filters, L1 × L2. This includes all ERD models (Fig. 1B) and, when L1 = L2, any linear-nonlinear (L-N) model where the nonlinearity is squaring, which includes opponent ME models. More generally, the requirement of a sinusoidal displacement tuning can be relaxed to include a broader class of function that shares properties of symmetry and convexity with sinusoids, but this goes beyond the scope of this paper. For a more general framework for studying symmetrized (opponent) systems, see Poggio and Reichardt (1973).
The effect of a direction-opponent architecture on displacement tuning. A, Displacement tuning curves (response vs displacement) are plotted for the 20 Hz Gabor ME model (black solid line) and for a similar model preferring oppositely directed motion (dotted line). The tuning curve for the opponent model (red line) is simply the difference of the black lines (solid minus dotted). B, Spatiotemporal filters were chosen by randomly setting pixels to black (−1) or white (+1), multiplying by a Gaussian window in space and time, and smoothing with a Gaussian (SD 4 ms) in time. The left and right filters are mirror images in the x-axis. The output of each filter was squared and the integral of the difference of the two squared signals was the final output. C, The displacement tuning curve for the random filter model is sinusoidal and peaks at 1/4 cyc, like that of the opponent ME model in A.
It may seem counterintuitive that virtually any filter can be used to construct a model that prefers a 1/4 cycle step and is direction selective for the two-flash grating stimulus. To verify this, we created an opponent model using subunits consisting of a random spatiotemporal filter (Fig. 3B) followed by squaring. When tested with the two-flash grating stimulus, the displacement tuning curve (Fig. 3C) was sinusoidal, showing the consistency of the computer model and analytical derivations.
Discussion
Our results overturn the link between space–time oriented-filter quadrature motion models and the preference for 1/4 cycle displacement hypothesized in the literature for 25 years. We will discuss the origin of this assumption, how our results relate to past work and how they change the interpretation of selected studies.
To appreciate the relationship, or lack thereof, between quadrature models and stimulus displacement, it is useful to recall that quadrature was introduced into the motion literature in at least two distinct ways. First, Watson and Ahumada (1983) demonstrated that a single oriented, X–T inseparable filter (Fig. 1A, either icon) can be constructed by an appropriate combination of two separable filters in quadrature. The preferred speed, thus displacement, of the resulting filter has nothing to do with the constructional artifice of quadrature, but depends on the degree of X–T slant of the oriented filter. To best drive the filter, a grating should be displaced by an amount in space–time that matches the slant of the filter. This precludes any special status of 1/4 cycle step. Adelson and Bergen (1985) introduced quadrature in a second sense by pairing one oriented filter with another in quadrature (Fig. 1A, icons). Nevertheless, each filter senses motion separately and each responds to both flashes of a displaced grating. In no sense can one grating flash be assigned to one filter and the second flash to the other. Thus, there is no relationship between the displacement of a grating stimulus and the 90° shift between the filters. The quadrature pair (with squaring) is merely a device to remove the phase modulation of the output of oriented motion filters, making responses more like those of complex rather than simple DS cells.
This provides a foundation to understand how Nakayama and Silverman (1985) connected the quadrature models of van Santen and Sperling (1985), which encompass the ME model (Adelson and Bergen, 1985) and were a novelty at the time, to the idea that “the 90-deg phase shift is optimal for discrimination.” They equated the 90° offset of two quadrature spatial RFs with the spatial offset of the displaced grating, explicitly assigning each grating flash to a different filter. It is attractive to connect the offset of two spatial filters in a motion detector to the spatial offset of two frames of a motion stimulus, particularly because the earlier and simpler Reichardt model invokes this very principle (Reichardt, 1961). Nevertheless, this omits critical temporal aspects of robust ERDs and is not how the quadrature motion models of Watson and Ahumada (1983) or Adelson and Bergen (1985) work, as outlined above. Thus, the coexistence of simpler Reichardt-inspired quadrature models [e.g., Nakayama (1985), his Fig. 3B] and the ME model is one major source of confusion in the literature.
The second source of confusion is that, while quadrature does not confer a preference for 1/4 cycle motion in the oriented-filter models, an opponent architecture often does (see Results), and the influential quadrature ME model is frequently used in its opponent form. This reconciles the apparent contradiction between our results and Watson (1990), who concluded that, “Quadrature models generally predict an optimal displacement of 1/4 cycle for two frame displays.” Watson studied quadrature energy models that also happened to be opponent, where signals of oppositely tuned subunits were subtracted. We conclude that opponency was a critical factor and that quadrature did not matter to the 1/4 cycle preference that Watson observed.
Consistent with our derivations, van Santen and Sperling (1985) showed that some forms of the ERD prefer 1/4 cycle displacement in two-frame displays. This potential clue that quadrature was not the determining factor was overlooked. Our derivations extend those of van Santen and Sperling by showing that (1) all ERDs have a 1/4 cycle preference, and (2) a more general class of model, one not restricted to separable filters like the ERD, also has a 1/4 cycle preference.
Our results allow reinterpretation of the relevant literature since 1985. This literature is vast, and the examples mentioned below are not exhaustive. Nakayama and Silverman (1985) showed that 1/4 cycle displacement of a grating was optimal for human psychophysical direction discrimination and suggested that this is consistent with a 90° phase shift in newly described motion detector models. Based on our findings, their results more likely reflect opponency, not quadrature, in the motion pathway. This interpretation is appealing because opponency has a long history of support. For example, Adelson and Bergen (1985) argued for opponency based on (1) the difficulty in perceiving left and right motion at the same place, time, and frequency band, (2) the existence of the motion aftereffect, and (3) the ability of opposite motions to cancel perceptually. This raises the question as to where opponency occurs. V1 cells are not consistent with fully opponent mechanisms (Emerson et al., 1992). If motion discrimination is driven by a higher-level population response, perhaps opponency is implemented using signals within or extracted from V5/MT (Heeger et al., 1999). Our results show that two population signals tuned for opposite directions and having sinusoidal displacement tuning curves are sufficient to create the 1/4 cycle preference via simple subtraction.
Just as 1/4 cycle preference does not imply quadrature, later psychophysical studies reporting 1/6 to 1/5 cycle (Boulton and Hess, 1990) need not imply deviations from quadrature in the underlying mechanisms. We demonstrated that an ideal quadrature ME model (nonopponent) can easily display any of these values, determined by the TF of the filters (Fig. 2) and the ΔT of the display. Thus, it is more likely that a deviation from 1/4 cycle reflects a lack of, or imbalanced, opponency.
The same reasoning applies to electrophysiological studies (Baker and Cynader, 1986; Baker et al., 1991) that were premised upon, or interpreted in terms of, the hypothesis that finding 1/4 cycle would provide evidence for quadrature at the level of single units in V1. This suggests that other techniques are needed to ascertain what phase relationship holds for DS subunits (Peterson et al., 2004; Rust et al., 2005; Touryan et al., 2005; Peterson et al., 2006).
A variety of other motion studies have incorporated the connection between a 1/4 cycle preference and quadrature ME models into their reasoning. Pinkus and Pantle (1997) probed the origin of visual motion priming (VMP) and reasoned that if ME units drive the VMP signal, then VMP should be strongest for 1/4 cycle displacements, citing Watson (1990). They found VMP to be maximal for a 90° step, but this is no longer evidence for quadrature or ME over other opponent motion models. A recent economic theory approach to motion perception (Gepshtein et al., 2007) also relies on a connection between quadrature models and 1/4 cycle displacement. However, the models most relevant to the study, which seeks to explain the SF-TF psychophysical sensitivity data of Kelly (1979), are the SF-TF-tuned oriented filter motion models (Watson and Ahumada, 1983; Adelson and Bergen, 1985), for which there is no connection between quadrature and 1/4 cycle displacement.
Presciently, one study hesitated to make this connection. Baker et al. (1989, p. 857) remarked, “It would seem intuitively plausible to expect a maximal response from one of these motion detector models if the spatial displacement in apparent motion were exactly one fourth the spatial wavelength (reciprocal of spatial frequency) of the detector's component spatial filters; however, it would be useful to verify this with computer simulations.” It was in fact computer simulation that revealed our finding.
An important implication of this is that well studied, widely accepted, and conceptually minimal models for sensory processing can remain obscure even to experts. This may relate to the absence of published, interactive simulations that allow investigators to easily test how commonly cited models respond to stimuli. To address this, we have put the models discussed here online to facilitate understanding and allow interactive testing with visual stimuli. The relevant models can be found at www.imodel.org under the topic “Opponency, Quadrature and Displacement Tuning for Two-Frame Motion.”
Footnotes
-
This work was supported by a Wellcome Trust Senior Research Fellowship in the Basic Biomedical Sciences. N.H. was supported by an Engineering and Physical Sciences Research Council/Medical Research Council scholarship from the Neuroinformatics and Computational Neuroscience Doctoral Training Centre at the University of Edinburgh. W.B. was also supported by a Science Research Fellowship at St. John's College, Oxford. We thank Anitha Pasupathy, Pamela Baker, and Douglas McLelland for comments on the manuscript.
- Correspondence should be addressed to Nicolas Heess, Neuroinformatics and Computational Neuroscience Doctoral Training Centre, Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK. n.m.o.heess{at}sms.ed.ac.uk