Neurophysiological data support two models for the disparity selectivity of binocular simple and complex cells in primary visual cortex. These involve binocular combinations of monocular receptive fields that are shifted in retinal position (the position-shift model) or in phase (the phase-shift model) between the two eyes. This article presents a formal description and analysis of a binocular energy model with these forms of disparity selectivity. We propose how one might measure the relative contributions of phase and position shifts in simple and complex cells. The analysis also reveals ambiguities in disparity encoding that are inherent in these model neurons, suggesting a need for a second stage of processing. We propose that linear pooling of the binocular responses across orientations and scales (spatial frequency) is capable of producing an unambiguous representation of disparity.