Dot patterns sliding transparently across one another are normally perceived as independently moving surfaces. Recordings from direction-selective neurons in area MT of the macaque suggested that this perceptual segregation did not depend on the presence of two peaks in the population activity. Rather, the visual system seemed to use overall shape of the population response to determine the number and directions of motion components. This approach explained a number of perceptual phenomena, including susceptibility of the motion system to direction metamers, motion patterns combining three or five directions incorrectly perceived by subjects as comprising only two directions. Our findings offer insights into the coding of multi-valued sensory signals and provide constraints for biologically based computational models.