The visual system relies on patterns of light to provide information about the layout of objects that populate our environment. Light is structured by the way it interacts with the three-dimensional shape, reflectance, and transmittance properties of objects. The input for vision is therefore a complex, conflated mixture of different sources of physical variation that the brain must somehow disentangle to recover the intrinsic properties of the objects and materials that fill the world.