We used high-level configural aftereffects induced by adaptation to realistic faces to investigate visual representations underlying complex pattern perception. We found that exposure to an individual face for a few seconds generated a significant and precise bias in the subsequent perception of face identity. In the context of a computationally derived 'face space,' adaptation specifically shifted perception along a trajectory passing through the adapting and average faces, selectively facilitating recognition of a test face lying on this trajectory and impairing recognition of other faces. The results suggest that the encoding of faces and other complex patterns draws upon contrastive neural mechanisms that reference the central tendency of the stimulus category.