Face perception is fundamentally important for judging the characteristics of individuals, such as identification of their gender, age, ethnicity or expression. We asked how the perception of these characteristics is influenced by the set of faces that observers are exposed to. Previous studies have shown that the appearance of a face can be biased strongly after viewing an altered image of the face, and have suggested that these after-effects reflect response changes in the neural mechanisms underlying object or face perception. Here we show that these adaptation effects are pronounced for natural variations in faces and for natural categorical judgements about faces. This suggests that adaptation may routinely influence face perception in normal viewing, and could have an important role in calibrating properties of face perception according to the subset of faces populating an individual's environment.