It has been controversial whether the face-sensitive N170 is affected by selective attention. We manipulated attention sensu Lavie's perceptual load theory to short (200 ms) presentations of task-irrelevant unfamiliar faces or houses, while participants identified superimposed target letters 'X' versus 'N'. These targets were strings of either six identical (low load) or six different letters (high load). Under low load, we found a prominent face-selective N170 response. Under high load, however, we not only observed a dramatic reduction of the face N170 but also an unexpected enhancement of the house N170, such that face selectivity was almost completely lost. We conclude that the early stages of face processing indexed by the N170 strongly depend on selective attention.