People are extremely proficient at recognizing faces that are familiar to them, but are poor at identifying unfamiliar faces. We used fMR-adaptation to ask whether this difference in recognition might be reflected in the relative viewpoint-dependence of face-selective regions in the brain. A reduced response (adaptation) to repeated images of unfamiliar or familiar faces was found in the fusiform face area (FFA), but not in the superior temporal sulcus (STS) face-selective region. To establish if the neural representation of faces was invariant to changes in viewpoint, we parametrically varied the viewing angle of successive images using 3-dimensional models of unfamiliar and familiar faces. We found adaptation to familiar faces across all changes in viewpoint in the FFA. In contrast, a release from adaptation was apparent in the FFA when unfamiliar faces were viewed at increasing viewing angles. These results provide a neural basis for differences in the recognition of familiar and unfamiliar faces.