Immediate access to information about people that we encounter is an essential requirement for effective social interactions. In this manuscript we briefly review our work and work of others on familiar face recognition and propose a modified version of our model of neural systems for face perception with a special emphasis on processes associated with recognition of familiar faces. We argue that visual appearance is only one component of successful recognition of familiar individuals. Other fundamental aspects include the retrieval of "person knowledge" - the representation of the personal traits, intentions, and outlook of someone we know - and the emotional response we experience when seeing a familiar individual. Specifically, we hypothesize that the "theory of mind" areas, that have been implicated in social and cognitive functions other than face perception, play an essential role in the spontaneous activation of person knowledge associated with the recognition of familiar individuals. The amygdala and the insula, structures that are involved in the representation of emotion, also are part of the distributed network of areas that are modulated by familiarity, reflecting the role of emotion in face recognition.