Autism is a severe developmental disorder marked by a triad of deficits, including impairments in reciprocal social interaction, delays in early language and communication, and the presence of restrictive, repetitive and stereotyped behaviors. In this review, it is argued that the search for the neurobiological bases of the autism spectrum disorders should focus on the social deficits, as they alone are specific to autism and they are likely to be most informative with respect to modeling the pathophysiology of the disorder. Many recent studies have documented the difficulties persons with an autism spectrum disorder have accurately perceiving facial identity and facial expressions. This behavioral literature on face perception abnormalities in autism is reviewed and integrated with the functional magnetic resonance imaging (fMRI) literature in this area, and a heuristic model of the pathophysiology of autism is presented. This model posits an early developmental failure in autism involving the amygdala, with a cascading influence on the development of cortical areas that mediate social perception in the visual domain, specifically the fusiform "face area" of the ventral temporal lobe. Moreover, there are now some provocative data to suggest that visual perceptual areas of the ventral temporal pathway are also involved in important ways in representations of the semantic attributes of people, social knowledge and social cognition. Social perception and social cognition are postulated as normally linked during development such that growth in social perceptual skills during childhood provides important scaffolding for social skill development. It is argued that the development of face perception and social cognitive skills are supported by the amygdala-fusiform system, and that deficits in this network are instrumental in causing autism.