Face recognition is more strongly impaired by stimulus inversion than nonface object recognition. This phenomenon, known as the face inversion effect (FIE), suggests that the visual system contains specialized processing mechanisms that are more engaged by upright faces than by inverted faces or nonface objects. Neuroimaging and neuropsychological studies indicate that environmental scenes may also recruit specialized-purpose processing machinery but a comparable inversion effect for scenes has not been established. Here we demonstrate that both face and scene inversion lead to behavioral penalties during performance of a continuous visual matching task; however, the scene inversion effect was less robust and declined in magnitude over the course of the experiment. Scene inversion led to greater neural response in the functionally defined lateral occipital (LO) object area for inverted versus upright scenes and reduced response in the parahippocampal place area (PPA), while face inversion lead to greater response in LO and the right middle fusiform (MF) object area for inverted versus upright faces but no change in the fusiform face area (FFA). A whole-brain analysis revealed several regions that responded more strongly to either upright versus inverted faces or upright versus inverted scenes, some of which may be involved in post-recognition processing. These results demonstrate that both face and scene inversion cause a shift from specialized processing streams towards generic object-processing mechanisms; however, this shift only leads to a reliable behavioral penalty in the case of face inversion.