Ventral visual cortex contains specialized regions for particular object categories, but little is known about how these regions interact during object recognition. Here we examine how the face-selective fusiform gyrus (FG) and the scene-selective parahippocampal cortex (PHC) interact with each other and with the rest of the brain during different visual tasks. To assess these interactions, we developed a novel approach for identifying patterns of connectivity associated with specific task sets, independent of stimulus-evoked responses. We tested whether this "background connectivity" between the FG and PHC was modulated when subjects engaged in face and scene processing tasks. In contrast to what would be predicted from biased competition or intrinsic activity accounts, we found that the strength of FG-PHC background connectivity depended on which category was task relevant: connectivity increased when subjects attended to scenes (irrespective of whether a competing face was present) and decreased when subjects attended to faces (irrespective of competing scenes). We further discovered that posterior occipital cortex was correlated selectively with the FG during face tasks and the PHC during scene tasks. These results suggest that category specificity exists not only in which regions respond most strongly but also in how these and other regions interact.