It is widely held that the frontal eye field (FEF) in prefrontal cortex (PFC) modulates processing in visual cortex with attention, although the evidence that it is necessary is equivocal. To help identify critical sources of attentional feedback to area V4, we surgically removed the entire lateral PFC, including the FEF, in one hemisphere and transected the corpus callosum and anterior commissure in two macaques. This deprived V4 of PFC input in one hemisphere while keeping the other hemisphere intact. In the absence of PFC, attentional effects on neuronal responses and synchrony in V4 were substantially reduced and the remaining effects of attention were delayed in time, indicating a critical role for PFC. Conversely, distracters captured attention and influenced V4 responses. However, because the effects of attention in V4 were not eliminated by PFC lesions, other sources of top-down attentional control signals to visual cortex must exist outside of PFC.