Humans have an extraordinary ability to maintain and manipulate visual image information in the absence of perceptual stimulation. The neural substrates of visual working memory have been extensively researched, but there have been few attempts to integrate these findings into a model of how different cortical areas interact to form and maintain visual memories. In this paper, I review findings from neurophysiological, neuropsychological, and neuroimaging studies of visual working memory in human and nonhuman primates. These data support a model in which visual working memory operations rely on activation of object representations in inferior temporal cortex, via top-down feedback from neocortical areas in the prefrontal and medial temporal cortex, and also from the hippocampus.