Top-down modulation underlies our ability to selectively attend to relevant stimuli and to ignore irrelevant stimuli. Theories addressing neural mechanisms of top-down modulation are driven by studies that reveal increased magnitude of neural activity in response to directed attention, but are limited by a lack of data reporting modulation of neural processing speed, as well as comparisons with a perceptual (passive view) baseline necessary to evaluate the presence of enhancement and suppression. Utilizing functional MRI (fMRI) and event-related potential recordings (ERPs), we provide converging evidence that both the magnitude of neural activity and the speed of neural processing are modulated by top-down influences. Furthermore, both enhancement and suppression occur relative to a perceptual baseline depending on task instruction. These findings reveal the fine degree of influence that goal-directed attention exerts upon activity within the visual association cortex. We further document capacity limitations in top-down enhancement corresponding with working memory performance deficits.