In natural viewing, a visual stimulus that is the target of attention is generally surrounded by many irrelevant distracters. Stimuli falling in the receptive field surround can influence the neuronal response evoked by a stimulus appearing within the classical receptive field. Such modulation by task-irrelevant distracters may degrade the target-related neuronal signal. We therefore examined whether directing attention to a target stimulus can reduce the influence of task-irrelevant distracters on neuronal response. We find that in area V4 attention to a stimulus within a neuron's receptive field filters out a large fraction of the suppression induced by distracters appearing in the surround. When attention is instead directed to the surround stimulus, suppression is increased, thereby filtering out part of the neuronal response to the irrelevant distracter positioned within the receptive field. These findings demonstrate that attention modulates the neural mechanisms that give rise to center-surround interactions.