The responses of neurons in the visual cortex to stimuli presented within their receptive fields can be markedly modulated by stimuli presented in surrounding regions that do not themselves evoke responses. This modulation depends on the relative orientation and direction of motion of the centre and surround stimuli, and it has been suggested that local cortical circuits linking cells with similar stimulus selectivities underlie these phenomena. However, the functional relevance and nature of these integrative processes remain unclear. Here we investigate how such integration depends on the relative activity levels of neurons at different points across the cortex by varying the relative contrast of stimuli over the receptive field and surrounding regions. We show that simply altering the balance of the excitation driving centre and surround regions can dramatically change the sign and stimulus selectivity of these contextual effects. Thus, the way that single neurons integrate information across the visual field depends not only on the precise form of stimuli at different locations, but also crucially on their relative contrasts. We suggest that these effects reflect a complex gain-control mechanism that regulates cortical neuron responsiveness, which permits dynamic modification of response properties of cortical neurons.