Integration of inputs by cortical neurons provides the basis for the complex information processing performed in the cerebral cortex. Here, we have examined how primary visual cortical neurons integrate classical and nonclassical receptive field inputs. The effect of nonclassical receptive field stimuli and, correspondingly, of long-range intracortical inputs is known to be context-dependent: the same long-range stimulus can either facilitate or suppress responses, depending on the level of local activation. By constructing a large-scale model of primary visual cortex, we demonstrate that this effect can be understood in terms of the local cortical circuitry. Each receptive field position contributes both excitatory and inhibitory inputs; however, the inhibitory inputs have greater influence when overall receptive field drive is greater. This mechanism also explains contrast-dependent modulations within the classical receptive field, which similarly switch between excitatory and inhibitory. In order to simplify analysis and to explain the fundamental mechanisms of the model, self-contained modules that capture nonlinear local circuit interactions are constructed. This work supports the notion that receptive field integration is the result of local processing within small groups of neurons rather than in single neurons.