Neurons in primary visual cortex (V1) respond differently to a simple visual element presented in isolation from when it is embedded within a complex image. This difference, a specific modulation by surrounding elements in the image, is mediated by short- and long-range connections within V1 and by feedback from other areas. Here we study the role of short-range connections in this process, and relate it to the layout of local inhomogeneities in the cortical maps of orientation and space. By measuring correlation between neuron pairs located in optically imaged maps of V1 orientation columns we show that the strength of local connections between cells is a graded function of lateral separation across cortex, largely radially symmetrical and relatively independent of orientation preferences. We then show the contextual influence of flanking visual elements on neuronal responses varies systematically with a neuron's position within the cortical orientation map. The strength of this contextual influence on a neuron can be predicted from a model of local connections based on simple overlap with particular features of the orientation map. This indicates that local intracortical circuitry could endow neurons with a graded specialization for processing angular visual features such as corners and T junctions, and this specialization could have its own functional cortical map, linked with the orientation map.