In everyday life, people select motor responses according to arbitrary rules. For example, our movements while driving a car can be instructed by color cues that we see on traffic lights. These stimuli do not spatially relate to the actions that they specify. Associations between these stimuli and actions are called arbitrary visuomotor conditional associations. Earlier fMRI studies have tried to dissociate the sensory and motor components of these associations by introducing delays between the presentation of arbitrary cues and go-signals that instructed participants to perform actions. This approach, however, also introduces neural processes that are not necessarily related to the normal real-time production of arbitrary visuomotor responses, such as working memory and the suppression of motor responses. We used fMRI adaptation as an alternative approach to dissociate sensory and motor components. We found that visual areas in the occipital-temporal cortex adapted only to the presentation of arbitrary visual cues whereas a number of sensorimotor areas adapted only to the production of response. Visual areas in the occipital-temporal cortex do not have any known connections with parts of the brain that can control hand musculature. Therefore, it is conceivable that the brain areas that we report as having adapted to both stimulus presentation and response production (namely, the dorsal premotor area, the supplementary motor area, the cingulate, the anterior intra-parietal sulcus area, and the thalamus) are involved in the multiple steps between processing visual stimuli and activating the motor commands that these cues specify.