Visual selection is thought to be guided by both bottom-up intrinsic visual saliency and top-down visual attention. We examined how the relative importance of each of these processes dynamically changes over the course of a visual search in area V4 of two macaque monkeys. The animals were trained to perform a multidimensional visual search task in which a search array that consisted of two singleton stimuli, unique in shape and color, and four nonsingleton stimuli was presented. The task was to select one of the singletons as the saccade target, depending on the instructed search dimension, which was switched between shape and color. The strengths of the neural modulations caused by the bottom-up and top-down factors were, respectively, evaluated by assessing the dependence of the visual responses on the singleton type and the behavioral significance of the receptive field stimulus. We found that the initial visual responses of V4 neurons predominantly specified the singleton type of the receptive field stimulus, while the late presaccadic response primarily specified its behavioral significance. For comparison, recordings were also made from the frontal eye field (FEF) of one monkey and we found that the predominant activity of FEF neurons signaled the behavioral significance of the receptive field stimulus over time. These findings suggest that area V4 might function as an intermediate stage where both sensory- and behavior-based signals are dynamically represented, depending on their online requirements during a visual search, whereas the FEF predominantly represents behavior-based signals for upcoming saccade responses.