In daily life, visual working memory (VWM) typically works in contexts in which people make frequent saccades. Here, we investigated whether people can effectively control the allocation of VWM when making a saccade. Subjects were required to make an intervening saccade in the process of a memory task. The saccade target was either a to-be-remembered item or an extra, not-to-be-remembered item. The results showed that memory performance was poorer when a saccade was made to the extra, not-to-be-remembered item, regardless of its similarity to the memory item(s). In contrast, when memorizing the items while remaining fixated, subjects had similar memory performance whether an extra, not-to-be-remembered item was present or not. Taken together, these results demonstrated that volitional control over VWM allocation is greatly impaired when a saccade is made, indicating that VWM contains an automatic part that cooperates with eye movements and is allocated to a saccade target obligatorily.