The capacity of visual short-term memory is highly limited, maintaining only three to four objects simultaneously. This extreme limitation necessitates efficient mechanisms to select only the most relevant objects from the immediate environment to be represented in memory and to restrict irrelevant items from consuming capacity. Here we report a neurophysiological measure of this memory selection mechanism in humans that gauges an individual's efficiency at excluding irrelevant items from being stored in memory. By examining the moment-by-moment contents of visual memory, we observe that selection efficiency varies substantially across individuals and is strongly predicted by the particular memory capacity of each person. Specifically, high capacity individuals are much more efficient at representing only the relevant items than are low capacity individuals, who inefficiently encode and maintain information about the irrelevant items present in the display. These results provide evidence that under many circumstances low capacity individuals may actually store more information in memory than high capacity individuals. Indeed, this ancillary allocation of memory capacity to irrelevant objects may be a primary source of putative differences in overall storage capacity.