Precision in Visual Working Memory Reaches a Stable Plateau When Individual Item Limits Are Exceeded

By varying the number of orientations observers must hold in mind and examining the precision of the representations held in working memory, Anderson, Vogel and Awh (2011) show that memory precision hits a plateau at approximately 3 items, after which observers are more likely to guess but seem to have no loss in precision. This is compatible with a discrete resource model in which observers have, on average, 3 fixed resolution slots that can each hold one individual item or can be pooled to represent 1 or 2 items with greater precision (Zhang & Luck, 2008). However, since individuals differ in working memory capacity (e.g., Vogel & Awh, 2008), there should be a variable rather than a constant number of slots across individuals. To address this important issue of individual differences, Anderson et al. (2011) present an analysis correlating two different, putatively independent measures of how many discrete slots observers have. In particular, they look at both the "inflection point" in observers' precision - the number of items after which each individual observer's representations seems to have stopped decreasing in precision (SD) - and also at the number of items observers seem to have information about (P(mem)) at the highest set size. They find a striking correlation between these two measures, which they interpret as evidence that each individual observer has a specific fixed number of discrete memory slots.

Unfortunately, these results are not evidence for the discrete resource model, because the analysis correlated non-independent measures - the same data contributes to both the x- and y-axes. In particular, the SD and P(mem) parameters in the model are fit simultaneously and thus trade off with each other. Consider that models are inferred based on only a limited number of samples. A model will underestimate both P(mem) and SD if some of the more inaccurate memory responses are estimated to be guess responses, whereas the converse will lead to an overestimation of P(mem) and SD. To illustrate this point, we performed several simulations and found that even in randomly generated data, if the estimated P(mem) is higher, the estimated SD tends to be higher as well. This means the SD from the highest set size will necessarily be correlated with the P(mem) from the highest set size. Concretely, if we generate random data sampled from a true SD of 18 and a true P(mem)=0.30 (mimicking Anderson et al. Experiment 1, set size 8), and sample 100 "people" with 120 trials each, we find a correlation of r=0.66 between the fit P(mem) and SD for each person. In other words, even for randomly generated data, those individuals estimated to have a larger P(mem) are also estimated to have larger SDs, even though there is no correlation between these parameters in the simulation, and even though on average the correct parameters are recovered by the model.

The non-independent analysis reported in Anderson et al. will also produce a spurious correlation between the SD inflection point and P(mem) at set size 8 since high estimates of SD at set size 8 will push the inflection point higher. In our simulations, when we sample random data for 100 "people" using the set sizes and mean SDs from Anderson et al. Exp 1 we find a correlation of r=0.50 between SD inflection and P(mem) at set size 8. Importantly, this correlation, which was taken by Anderson et al. as evidence for a variable number of discrete slots, arose from randomly generated data with no inherent variability across simulated observers.

To correct this problem, the data should be reanalyzed by estimating the inflection point in SD using set sizes 1-6 only (i.e., not using the same data that is used to estimate P(mem)). This corrects the non- independence in the analysis. However, care must still be taken when interpreting such a correlation, because lower values of P(mem) (across set sizes or across individuals who may be consistently worse in the task) result in greater noise in the estimation of SD. In some cases, depending on how the inflection in SD is measured, this can result in inflated estimates of the inflection point, thus reintroducing non-independence into the analysis.

In general, neuroscientists must be aware of the potential for non- independence between different factors in data analysis - such non- independence can occur not only from selection biases in choosing which data to analyze (Kriegeskorte, Simmons, Bellgowan & Baker, 2009), but also when fitting models to data.

References

Anderson, D.E., Vogel, E.K., & Awh, E. (2011). Precision in visual working memory reaches a plateau when individual item-limits are exceeded. Journal of Neuroscience, 31(3), 1128-1138.

Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S.F. & Baker, C.I. (2009). Circular analysis in systems neuroscience--the dangers of double dipping. Nature Neuroscience, 12(5), 535.

Vogel, E. & Awh, E. (2008). How to exploit diversity for scientific gain: Using individual differences to constrain cognitive theory. Current Directions in Psychological Science, 17(2), 171-176.

Zhang, W. & Luck, S.J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233-235.

Conflict of Interest:

None declared