Precision in Visual Working Memory Reaches a Stable Plateau When Individual Item Limits Are Exceeded

David E. Anderson; Edward K. Vogel; Edward Awh

doi:10.1523/JNEUROSCI.4125-10.2011

Published eLetters

Guidelines

As a forum for professional feedback, submissions of letters are open to all. You do not need to be a subscriber. To avoid redundancy, we urge you to read other people's letters before submitting your own. Name, current appointment, place of work, and email address are required to send a letter, and will be published with your review. We also require that you declare any competing financial interests. Unprofessional submissions will not be considered or responded to.

Submit a Response to This Article

Compose eLetter

Title *

Author Information

Contributors
First Name and Middle Initial * First or given name, e.g. 'Peter'. Last Name * Your last, or family, name, e.g. 'MacMoody'. Email Address * Your email address, e.g. higgs-boson@gmail.com Occupation * Your role and/or occupation, e.g. 'Orthopedic Surgeon'. Affiliation * Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.

Statement of Competing Interests

Competing interests? *

Yes

Please describe the competing interests

CAPTCHA

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Math question * 17 + 2 =

Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.

Vertical Tabs

Jump to comment:

Working memory capacity continues to predict discrete limits in resource allocation after correcting for non-independence
David E. Anderson

Submitted on: 15 November 2011
Comparisons between different measures of working memory capacity must be made with estimates that are derived from independent data
Timothy F. Brady

Submitted on: 14 October 2011

Submitted on: (15 November 2011)

Working memory capacity continues to predict discrete limits in resource allocation after correcting for non-independence
David E. Anderson
Other Contributors:
Edward Awh
Precision in working memory declines as the number of items stored increases (Zhang and Luck, 2008). Discrete resource models predict that this decline will reach a stable plateau at relatively small set sizes, however, because fixed item limits preclude the storage of additional items. We recently confirmed this prediction by showing that mnemonic precision declined until about set size three, and then reach a stable...
Show More

Precision in working memory declines as the number of items stored increases (Zhang and Luck, 2008). Discrete resource models predict that this decline will reach a stable plateau at relatively small set sizes, however, because fixed item limits preclude the storage of additional items. We recently confirmed this prediction by showing that mnemonic precision declined until about set size three, and then reach a stable plateau for larger set sizes (Anderson, Vogel and Awh, 2011). In addition, we confirmed another clear prediction of discrete resource models, that variations in item limits across individuals should correlate with the set size at which precision reached a plateau for each observer. In line with this prediction, behavioral and neural estimates of individual item limits strongly predicted the set size at which mnemonic precision reached a plateau for each observer.
In their recent comment, however, Brady, Fougnie and Alvarez make the important point that our behavioral estimate of item limits (Pmem, for the probability of storage) was calculated using the same dataset that was used to estimate mnemonic precision (SD, for the standard deviation of the target-related part of the response distribution), and that the dependence between these measures can artificially inflate the correlation between Pmem and the inflection point of the SD by set size function. We acknowledge that this non-independence problem inflated our estimate of the correlation between the behavioral measure of capacity and SD inflection, and we thank Brady et al. for pointing out this issue. That said, we would also like to emphasize that the correlation in question was also clearly evident in our second experiment when individual item limits were estimated using only storage-related neural activity (Vogel and Machizawa, 2004). Because this neural measure of individual item limits was independent of the behavioral data used to estimate precision, the resulting correlation between item limits and SD inflection (R2 = .45) in that study is not subject to the criticism raised by Brady et al. Moreover, a re-analysis of our behavioral data shows that the correlation between Pmem and SD inflection is still robust when SD inflection is estimated while leaving out the largest set so that independent data sets contribute to each parameter estimate (R2=.38 and R2=.32 for experiments 1 and 2, respectively).
In summary, the key correlation reported by Anderson et al. (2011) is robust when issues of non-independence are fully resolved.. Furthermore, the bilinear form of the precision by set size function cannot be explained by non-independence issues, and this empirical pattern also provides clear evidence for the item limit suggested by discrete resource models (see also Zhang and Luck, 2008). In conclusion, we respectfully acknowledge the technical flaw pointed out by Brady et al., but we have also shown that our data continue to provide strong support for discrete resource models of capacity in working memory.
References:
Anderson, D.E., Vogel, E.K., & Awh, E. (2011). Precision in visual working memory reaches a stable plateau when individual item limits are exceeded. Journal of Neuroscience, 31, 1128-1138.
Vogel, E.K. & Machizawa, M.G. (2004). Neural activity predicts individual differences in working memory capacity. Nature, 428, 784-751.
Zhang W. & Luck, S.J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453, 233-235.
Conflict of Interest:
None
Show Less

Competing Interests: None declared.
Submitted on: (14 October 2011)

Comparisons between different measures of working memory capacity must be made with estimates that are derived from independent data
Timothy F. Brady, Postdoctoral Fellow
Other Contributors:
Daryl Fougnie, George A. Alvarez
By varying the number of orientations observers must hold in mind and examining the precision of the representations held in working memory, Anderson, Vogel and Awh (2011) show that memory precision hits a plateau at approximately 3 items, after which observers are more likely to guess but seem to have no loss in precision. This is compatible with a discrete resource model in which observers have, on average, 3 fixed reso...
Show More

By varying the number of orientations observers must hold in mind and examining the precision of the representations held in working memory, Anderson, Vogel and Awh (2011) show that memory precision hits a plateau at approximately 3 items, after which observers are more likely to guess but seem to have no loss in precision. This is compatible with a discrete resource model in which observers have, on average, 3 fixed resolution slots that can each hold one individual item or can be pooled to represent 1 or 2 items with greater precision (Zhang & Luck, 2008). However, since individuals differ in working memory capacity (e.g., Vogel & Awh, 2008), there should be a variable rather than a constant number of slots across individuals. To address this important issue of individual differences, Anderson et al. (2011) present an analysis correlating two different, putatively independent measures of how many discrete slots observers have. In particular, they look at both the "inflection point" in observers' precision - the number of items after which each individual observer's representations seems to have stopped decreasing in precision (SD) - and also at the number of items observers seem to have information about (P(mem)) at the highest set size. They find a striking correlation between these two measures, which they interpret as evidence that each individual observer has a specific fixed number of discrete memory slots.
Unfortunately, these results are not evidence for the discrete resource model, because the analysis correlated non-independent measures - the same data contributes to both the x- and y-axes. In particular, the SD and P(mem) parameters in the model are fit simultaneously and thus trade off with each other. Consider that models are inferred based on only a limited number of samples. A model will underestimate both P(mem) and SD if some of the more inaccurate memory responses are estimated to be guess responses, whereas the converse will lead to an overestimation of P(mem) and SD. To illustrate this point, we performed several simulations and found that even in randomly generated data, if the estimated P(mem) is higher, the estimated SD tends to be higher as well. This means the SD from the highest set size will necessarily be correlated with the P(mem) from the highest set size. Concretely, if we generate random data sampled from a true SD of 18 and a true P(mem)=0.30 (mimicking Anderson et al. Experiment 1, set size 8), and sample 100 "people" with 120 trials each, we find a correlation of r=0.66 between the fit P(mem) and SD for each person. In other words, even for randomly generated data, those individuals estimated to have a larger P(mem) are also estimated to have larger SDs, even though there is no correlation between these parameters in the simulation, and even though on average the correct parameters are recovered by the model.
The non-independent analysis reported in Anderson et al. will also produce a spurious correlation between the SD inflection point and P(mem) at set size 8 since high estimates of SD at set size 8 will push the inflection point higher. In our simulations, when we sample random data for 100 "people" using the set sizes and mean SDs from Anderson et al. Exp 1 we find a correlation of r=0.50 between SD inflection and P(mem) at set size 8. Importantly, this correlation, which was taken by Anderson et al. as evidence for a variable number of discrete slots, arose from randomly generated data with no inherent variability across simulated observers.
To correct this problem, the data should be reanalyzed by estimating the inflection point in SD using set sizes 1-6 only (i.e., not using the same data that is used to estimate P(mem)). This corrects the non- independence in the analysis. However, care must still be taken when interpreting such a correlation, because lower values of P(mem) (across set sizes or across individuals who may be consistently worse in the task) result in greater noise in the estimation of SD. In some cases, depending on how the inflection in SD is measured, this can result in inflated estimates of the inflection point, thus reintroducing non-independence into the analysis.
In general, neuroscientists must be aware of the potential for non- independence between different factors in data analysis - such non- independence can occur not only from selection biases in choosing which data to analyze (Kriegeskorte, Simmons, Bellgowan & Baker, 2009), but also when fitting models to data.
References
Anderson, D.E., Vogel, E.K., & Awh, E. (2011). Precision in visual working memory reaches a plateau when individual item-limits are exceeded. Journal of Neuroscience, 31(3), 1128-1138.
Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S.F. & Baker, C.I. (2009). Circular analysis in systems neuroscience--the dangers of double dipping. Nature Neuroscience, 12(5), 535.
Vogel, E. & Awh, E. (2008). How to exploit diversity for scientific gain: Using individual differences to constrain cognitive theory. Current Directions in Psychological Science, 17(2), 171-176.
Zhang, W. & Luck, S.J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233-235.
Conflict of Interest:
None declared
Show Less

Competing Interests: None declared.

In this issue

View Full Page PDF

Article Alerts

Citation Tools

Respond to this article

Request Permissions

Responses to this article

Respond to this article

Jump to comment:

Working memory capacity continues to predict discrete limits in resource allocation after correcting for non-independence
David E. Anderson

Published on: 15 November 2011
Comparisons between different measures of working memory capacity must be made with estimates that are derived from independent data
Timothy F. Brady

Published on: 14 October 2011

Published on: (15 November 2011)

Working memory capacity continues to predict discrete limits in resource allocation after correcting for non-independence
David E. Anderson
Other Contributors:
Edward Awh
Precision in working memory declines as the number of items stored increases (Zhang and Luck, 2008). Discrete resource models predict that this decline will reach a stable plateau at relatively small set sizes, however, because fixed item limits preclude the storage of additional items. We recently confirmed this prediction by showing that mnemonic precision declined until about set size three, and then reach a stable...
Show More

Precision in working memory declines as the number of items stored increases (Zhang and Luck, 2008). Discrete resource models predict that this decline will reach a stable plateau at relatively small set sizes, however, because fixed item limits preclude the storage of additional items. We recently confirmed this prediction by showing that mnemonic precision declined until about set size three, and then reach a stable plateau for larger set sizes (Anderson, Vogel and Awh, 2011). In addition, we confirmed another clear prediction of discrete resource models, that variations in item limits across individuals should correlate with the set size at which precision reached a plateau for each observer. In line with this prediction, behavioral and neural estimates of individual item limits strongly predicted the set size at which mnemonic precision reached a plateau for each observer.
In their recent comment, however, Brady, Fougnie and Alvarez make the important point that our behavioral estimate of item limits (Pmem, for the probability of storage) was calculated using the same dataset that was used to estimate mnemonic precision (SD, for the standard deviation of the target-related part of the response distribution), and that the dependence between these measures can artificially inflate the correlation between Pmem and the inflection point of the SD by set size function. We acknowledge that this non-independence problem inflated our estimate of the correlation between the behavioral measure of capacity and SD inflection, and we thank Brady et al. for pointing out this issue. That said, we would also like to emphasize that the correlation in question was also clearly evident in our second experiment when individual item limits were estimated using only storage-related neural activity (Vogel and Machizawa, 2004). Because this neural measure of individual item limits was independent of the behavioral data used to estimate precision, the resulting correlation between item limits and SD inflection (R2 = .45) in that study is not subject to the criticism raised by Brady et al. Moreover, a re-analysis of our behavioral data shows that the correlation between Pmem and SD inflection is still robust when SD inflection is estimated while leaving out the largest set so that independent data sets contribute to each parameter estimate (R2=.38 and R2=.32 for experiments 1 and 2, respectively).
In summary, the key correlation reported by Anderson et al. (2011) is robust when issues of non-independence are fully resolved.. Furthermore, the bilinear form of the precision by set size function cannot be explained by non-independence issues, and this empirical pattern also provides clear evidence for the item limit suggested by discrete resource models (see also Zhang and Luck, 2008). In conclusion, we respectfully acknowledge the technical flaw pointed out by Brady et al., but we have also shown that our data continue to provide strong support for discrete resource models of capacity in working memory.
References:
Anderson, D.E., Vogel, E.K., & Awh, E. (2011). Precision in visual working memory reaches a stable plateau when individual item limits are exceeded. Journal of Neuroscience, 31, 1128-1138.
Vogel, E.K. & Machizawa, M.G. (2004). Neural activity predicts individual differences in working memory capacity. Nature, 428, 784-751.
Zhang W. & Luck, S.J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453, 233-235.
Conflict of Interest:
None
Show Less

Competing Interests: None declared.
Published on: (14 October 2011)

Comparisons between different measures of working memory capacity must be made with estimates that are derived from independent data
Timothy F. Brady, Postdoctoral Fellow
Other Contributors:
Daryl Fougnie, George A. Alvarez
By varying the number of orientations observers must hold in mind and examining the precision of the representations held in working memory, Anderson, Vogel and Awh (2011) show that memory precision hits a plateau at approximately 3 items, after which observers are more likely to guess but seem to have no loss in precision. This is compatible with a discrete resource model in which observers have, on average, 3 fixed reso...
Show More

By varying the number of orientations observers must hold in mind and examining the precision of the representations held in working memory, Anderson, Vogel and Awh (2011) show that memory precision hits a plateau at approximately 3 items, after which observers are more likely to guess but seem to have no loss in precision. This is compatible with a discrete resource model in which observers have, on average, 3 fixed resolution slots that can each hold one individual item or can be pooled to represent 1 or 2 items with greater precision (Zhang & Luck, 2008). However, since individuals differ in working memory capacity (e.g., Vogel & Awh, 2008), there should be a variable rather than a constant number of slots across individuals. To address this important issue of individual differences, Anderson et al. (2011) present an analysis correlating two different, putatively independent measures of how many discrete slots observers have. In particular, they look at both the "inflection point" in observers' precision - the number of items after which each individual observer's representations seems to have stopped decreasing in precision (SD) - and also at the number of items observers seem to have information about (P(mem)) at the highest set size. They find a striking correlation between these two measures, which they interpret as evidence that each individual observer has a specific fixed number of discrete memory slots.
Unfortunately, these results are not evidence for the discrete resource model, because the analysis correlated non-independent measures - the same data contributes to both the x- and y-axes. In particular, the SD and P(mem) parameters in the model are fit simultaneously and thus trade off with each other. Consider that models are inferred based on only a limited number of samples. A model will underestimate both P(mem) and SD if some of the more inaccurate memory responses are estimated to be guess responses, whereas the converse will lead to an overestimation of P(mem) and SD. To illustrate this point, we performed several simulations and found that even in randomly generated data, if the estimated P(mem) is higher, the estimated SD tends to be higher as well. This means the SD from the highest set size will necessarily be correlated with the P(mem) from the highest set size. Concretely, if we generate random data sampled from a true SD of 18 and a true P(mem)=0.30 (mimicking Anderson et al. Experiment 1, set size 8), and sample 100 "people" with 120 trials each, we find a correlation of r=0.66 between the fit P(mem) and SD for each person. In other words, even for randomly generated data, those individuals estimated to have a larger P(mem) are also estimated to have larger SDs, even though there is no correlation between these parameters in the simulation, and even though on average the correct parameters are recovered by the model.
The non-independent analysis reported in Anderson et al. will also produce a spurious correlation between the SD inflection point and P(mem) at set size 8 since high estimates of SD at set size 8 will push the inflection point higher. In our simulations, when we sample random data for 100 "people" using the set sizes and mean SDs from Anderson et al. Exp 1 we find a correlation of r=0.50 between SD inflection and P(mem) at set size 8. Importantly, this correlation, which was taken by Anderson et al. as evidence for a variable number of discrete slots, arose from randomly generated data with no inherent variability across simulated observers.
To correct this problem, the data should be reanalyzed by estimating the inflection point in SD using set sizes 1-6 only (i.e., not using the same data that is used to estimate P(mem)). This corrects the non- independence in the analysis. However, care must still be taken when interpreting such a correlation, because lower values of P(mem) (across set sizes or across individuals who may be consistently worse in the task) result in greater noise in the estimation of SD. In some cases, depending on how the inflection in SD is measured, this can result in inflated estimates of the inflection point, thus reintroducing non-independence into the analysis.
In general, neuroscientists must be aware of the potential for non- independence between different factors in data analysis - such non- independence can occur not only from selection biases in choosing which data to analyze (Kriegeskorte, Simmons, Bellgowan & Baker, 2009), but also when fitting models to data.
References
Anderson, D.E., Vogel, E.K., & Awh, E. (2011). Precision in visual working memory reaches a plateau when individual item-limits are exceeded. Journal of Neuroscience, 31(3), 1128-1138.
Kriegeskorte, N., Simmons, W.K., Bellgowan, P.S.F. & Baker, C.I. (2009). Circular analysis in systems neuroscience--the dangers of double dipping. Nature Neuroscience, 12(5), 535.
Vogel, E. & Awh, E. (2008). How to exploit diversity for scientific gain: Using individual differences to constrain cognitive theory. Current Directions in Psychological Science, 17(2), 171-176.
Zhang, W. & Luck, S.J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(7192), 233-235.
Conflict of Interest:
None declared
Show Less

Competing Interests: None declared.

Cited By...

More in this TOC Section

Show more Retracted

Umbrella menu

Main menu

User menu

Search

Precision in Visual Working Memory Reaches a Stable Plateau When Individual Item Limits Are Exceeded

Published eLetters

Guidelines

Plain text

Vertical Tabs

Jump to comment:

Conflict of Interest:

Conflict of Interest:

In this issue

Citation Manager Formats

Responses to this article

Jump to comment:

Conflict of Interest:

Conflict of Interest:

Related Articles

Cited By...

More in this TOC Section

Main menu

User menu

Search

Precision in Visual Working Memory Reaches a Stable Plateau When Individual Item Limits Are Exceeded

Published eLetters

Guidelines

Plain text

Vertical Tabs

Jump to comment:

Conflict of Interest:

Conflict of Interest:

In this issue

Citation Manager Formats

Jump to section

Responses to this article

Jump to comment:

Conflict of Interest:

Conflict of Interest:

Related Articles

Cited By...

More in this TOC Section