Abstract
Limitations of working memory (WM) capacity depend strongly on the cognitive resources that are available for maintaining WM contents in an activated state. Increasing the number of items to be maintained in WM was shown to reduce the precision of WM and to increase the variability of WM precision over time. Although WM precision was recently associated with neural codes particularly in early sensory cortex, we have so far no understanding of the neural bases underlying the variability of WM precision, and how WM precision is preserved under high load. To fill this gap, we combined human fMRI with computational modeling of behavioral performance in a delayed color-estimation WM task. Behavioral results replicate a reduction of WM precision and an increase of precision variability under high loads (5 > 3 > 1 colors). Load-dependent BOLD signals in primary visual cortex (V1) and superior intraparietal sulcus (IPS), measured during the WM task at 2–4 s after sample onset, were modulated by individual differences in load-related changes in the variability of WM precision. Although stronger load-related BOLD increase in superior IPS was related to lower increases in precision variability, thus stabilizing WM performance, the reverse was observed for V1. Finally, the detrimental effect of load on behavioral precision and precision variability was accompanied by a load-related decline in the accuracy of decoding the memory stimuli (colors) from left superior IPS. We suggest that the superior IPS may contribute to stabilizing visual WM performance by reducing the variability of memory precision in the face of higher load.
SIGNIFICANCE STATEMENT This study investigates the neural bases of capacity limitations in visual working memory by combining fMRI with cognitive modeling of behavioral performance, in human participants. It provides evidence that the superior intraparietal sulcus (IPS) is a critical brain region that influences the variability of visual working memory precision between and within individuals (Fougnie et al., 2012; van den Berg et al., 2012) under increased memory load, possibly in cooperation with perceptual systems of the occipital cortex. These findings substantially extend our understanding of the nature of capacity limitations in visual working memory and their neural bases. Our work underlines the importance of integrating cognitive modeling with univariate and multivariate methods in fMRI research, thus improving our knowledge of brain-behavior relationships.
- capacity limitations
- cognitive modeling
- cross-task classification
- fMRI
- variability of precision
- working memory precision
Introduction
Recent computational modeling studies have established that the limitation of visual working memory (WM) capacity depends both on the number of items that can be stored and the quality or precision with which they can be stored (Bays and Husain, 2008; Zhang and Luck, 2008). Further, it was suggested that performance impairments under high load may result from a decrease in WM precision as well as from an increased variability of precision (i.e., higher amount of variation in precision across trials) (Fougnie et al., 2012; van den Berg et al., 2012, 2014). However, the neural mechanisms underlying the deterioration in the quality of WM representations under high load and individual differences in this load-related performance impairment are not yet understood. Understanding the nature of these limits, however, is critical given the existence of large individual and developmental differences in WM capacity, which in turn may be important for education and life-long learning (Todd and Marois, 2005; Burnett Heyes et al., 2012; Simmering and Perone, 2012; Peich et al., 2013).
WM storage is achieved by sustained neural activity, mainly in frontal and parietal cortex (Curtis and D'Esposito, 2003), whereby parietal regions show a capacity-limited response to WM load (Todd and Marois, 2004, 2005; Xu and Chun, 2006), but also perceptual load (Mitchell and Cusack, 2008). In this context, it has been proposed that the function of frontoparietal cortex during WM relates more to attentional processes (such as directing selective attention to sensory representations), rather than storage per se (Magen et al., 2009; Riggall and Postle, 2012; Sreenivasan et al., 2014). Moreover, by using multi-voxel pattern analysis (MVPA), WM contents could be decoded from low-level sensory regions, reflecting the maintenance of high-fidelity representations mediated by the sustained activation of sensory codes (Postle, 2006; Harrison and Tong, 2009; Emrich et al., 2013; Ester et al., 2013). However, given that WM content could also be decoded from parietal (Christophel et al., 2012; Peters et al., 2015) and prefrontal regions (Lee et al., 2013), it is assumed that the exact regions representing WM contents depend on task characteristics (e.g., visual versus verbal memoranda) (Lee et al., 2013).
While this research substantially advances our knowledge of the neural mechanisms underlying WM, we have so far only limited understanding of how the capacity limitation of WM emerges from the activity of these systems. Given that recent modeling work suggests that the detrimental effect of increased task load (i.e., WM set size) on behavioral performance is mainly driven by decline in WM precision coupled with increasing precision variability, we used fMRI to investigate the neural mechanisms underlying the precision of visual WM, as well as the variability of representational precision over time, in situations of increasing set size (i.e., up to five items). We assessed whether individual differences in behavioral precision and precision variability covaried with the activation strength of cortical systems underlying visual WM (i.e., V1, V4, superior parietal areas 7A and 7PC, and superior/middle frontal gyrus). We then explored in those brain areas supporting effects of WM precision, whether the precision of neural coding (here defined as the accuracy of decoding color representations from brain activation patterns) (Emrich et al., 2013) also relates to behavioral precision and precision variability, which may provide a potential mechanistic link between brain activation patterns, WM precision, and, ultimately, the capacity limit of WM. To this end, we combined fMRI with cognitive modeling of a color WM task (variable precision model) (Fougnie et al., 2012) and identified parietal and occipital brain regions whose activation strength predicted between-person differences in the variability of precision. We then independently trained a multivariate pattern classifier to distinguish three colors during attentive perception and tested the accuracy with which the presence of these colors during WM encoding can be decoded from fMRI activity. Our results particularly highlight the left superior IPS, as it relates behavioral WM performance to both neural precision and overall BOLD activity.
Materials and Methods
Participants
Twenty-three right-handed volunteers from Goethe University Frankfurt completed the experiment for monetary compensation (12€ per hour) or student credits. All participants had normal color vision, self-reported normal or corrected-to-normal visual acuity, and no history of psychiatric or neurological diseases. Informed consent was obtained in accordance with a protocol approved by the Medical Ethics Committee of Goethe University Frankfurt. One participant was removed from the sample due to high error rates in all memory load conditions (i.e., guess rates >3 SDs from the mean in each load condition; load 1: g = 0.13, load 3: g = 0.29, load 5: g = 0.76). The final sample consisted of 22 participants (16 females, mean ± SD age, 21.77 ± 3.16 years).
Stimuli and procedure
While undergoing fMRI, the participants briefly viewed and memorized a set of colored squares and, after a delay, reported the color of one randomly chosen square using a continuous response format (Wilken and Ma, 2004). The visual delayed response WM task (for more details, see Visual delayed response WM task) was followed by a perceptual attention task (see below), in which participants had to detect small color changes of presented color stimuli. The perceptual task served to train a pattern classification algorithm to discriminate colors, based on an independent dataset (more details given below). All stimuli were generated in MATLAB (The MathWorks) using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) and MemToolbox extensions (Suchow et al., 2013) and were presented at a viewing distance of ∼64.5 cm. Stimuli were back-projected from an LCD projector onto a screen viewable by a mirror attached to the head coil of the MR scanner. In addition, participants performed the WM task outside the scanner to obtain sufficient data for behavioral model estimation of the variable precision model. In two additional behavioral experimental sessions (lasting 0.5 and 1 h) preceding the scanning session, participants performed the same visual WM task as in the MR scanner, with the exceptions of more trials per condition. To reduce possible fatigue effects in the behavioral sessions, the intertrial interval (ITI) was shortened and response time parameters were slightly adjusted (see below).
Visual delayed response WM task.
One, 3, or 5 colored squares (2 × 2° of visual angle) were arranged on 5 possible locations on an invisible circle (radius 4.5°) around a central fixation mark. Each square was pseudo-randomly allotted one of 180 isoluminant, equally spaced colors that were drawn from the CIE 1976 (L*, a*, b*) color space, centered at L = 54, a = 18, b = −8 (Fougnie et al., 2012). The minimal difference between colors that appeared at the same time on the sample display was constrained to 25 degrees to reduce the potential influence of the similarity between memory items (Kahana and Sekuler, 2002). At the beginning of each trial, a black fixation square (presented centrally on a gray background) changed to white 1.5 s before the onset of the memory sample. Participants then briefly viewed a set of 1, 3, or 5 colors (0.5 s). After a brief delay (4 s), participants reported the color of a pseudo-randomly chosen target square (i.e., the color of one of the previously presented stimuli) by scrolling through an invisible colorwheel using a trackball (van den Berg et al., 2012).
We limited the delay to 4 s because information can be held in memory with little loss in quantity or quality for this amount of time, while guessing increases rapidly after longer delays (Zhang and Luck, 2009). This short and unjittered delay also allowed to increase the number of repetitions per condition, which was crucial for behavioral modeling. Further, we did not use an overtly presented color wheel for responding (as was done in some previous studies) because this response condition, compared with a scrolling condition as used here, showed significantly higher guess rates and, in addition, evidence for nontarget responses (van den Berg et al., 2012). The probe display was presented for 6.5 s and was followed by a feedback barometer informing the participant about trial performance (0.5 s). A 7 s ITI followed the feedback at the end of each trial and comprised a gray display with a central black fixation mark (Fig. 1A). In the behavioral experiment, the ITI was shortened (jittered ITI: 1, 1.5, or 2 s), and participants could immediately start the next trial after their response, to reduce possible fatigue effects (maximum duration of probe display was 6 s).
Experimental fMRI paradigms. A, Schematic illustration of continuous response visual WM task (example showing set size = 5). The color delayed estimation WM task (adapted from Wilken and Ma, 2004 and van den Berg et al., 2012) involved loads 1, 3, and 5 and required that subjects use an MR-compatible trackball to reproduce the color of the indicated stimulus during the retrieval phase. B, The color attention task was measured after the WM task. Participants attended to color stimuli presented in isolation, to detect a small color change (25°) that appeared in 4 of 40 trials per session. The color attention task was used for training a classifier algorithm to differentiate three different colors (“fixed colors”) based on their brain activation patterns. This classifier was then applied to the WM task, to decode the presence of these colors during an early period of the WM task (i.e., at 2–4 s after memory array onset). In 9 of 18 WM trials (per run), the memory sample contained one of the three fixed colors. The presentation of load (1, 3, or 5) and fixed color (present/absent) was implemented in the WM task in a full-factorial design.
The MRI experiment included 4 runs in total, each consisting of 18 trials with a duration of 20 s (180 volumes per run). In 9 of 18 trials per run, the memory sample contained 1 of 3 “fixed colors” that differed 120 degrees from each other in the CIE 1976 (L = 54, a = 18, b = −8) color space. The fixed colors were red, green, and violet (RGB: 241, 57, 73; 102, 140, 71; 140, 117, 190, respectively). The presentation of different memory loads (1, 3, or 5 items) and fixed color (present/absent) were implemented in a full-factorial design. Hence, each fixed color appeared equally often under the different load conditions. The full-factorized combinations were thus presented in a pseudo-random fashion. The three fixed colors (which never appeared together in one trial) were used to test a classifier that was trained on fMRI data from a second, independent experiment in which participants performed a target detection task that only involved these three colors (see Color attention task). Thus, half of the WM trials in the fMRI session (i.e., 36 trials) were used for classifier testing. Behavioral pretesting had indicated that the repeated presentation (during WM stimulus presentation) of the same colors among randomly chosen samples of colors did not systematically bias behavioral error rates due to possible implicit learning processes in the course of the WM task (4 fixed colors; fixed colors appearing in 40% of 180 trials; memory load 1, 3, or 5; N = 35; absolute errors between presented and reported colors for targets with fixed vs other colors across memory load, F(df = 4,52) = 1.29; p = 0.29, not significant). Postexperiment interviews indicated that participants were indeed not explicitly aware of the increased probability of the fixed colors, and behavioral performance was not improved for trials with fixed colors (see Behavioral results). Fixed colors were only presented during the fMRI session, not during the behavioral sessions.
Color attention task.
The color attention task served to provide a training dataset for MVPA classifier analyses of those three colors that were presented during the WM task with slightly increased probability (“fixed colors”; see above). One colored square at a time was presented, drawn from the three fixed color values. In addition, trials with gray squares (RGB: 132, 132, 132) were presented but not used for classifier training. The four different items were presented equally often (i.e., 10 trials per item, from which 1 trial per item was used as probe trial, but not for classifier training) in a pseudo-random fashion, whereby the locations of the colored squares were also pseudo-randomly selected from the five positions used in the WM task. Based on previous results suggesting that sensory information of different feature dimensions (i.e., color and location) do not compete and can be stored in parallel (Wheeler and Treisman, 2002; Baddeley et al., 2011; Brady et al., 2011), we intentionally designed this task to train a classifier to differentiate colors independent of the specific location of stimulus appearance.
Each trial began with the presentation of a randomly chosen color on a gray background with a white fixation mark at the center. A jittered uniformly distributed ITI (3, 5, or 7 s), where participants viewed a white fixation mark on a gray display, followed. Participants had to attend to the presented colors and were instructed to press the left mouse button of an MR-compatible trackball if a small color change (25°; 4 of 40 trials per run) of the presented square occurred (Fig. 1B). In total, participants completed 4 runs of the perceptual attention task (120 volumes per run). The training of the classifier was based on 36 trials for each fixed color of the perceptual attention task because trials designed for probing performance were excluded, so that 4 runs × 9 trials = 36 trials were available per fixed color.
Behavioral data analysis
Behavioral data analysis of the WM task was performed using the Statistics toolbox and the MemToolbox (Suchow et al., 2013) for MATLAB (The MathWorks). Because several previous studies found that mnemonic precision is continuous and variable (Wilken and Ma, 2004; Bays and Husain, 2008; Zhang and Luck, 2008; Fougnie et al., 2012; van den Berg et al., 2012, 2014), rather than discrete and equal across trials (Luck and Vogel, 1997), the variable precision model was fit to each subject's data (Fougnie et al., 2012; van den Berg et al., 2012), an expansion of the mixture model by Zhang and Luck (2008) that additionally takes varying instead of fixed mnemonic precision between trials into account. As described above, we used a task that enabled a direct measure of mnemonic precision along a continuous color dimension requiring subjects to precisely remember the colors of 1, 3, or 5 colored squares (Wilken and Ma, 2004; van den Berg et al., 2012). Many previous neurophysiological studies used change detection tasks, including discrete measures of WM performance (i.e., target present vs absent in WM), which gave less information about the quality of WM representation (for review, see Brady et al., 2011).
The difference between presented and reported color, a continuous measure on the basis of the 360 degree color wheel in each trial (i.e., the “error”) was used for model estimation. In the classical (i.e., fixed precision model), these errors in recall are assumed to be distributed according to a von Mises distribution (the circular analog of a Gaussian distribution) centered at zero and with the free precision parameter κ (which is the inverse width of the distribution of errors). In this model, the precision of WM representations corresponds to the SD of the von Mises distribution (Zhang and Luck, 2008). Variable precision models, in contrast, allow precision to vary from trial to trial; they can be distinguished in respect to how this trial-to-trial variation of the κ parameter is modeled (i.e., truncated normal or gamma distribution) and in the extent to which they allow for estimating guessing behavior or not (Fougnie et al., 2012; van den Berg et al., 2012, 2014). In this study, trial-to-trial variation of the κ parameter was modeled by a higher-order truncated normal distribution with two parameters (mean [μ] and SD [σ]), which we will refer to as the mean of precision (mean SD) and the variability of precision across trials (SD variability), respectively. Higher values in mean SD signify less precise memory representations, whereas higher values in SD variability reflect a higher trial-to-trial variation in memory precision. The model also implements the possibility of guessing behavior (in the form of a guess rate parameter), where viewers either remember the item with probability 1 − guess rate, or they remember nothing about the item at recall and guess randomly, resulting in uniformly distributed errors (Fougnie et al., 2012). Thus, the probability density function of an “error” x (in radians) is as follows:
where the probability density function, ψ, is given by integrating the von Mises distribution (VM) with mean 0 and precision κ over the higher order density of κ as follows:
whereas Φ(s; μ, σ, a, b) denotes a truncated normal distribution with mean SD (μ) and SD variability (σ) bound within a and b, and κ(s) denotes the transformation of the width parameter s into precision κ (Jammalamadaka and Sengupta, 2001). Following Fougnie et al. (2012), we chose to bound the width parameter s within a = 0 (0°) and b = 1.75 (100°). The three critical parameters (i.e., mean SD, SD variability, and guess rate) were estimated separately for each load condition in each participant using routines of the MemToolbox, including the Metropolis-Hasting algorithm, a Markov Chain Monte Carlo technique that samples from the posterior density of the parameter space, and noninformative Jeffreys priors, which were placed on all free parameters (Fougnie et al., 2012; Suchow et al., 2013).
As a first step, we compared model fits from this specific variant of the variable precision model (i.e., VP-Gaussian-SD-g) with three other variants of the variable precision model using each model's Akaike Information Criterion (AIC; i.e., the relative goodness of fit corrected for the number of free parameters) (e.g., Suchow et al., 2013). As alternative models, we included the following: (1) a VP model with a truncated normal over precision but with no guess rate parameter (i.e., VP-Gaussian-SD-g-zero); (2) a model that used a gamma distribution as higher-order distribution and estimated guessing behavior (i.e., VP-Gamma-SD-g), and the same model but without guess rate parameter (VP-Gamma-SD-g-zero) (Fougnie et al., 2012; Suchow et al., 2013; van den Berg et al., 2014). On average, across participants and load conditions, the AIC of the VP-Gaussian-SD-g was 1494.24, which was slightly lower than the AICs of the other three model variants (VP-Gamma-SD-g: AIC = 1494.32; VP-Gaussian-SD-g-zero: AIC = 1497.24; VP-Gamma-SD-g-zero: AIC = 1498.45; lower AIC values correspond to better fit). However, pairwise comparisons of AIC differences for each model within each load condition, using t tests, showed no significant differences in 16 of 18 comparisons (all p values >0.05, corrected for multiple comparison). Based on these results, we conclude that model fits from other variants of the variable precision model were statistically not distinguishable from the VP-Gaussian-SD-g model; we arbitrarily chose the VP-Gaussian-SD-g model.
In total, we used 477 trials (159 trials per load condition) per subject based on the sum of trials from the behavioral and scanning sessions. Most specifically, we had 405 trials (135 trials per load condition) from the behavioral sessions plus 72 trials from the scanning session (24 trials per load condition). To evaluate how performance changed as a function of memory load, we conducted a one-way repeated-measures ANOVA for each behavioral parameter (i.e., precision mean, precision variability, or guess rate) with the factor memory load that contained three conditions (levels): load 1, load 3, and load 5. Assumptions of equal variances of the differences between levels were checked using Mauchly's test for sphericity. Bonferroni correction was applied to adjust for multiple comparisons of main effects. As well as an assessment of the within-subjects effects of WM load (i.e., variation between the different levels 1, 3, and 5 compared with the error term) on mean SD, SD variability, and guess rate, we tested for linear and quadratic trends of the relationship between the levels of load. In the case of significant quadratic trends, individual slopes from load 1 to load 3 and from load 3 to load 5 were computed and analyzed separately, to account for nonlinearity in the rate of change in behavioral performance.
fMRI data acquisition
Functional BOLD MR images were acquired on a 3-T Siemens Magnetom Allegra Scanner using a T2*-weighted BOLD-sensitive gradient-echo, EPI sequence (TR = 2000 ms; TE = 30 ms) with 32 slices (thickness = 3 mm, gap = 1 mm, descending order, FOV = 192 mm, flip angle = 90°). In each run, we collected either 185 (WM task) or 125 (perceptual attention task) functional images, with a total of 4 runs per task. The first 5 volumes of each run were discarded to allow for equilibrium magnetization. After the functional data, structural MR images were recorded using a T1-weighted MP-RAGE sequence (TR = 2200 ms, TE = 3.93 ms, TI = 900 ms, slice thickness = 1 mm, gap = 0.5 mm, voxel size = 1 × 1 × 1 mm3, FOV = 256, whole brain coverage, flip angle = 9°).
fMRI data analysis
fMRI data were analyzed using MATLAB (The MathWorks) and the Statistical Parametric Mapping Software (SPM 8; Wellcome Trust Centre for Neuroimaging). The preprocessing of the EPI images included slice-time correction, spatial realignment (motion correction), and normalization into the standard MNI 152 space. Data were smoothed using a 9 mm FWHM Gaussian Kernel. For multivariate analysis (MVPA), data preprocessing was limited to spatial realignment; to preserve the fine-grained spatial pattern of fMRI data, neither slice-time correction, nor normalization, nor spatial smoothing was applied to the functional images (Norman et al., 2006).
The goal of the fMRI analysis was to assess the neural substrate of the individual deterioration of mnemonic representations with increasing set size (i.e., the neural mechanisms that are related to the decline in mnemonic precision) and the increase in precision variability and guess rate with load. For this aim, we defined five ROIs for each hemisphere and tested for relationships between the slopes of load effects on behavioral performance (i.e., mean SD, SD variability, and guess rate) and load-related univariate BOLD signal estimates from these regions. Further, we used MVPA to explore the relationship between load effects on behavioral precision and decoding accuracy (which is assumed to represent the precision of the neural representation) (e.g., Emrich et al., 2013) from brain regions that were identified as playing a crucial role for load-dependent behavioral performance deterioration during univariate analyses. The analysis of functional data relied on ROIs, so that results are best comparable between univariate and multivariate analyses.
ROI definition.
Anatomical ROIs (see Fig. 2A) from 3D probabilistic cytoarchitectonic maps were created using the Anatomy Toolbox (Eickhoff et al., 2005) and wfu pickatlas (Maldjian et al., 2003) implemented in SPM 8. ROI definition was informed by previous findings from univariate and multivariate fMRI analyses, which showed WM encoding and delay related load effects in occipital (visual), parietal, and frontal regions and which had established that WM contents could be decoded from these areas (Todd and Marois, 2004, 2005; Xu and Chun, 2006; Mitchell and Cusack, 2008; Magen et al., 2009; Xu, 2009; Emrich et al., 2013; Lee et al., 2013; Sprague et al., 2014). Specifically, the ROIs included areas “V1” and “V4,” “7A” and “7PC,” and “superior/middle frontal gyrus (FG),” for each hemisphere (compare Fig. 2A). Areas V1, V4, 7A, and 7PC were derived from the Anatomy Toolbox, a probabilistic atlas of brain regions based on a combination of functional imaging results and probabilistic cytoarchitectonic maps obtained from the analysis of 10 postmortem brains (Eickhoff et al., 2005). Probabilistic area V1 corresponds to primary visual cortex. Most specifically, this probabilistic map covered the upper and lower bank of the calcarine sulcus extending to the mesial surface of the cuneus (Amunts et al., 2000). Probabilistic area V4 included the ventral extrastriate cortex lateral to BA 18; in particular, the V4 mask was located in the collateral sulcus covering the fusiform gyrus (Rottschy et al., 2007). Brouwer and Heeger (2009) demonstrated empirically that stimulus color can be decoded from these two visual regions. Masks for the parietal areas 7A and 7PC represent two adjacent but anatomically distinguishable areas within the superior parietal cortex (Scheperjans et al., 2008). These two probabilistic ROI maps overlap with superior portions of the IPS that proved relevant for individual differences in visual WM in earlier studies (e.g., Todd and Marois, 2004, 2005; Xu and Chun, 2006). The ROI masks for the superior/middle frontal gyrus are located in posterior portions of the frontal lobe and were chosen so as to plausibly include the putative human analog of the frontal eye fields (the exact description of the anatomical localization of the human analog of the frontal eye fields differs between studies) (see also Paus, 1996; Blanke et al., 2000; Magen et al., 2009; Akaishi et al., 2013; for review, see Vernet et al., 2014). Frontal ROI masks encompassed dorsal portions of the precentral, posterior and mid-portions of the superior frontal gyrus, and middle frontal gyrus. The superior/middle frontal gyrus ROIs were constructed using the wfu pickatlas toolbox and according to a procedure described by Maldjian et al. (2003); they consist of joined Brodmann areas 6 and 8 from the Brodman area atlas (generated in MNI space based on Talairach Daemon) intersected by the middle frontal gyrus mask from the Talairach Daemon label atlas (Lancaster et al., 2000; Maldjian et al., 2003). To warp MNI space ROIs into subjects' native space (for MVPA), we used the individual segmentation parameters derived during preprocessing to perform a reversed normalization of the defined ROIs.
WM load-related ROIs and their load-dependent activation patterns. A, Anatomical ROIs. Bilateral human primary visual cortex (probabilistic area V1; blue) (Amunts et al., 2000), human ventral visual area 4 (probabilistic area V4; cyan) (Rottschy et al., 2007), and two adjacent but anatomically distinct regions of superior IPS, probabilistic areas 7A (red) and 7PC (orange) (Scheperjans et al., 2008) were defined from 3D probabilistic cytoarchitectonic maps using the Anatomy Toolbox (Eickhoff et al., 2005). ROIs also included a frontal region (superior/middle frontal gyrus; green) designed as such to encompass also the human homolog of the frontal eye fields (Maldjian et al., 2003). For details concerning ROI definition, see Materials and Methods. B, From each ROI, BOLD signal contrast estimates under loads 1, 3, and 5 (against baseline) from time bin 2 (i.e., 2–4 s after sample onset) were extracted using the Marsbar toolbox (http://marsbar.sourceforge.net/) for MATLAB (The MathWorks) to illustrate set size effects on BOLD signal activity. Color-coded dots connected by lines indicate averaged contrast estimates across participants (N = 22); SE for within-subjects effects was computed according to Cousineau (2005). Blue represents V1. Cyan represents V4. Red represents 7A. Orange represents 7PC. Green represents superior/middle frontal gyrus/FG.
As a validation of our definition of anatomical ROIs, when analyzing data across the entire brain in a voxelwise fashion, we observed task-related univariate activity in a broad network, encompassing clusters bilaterally in the primary and extrastriate visual cortex, including areas covered by our V1 and V4 masks, IPS, superior parietal lobule, precentral gyrus, and middle frontal gyrus. Activity in these clusters changed parametrically with the number of items that were encoded, revealed by a second-level t test of the parametric load estimates during an early trial phase of the WM task (i.e., 2–4 s after visual onset; time bin 2) (compare Table 1; T = 2.83, p < 0.05, corrected), thus largely overlapping with our ROIs and also replicating earlier research on visual WM (Pessoa et al., 2002; Curtis and D'Esposito, 2003; Mitchell and Cusack, 2008; Magen et al., 2009; Fusser et al., 2011; Todd et al., 2011; Rottschy et al., 2012).
Univariate resultsa
Univariate analysis.
We modeled the sequence of events of the WM trial with a finite impulse response approach with 12 time points in intervals of 2 s (corresponding to the TR) due to the temporal proximity of WM events. Our analysis thus encompassed the time window from 0 to 24 s after event onset (i.e., after the onset of the encoding phase). The design matrix also contained movement parameters and a constant term for each run. We estimated responses to the memory load manipulation in terms of a parametric modulation of the WM trial predictor representing memory load (pmod regressor, load 5 > 3 > 1). Additionally, in separate models, separate regressors for each of the three levels of memory load were estimated to further assess contrasts between load 1 and load 3, and load 3 and load 5 given the behavioral finding of nonlinear shapes of load effects on the guess rate and mean SD parameters. Thus, for each participant, the following three contrasts were computed with separate t tests by using regressors at time bin 2 after visual cue onset: pmod contrast 5 > 3 > 1, load 3 > load 1, and load 5 > load 3. For each ROI, we extracted these contrast values of the BOLD signal using the Marsbar toolbox (http://marsbar.sourceforge.net/) for MATLAB (The MathWorks). Figure 2B represents extracted contrast values in each load condition for each anatomical region, averaged across participants. Within each participant, individual contrast estimates of the load effect for each ROI were normalized using the contrast estimates across ROIs as variables for mean-centering (i.e., contrast estimate of the respective ROI minus averaged contrast estimate across all ROIs), which ensured higher comparability of brain activity between ROIs within subjects and also reduced between-subject variance. Individual contrast values, most specifically their slopes, were integrated at the group level as dependent variables of a general linear model (GLM) with factors ROI (5 levels: V1, V4, 7A, 7PC, superior/middle frontal gyrus), hemisphere (2 levels: left, right), and behavioral model parameter (i.e., mean SD, SD variability, or guess rate) as covariates. This model corresponded to a two-way repeated-measures ANCOVA and allowed us to directly test interactions between ROI, hemisphere, and the respective behavioral performance estimate on the slope of the load-related BOLD signal changes (load 5 > 3 > 1). To further resolve significant interactions, we conducted post hoc correlation analyses between behavioral performance estimates and normalized contrast estimates of BOLD load effects. We applied a significance level of 5% and Bonferroni correction for multiple comparisons. Thus, by considering three GLMs (load 5 > 3 > 1; 5 > 3; 3 > 1), the corrected threshold was p = 0.016. Assumptions of equal variances of the differences between levels were checked using Mauchly's test for sphericity. Greenhouse–Geisser correction was applied when violation of sphericity between groups occurred.
Multivariate pattern classification.
Multivariate analysis was accomplished using a classification and regression tree (CART), a nonparametric binary decision tree learning technique that produces discrete classification or continuous regression trees (Breiman, 2001). CART was found to effectively deal with high-dimensional datasets and provides a well-suitable method for the evaluation of multifactorial brain-behavior relationships (Godefroy et al., 1998; Neeley et al., 2007; Duchesnay et al., 2011). The CART algorithm was chosen because it can naturally handle multiclass problems (i.e., here: decoding three fixed colors from brain activity), whereas for other approaches such as for the widely used Support Vector Machine algorithm, finding an efficient extension to multiclass problems is not straightforward and still heavily debated within the field (Madzarov et al., 2009). We used the prediction of classification trees, in which the estimated outcome variable was the class (i.e., here: fixed color) to which the data of the multivariate brain activity belonged. We conducted a cross-task classification analysis in which we trained the nonparametric CART classifier on the trial-by-trial β estimates of the three fixed colors (see above) obtained from the color attention task, and tested the estimated information on the second time window (i.e., 2–4 s = 1 TR = 2 s) of the finite impulse response model of the WM trial.
Color information was decoded from those ROIs that were identified in the univariate analysis as covarying significantly with the load-related slope of one or more of the parameters of the variable precision model (i.e., left and right probabilistic areas V1 and 7A; compare Univariate results). In the first step of the multivariate analysis, we estimated parameters of a multivariate predictive model (based on the CART algorithm) for each participant, each load, and each ROI separately and thus obtained 22 (N) × 3 (load conditions) × 4 (ROIs) = 264 decoding accuracies in total. For each participant, in each cross-classification fold, all data (i.e., 2D matrix: number of trials × number of voxel of respective ROI) was transformed to a lower dimensional representation using principal components analysis as this reduces the dimensionality of the problem from the number of betas to the number of samples (for details of this dimensionality reduction approach, see Mourão-Miranda et al., 2005). Thus, for each ROI, the dimensionality of the feature space reduced to the number of samples/trials during classifier training (here: 3 × 36 training trials). Also, for each ROI, we applied feature selection to extract β estimates from those 10% voxels that showed highest predictive power within each cross-classification fold. In a second step, and before the assessment of brain-behavior relationships, we tested whether the obtained classifier accuracies were significantly different from chance level (i.e., 0.33) at the group level, separately for the tested ROIs and load conditions, using a Wilcoxon signed-rank test. A significance level of 5% was applied with a modified Bonferroni correction (i.e., number of tests per ROI = 3, df = 2: pcorrected = 0.033) (Keppel, 1991) to protect against false-positives.
The cross-task classification approach enabled the generalization from perceptual representations of colors to mnemonic representations of these colors in the WM task, thereby counteracting the risk of possible “double-dipping” (Kriegeskorte et al., 2009). Because both the perceptual attention task and the WM task are slow-event-related designs in which the correlation between trial-specific regressors is assumed to be low, we used a GLM to obtain trial-specific estimates, in which each trial of training and testing task was modeled as separate regressor (Mumford et al., 2012). The β estimates of the GLM of the perceptual attention task reflected the responses to each trial of each of the three fixed colors. For the visual delayed response task, a novel GLM was set up for the decoding analysis. We used again an finite impulse response design (window length = 24, order = 12) that consisted of regressors modeling set size (1, 3, or 5) and the presence versus absence of a fixed color within the memory sample. Therefore, this GLM of the WM task encompassed 9 different combinations of load and fixed color as conditions (i.e., load1_violet, load3_violet, load5_violet, load1_green, load3_green, etc.) within each session. The remaining 9 trials (i.e., those trials without fixed colors and thus not entering the decoding analysis) were modeled as regressors of no interest, only including information about memory load. It should be noted that data are often mean-centered before classification. For a decision tree (i.e., CART), however, mean-centering does not affect classification as node-specific rules are found for each variable separately (Breiman, 2001), which is why there was no need for mean-centering.
The multivariate data analysis served to explore whether or not load-related decreases in behavioral precision can be accounted for by decreases in classification accuracy of decoding fixed colors from WM, which we thus use as an indicator of the precision of neural representations during the WM task (compare Emrich et al., 2013). Only those ROIs showing significant above-chance accuracies in the decoding analysis in at least one of the three load conditions were considered for subsequent ANCOVA to evaluate the relationship between decoding accuracy and parameters of the variable precision model (i.e., mean SD, SD variability, and guess rate). To this end, we conducted within-subject correlations using the method of Bland and Altman (1995), by calculating ANCOVA with the factors behavioral performance (i.e., mean SD under loads 1, 3, and 5) and subjects as the covariates, and decoding accuracy (i.e., classifier accuracies under loads 1, 3, and 5) as the dependent measure (Bland and Altman, 1995; for a similar application of this approach, see Emrich et al., 2013). The magnitude of the correlation coefficient ρ of the variation in classifier accuracy due to variations in WM performance can be expressed as the square root of the proportion of the “sum of squares for the behavioral performance estimate” (e.g., mean SD) divided by the “sum of the sum of squares for behavioral parameter and the residual sum of squares.” The sign of the correlation coefficient ρ is given by the estimated slope parameter (regression coefficient) (Bland and Altman, 1995). In addition, we conducted the same analysis in a control region (i.e., left and right primary auditory cortex; probabilistic area TE12 from the SPM Anatomy Toolbox) (Morosan et al., 2001), as we would not expect any effects of visual load in this region. A significance level of 5% was applied throughout the ANCOVA.
Results
Behavioral results
Using the variable precision model (Fougnie et al., 2012; van den Berg et al., 2012), we estimated mean SD, SD variability, and guess rate for each subject and load condition. These three parameters reflect the mean of mnemonic precision of color representations (mean SD), its variability (SD variability), and the proportion of guessing across trials, respectively. On average, the mean SD across subjects was 14.99° (SD = 2.81°), 20.25° (SD = 3.43°), and 29.91° (SD = 6.79°) for loads 1, 3, and 5, respectively, reflecting a decline in mnemonic precision as load on WM increases (see distribution means on x-axis in Fig. 3A). Further, the average SD variability across subjects was 4.3° (SD = 2.95°), 6.98° (SD = 4.00°), and 13.88° (SD = 11.74°) (compare distribution widths in Fig. 3A), which suggests that the amount of variability of memory precision increased with load. Repeated-measures ANOVAs demonstrated a significant increase in mean SD (F(df = 1.4,28.4) = 96.03, p < 0.001; Fig. 3B, left) and SD variability (F(df = 1.4,28.4) = 10.94, p = 0.001; Fig. 3B, middle) with increasing load. We furthermore observed a significant increase in guess rate with increasing load (F(df = 1.2,24.8) = 70.45, p < 0.001; Fig. 3B, right). Furthermore, for each parameter, we observed a significant linear trend across memory load (mean SD: F(df = 1,21) = 138.75, p < 0.001; SD variability: F(df = 1,21) = 14.58, p = 0.001; guess rate: F(df = 1,21) = 75.12, p < 0.001). For the mean precision and the guess rate parameter, we additionally observed a significant quadratic trend (mean SD: F(df = 1,21) = 8.29, p = 0.009; guess rate: F(df = 1,21) = 54.4, p < 0.001), but not for SD variability (F(df = 1,21) = 2.25, p = 0.15). Therefore, we additionally computed separate slopes across load 1 and load 3, and load 3 and load 5 for the mean SD and guess rate parameter to account for the nonlinear relation when relating behavioral data to neural data.
Results of model estimation of behavioral data. The difference between presented and reported color (“error”) in each trial was used to estimate three critical model parameters: the mnemonic precision (mean SD), the variability of precision (SD variability), and the guess rate, separately for each load condition (1, 3, and 5) and participant (N = 22) by using the variable precision model (Fougnie et al., 2012; van den Berg et al., 2012). A, Average mean SD and SD variability (i.e., as reflected by across participants averaged mean and SD of the higher-order truncated normal distribution indicated by vertical and horizontal shaded lines, respectively, on the right panels) across participants in loads 1, 3, and 5, presented as probability density functions of the SD in degrees. B, Parameter estimates from each participant are plotted for each load condition (left: mnemonic precision/mean SD; middle: precision variability/SD variability; right: guess rate). Each subject's data points across loads 1, 3, and 5 are plotted by either dots or squares in a different color. Gray lines indicate the individual linear regression slopes. To increase the clarity of visual display, for each load condition, individual data points were slightly jittered along the x-axis and load conditions were separated by vertical dashed lines.
Reliability of behavioral parameter estimates across sessions
Before incorporating the results of behavioral modeling with the BOLD signal estimates, we validated whether the behavioral data measured in the scanner (i.e., the distribution of errors) match behavior outside the MR scanner. Here, we compared the two distributions of errors (i.e., inside vs outside the scanner) using nonparametric two-sample Kolmogorov–Smirnov tests, separately for each load condition. Within each load condition, we observed no significant differences between these two distributions (load 1: p = 0.134, load 3: p = 0.616, load 5: p = 0.713), suggesting that the behavioral data were comparable between the sessions inside and outside the MR scanner. Furthermore, we fitted the variable precision model to each subject's data that were acquired outside the scanner (i.e., 135 trials per load condition) and compared parameter estimates with the current estimates from all trials (i.e., 159 trials per load condition; trials per load: 135 + 24 trials acquired during fMRI). Here, trial numbers can be considered, by and large, comparable and should not have a strong influence on the accuracy of parameter estmation. Correlational analyses between parameters based on all trials and those based only on outside-scanner sessions revealed strong correlations for guess rates (load 1: r = 0.973, load 3: r = 0.979, load 5: r = 0.954, all p < 0.001), mean SD (r = 0.988, r = 0.943, r = 0.843; all p < 0.001), and SD variability parameters (r = 0.944, r = 0.911, r = 0.896; all p < 0.001). We additionally computed the intraclass correlation coefficient with the factor session (all vs outside) using the model parameters as dependent variable. Across participants, results demonstrated highly significant intraclass correlation coefficients in each load condition for each parameter (all p values < 0.001), suggesting high consistency between parameter estimates based on data measured inside versus outside of the scanner.
Control of effects of fixed color presentation
To exclude possible learning effects for the fixed colors (i.e., those three colors that appeared with a slightly increased probability during the fMRI session; compare Materials and Methods), we tested for possible differences between trials in which a fixed color was presented during memory encoding compared with trials in which no fixed color occurred, collapsed across loads to achieve sufficient trial numbers. Here we used the distribution of absolute errors as trial numbers were too small for parameter estimation after data splitting. Similar to the results of our pretesting, we observed no significant difference in error distributions (F(df = 1,21) = 2.89, p = 0.1), which indicates that no learning of the fixed colors occurred.
fMRI results of univariate analysis
To assess the relationship between the degree of load-dependent deterioration of WM performance and the recruitment of task-related brain systems, we entered the linear slopes of WM parameters (i.e., mean SD, SD variability, or guess rate) as individual differences covariates into group-level random effects models with the contrast estimates of the load effect on BOLD signal (equivalent to the slope of the neural load effect) as dependent variable. With ANCOVA models, we tested for possible interactions between these behavioral parameters and load-related activity increase in five a priori defined anatomical ROIs (i.e., probabilistic visual areas V1 and V4, probabilistic superior parietal regions 7A and 7PC, and superior/middle frontal gyrus; for more details, see Fig. 2A; Materials and Methods), separately for each hemisphere. Results revealed a significant interaction between the factor ROI and the variability of precision (most specifically the slope of the load 5 > 3 > 1 effect for the SD variability parameter; F(df = 2.4,47.8) = 5.62, p = 0.004), but no interaction with mean SD (p = 0.139) or the guess rate parameter (p = 0.375). No other interactions between ROI and/or hemisphere and respective model parameters were observed (all p values >0.176).
To evaluate which brain regions were in particular sensitive to load-related changes of precision variability, post hoc correlation analyses demonstrated that participants with more stable WM performance in the face of high task load (i.e., participants with lower load-dependent increase of the variability parameter − less steep slopes of SD variability) showed a stronger task-related activation increase within left and right area 7A of superior parietal cortex, indicated by a significant negative correlation between SD variability slope and the load effect in this area (left 7A: r = −0.539, p = 0.009; right 7A: r = −0.474, p = 0.026; across left and right: r = −0.55, p = 0.008; compare Fig. 4A, first row). In contrast, within left and right visual area V1, we observed a different pattern (i.e., individuals with more variable WM representations [higher load-related increase in SD variability] showed a stronger BOLD-signal increase: left V1: r = 0.592, p = 0.004; right V1: r = 0.675, p = 0.001; across left and right: r = 0.65 p = 0.0009; compare Fig. 4A, second row). We observed no significant correlation between SD variability slope and any of the other ROIs (all p values >0.192). Figure 4A also indicates a significantly positive relationship between the load-dependent slopes of mean SD and BOLD activity in visual area V1. Collapsed across left and right V1, we see a trend toward greater V1 activity increase in persons that showed greater reductions of precision under increasing load (i.e., greater increase in mean SD: r = 0.44, p = 0.04; Fig. 4A, second row, left). (However, this result does not pass the Bonferroni-corrected threshold of significance).
Results of univariate and multivariate analysis. A, Variability of precision modulates load-dependent activity in superior IPS (probabilistic area 7A; first row, in red) and probabilistic area V1 (second row, in blue). While area 7A shows a negative correlation between subject-specific slopes of SD variability (load 5 > 3>1; x-axis; middle panel) and parametric load-modulated BOLD signal (i.e., averaged load-dependent contrast estimates across left and right ROIs; y-axis), indicating that this area is more strongly active in persons with less variable WM performance, V1 shows a positive correlation with SD variability increase. B, Mean classification accuracies from left and right superior IPS (area 7A; red bars) and visual area V1 (blue bars) across participants (N = 22) (top). Error bars indicate SE for within-subject effects (Cousineau, 2005). Bottom, The lower decoding accuracy from left area 7A (y-axis) was associated with higher mean SD (i.e., lower behavioral precision) and with higher SD variability (x-axes) with increasing set size (load 5 > 3 > 1). Parallel lines indicate within-subject correlations. Each participant is displayed using a different combination of line color and line type (Bland and Altman, 1995; compare Emrich et al., 2013). *p < 0.05. Right, Anatomical localization of probabilistic area 7A (red) in relation to the IPS, postcentral sulcus (PoCS), central sulcus (CS), and Sylvian fissure (SF) (adapted from Scheperjans et al., 2008).
To evaluate whether the nonlinear load effects of behavioral performance estimates (i.e., mean SD and guess rate; see Behavioral results) may additionally contribute to the association between brain regions and behavioral parameters, we conducted two additional group-level GLMs with BOLD signal contrast estimates for “load 3 > 1” and “load 5 > 3” as dependent variables, respectively, for these two parameters. Under higher set size (i.e., load 5 > load 3), there was a trend toward a guessing × ROI interaction (p = 0.047), which however did not pass the Bonferroni-corrected significance threshold of p = 0.016. No other interactions between ROI and/or hemisphere and respective model parameters were observed (all p > 0.117).
To summarize, univariate BOLD results revealed that the linear increase in the variability of visual WM precision under increasing load (i.e., load 5 > 3 > 1) was significantly related to load-modulated BOLD activity in superior IPS (i.e., probabilistic area 7A), suggesting that superior parietal involvement helps to stabilize WM representations in the face of higher load. In addition, individuals with less stable representations (i.e., stronger load-related increase of SD variability with load) showed a stronger load-dependent recruitment of probabilistic visual area V1.
fMRI results of multivariate analysis
We had hypothesized that the precision of the neural representation of WM stimuli may mediate the association between behavioral performance parameters and brain activation. In this context, the precision of neural coding is quantified as the accuracy of decoding stimulus features during the WM task from brain activation patterns (Ester et al., 2013), as was previously proposed, for example, by Emrich et al. (2013). Thus, to evaluate how load-dependent deterioration in WM performance is related to the precision with which visual WM stimuli are represented at the neural level, we first identified those ROIs showing classifier accuracies that were significantly different from chance level, in at least one of the three load conditions, and then in a second step examined the relationship between behavioral parameters of the variable precision model and load-dependent changes in decoding accuracies. To this end, a cross-task classification was implemented for which we trained a classifier to discriminate between neural activation patterns associated with the perception of three colors based on the fMRI data from an independent color attention task (see Materials and Methods). These “fixed” colors were included into the WM study with slightly increased frequencies of occurrence (see Materials and Methods), and the trained classifier was applied to the WM task to decode which fixed color (of three different fixed colors) was encoded into WM during the current trial that included the presentation of a fixed color within the memory array.
The cross-task classification was conducted for left and right probabilistic areas 7A and V1 (i.e., for the four ROIs in which we observed significant correlations between precision variability and univariate BOLD activity). Results demonstrated that classification results from left area 7A were significantly above chance for load 1 (p = 0.002) and load 3 (p = 0.03), but not load 5 (p = 0.47) (compare Fig. 4B, top). In addition, we observed a trend for above-chance accuracies also for right area 7A at load 1 (p = 0.09), but none of the other tested combinations of ROI and load condition showed classifier magnitudes that were significantly different from chance (all p > 0.19) (compare Fig. 4B, top). Thus, at least for load 1 and load 3, cross-task decoding was successful within left superior IPS (i.e., probabilistic area 7A) at the group level, indicating that color information of the trained feature could be decoded from distributed patterns within this region during the WM task (at 2–4 s after onset of the sample stimulus display).
Based on these results, we evaluated whether load-dependent decoding accuracy from left superior IPS (i.e., probabilistic area 7A) is associated with the model parameters mean precision, variability of precision, or guessing behavior. We applied a within-subjects correlation approach (Bland and Altman, 1995) that takes into account repeated-measures (i.e., three load conditions) per participant and that has been used previously in a similar context (Emrich et al., 2013) (see also Materials and Methods). ANCOVA results showed significant relations between the load effects of decoding accuracy and mean precision (ρ = −0.30, F = 4.18, p = 0.04) as well as variability of precision (ρ = −0.31, F = 4.46, p = 0.04), such that lower decoding accuracy correlated with a higher mean SD (i.e., lower precision) and a higher variability of precision (compare Fig. 4B, bottom). No such relation was found for the guess rate parameter (p = 0.12). Visual inspection of Figure 4B indicates that three participants show no decoding accuracies above chance level in any of the three load conditions. Replicating the brain-behavior correlations after excluding these three participants did not substantially change the results (mean precision: ρ = −0.447, p = 0.004; precision variability: ρ = −0.302, p = 0.06). To ensure that the effects within left superior IPS are specific and not spuriously induced by confounding factors not related to the experimental intervention, we finally also conducted the same analysis for control ROIs in which no effects of visual WM precision were expected (i.e., for probabilistic masks of the left and right primary auditory cortex). Importantly, we observed no significant relationship between decoding accuracy and behavioral mean precision (left TE: p = 0.69; right TE: p = 0.34), variability of precision (left TE: p = 0.87; right TE: p = 0.07), and guessing behavior (left TE: p = 0.85; right TE: p = 0.51) in these ROIs.
Thus, to summarize the multivariate fMRI results, neural precision (as determined on the basis of the accuracy of decoding WM stimuli from brain activation patterns during the WM task at 2–4 s after memory array onset) correlated with the deterioration of behavioral mean precision under increasing set size. Precision variability showed a relation in the same direction, and these effects were restricted to left superior IPS.
Discussion
To understand the neural underpinnings of capacity limitations in visual WM, we combined fMRI with cognitive modeling of behavioral performance during a color-WM task with continuous response format. Consistent with previous reports, behavioral modeling revealed that, when WM task load increased (i.e., 5 > 3 > 1), encoded colors were remembered with lower precision (higher mean SD) and WM performance was more variable (higher SD variability) and more random (higher guess rate) (Bays and Husain, 2008; e.g., Zhang and Luck, 2008; Fougnie et al., 2012; van den Berg et al., 2012). Subjects differed substantially in their load-dependent deterioration patterns of these model parameters. Combining these modeling results with fMRI demonstrated that individual differences in the load-dependent increase of precision variability predict the strength of superior parietal (i.e., probabilistic area 7A) and primary visual (i.e., probabilistic V1) activation during the WM task (at 2–4 s after onset of memory array). Further, within left area 7A, both decreasing behavioral precision and increasing precision variability was mirrored by decreasing accuracy of decoding WM stimuli from brain activation patterns, suggesting the precision of parietal representations as a potential contributor of WM capacity limitations.
The main novel contribution of this study is our observation that precision variability, which has most recently been acknowledged as an important parameter contributing to individual capacity differences (Fougnie et al., 2012; van den Berg et al., 2012, 2014) but has so far not been investigated at the neural level, is associated both with the strength of BOLD activation and the quality of neural representations in superior IPS. Earlier studies suggested that superior IPS limits the number of items that can be retained in WM (Todd and Marois, 2004, 2005; Vogel and Machizawa, 2004; Vogel et al., 2005) or imposes variable capacity limits dependent on WM delay length (Magen et al., 2009) or object complexity (Xu and Chun, 2006; Xu, 2009; Domijan, 2011). However, other brain areas also showed capacity-limited activation patterns (e.g., Mitchell and Cusack, 2008), so that the neural mechanisms underlying WM capacity limitations are not yet sufficiently understood. Our results extend and further specify these proposals by relating the activation of superior IPS as well as early visual cortex to a specific aspect quantified in the variable precision model (i.e., the variability of precision of WM representations over time) and by indicating that these processing limitations are operative already during perception and encoding of stimuli. Our data further suggest, at least for left superior IPS, that behavioral precision estimates are affected by the precision of neural coding.
This result is consistent with single-neuron recordings from monkey visual cortex and lateral intraparietal area (LIP) demonstrating spike counts that could be described by doubly stochastic processes (Churchland et al., 2011), which in turn are possible neuronal correlates of behavioral variability (van den Berg et al., 2012). The IPS is considered as human homolog of monkey LIP, including its visual, attentional, memory, and saccade-related specializations (Culham and Kanwisher, 2001). Consistent with our work, monkey data show capacity limits during the initial sensory encoding in area LIP (Buschman et al., 2011). Thus, noise in the quality of WM encoding might be a critical component underlying limits in WM capacity (Palmer, 1990; Buschman et al., 2011; van den Berg et al., 2012), either stemming from perceptual and/or mnemonic processes. Moreover, trial-to-trial and item-to-item fluctuations in attentional gain could constitute possible sources of variability of encoding precision (van den Berg et al., 2012; Goris et al., 2014). Thus, it is possible that the variability of WM precision may result from random fluctuations of attention when (multiple) targets have to be remembered (e.g., Palmer, 1990; Pessoa et al., 2002; Wilken and Ma, 2004; Bays and Husain, 2008; Zhang and Luck, 2008; Fougnie et al., 2012; van den Berg et al., 2012; Ma et al., 2014). However, it has also been proposed that variable memory precision may arise from controlled shifts of attention occurring during WM encoding (Lara and Wallis, 2012; van den Berg et al., 2012), a function that is compatible with known superior IPS functionality (e.g., Serences and Yantis, 2006). This assumption is based on findings that visual cues during encoding can enhance memory precision, possibly due to a more focused allocation of selective attention resources (Bays and Husain, 2008; Lara and Wallis, 2012). Even without cueing, participants may have adopted a strategy of covertly shifting attention to subsets of items during encoding and greater engagement of brain regions controlling such attention shifts may stabilize performance even in the face of high load.
Another attention mechanism that may be consistent with the control of precision variability may be a higher vigilance or sustained attention during the task, which may indirectly modulate more transient attention processes during WM, which in turn may affect precision and/or variability (van den Berg et al., 2012; Ma et al., 2014). Lesion studies and a recent meta-analysis of neural mechanisms underlying sustained/vigilant attention identified a mainly right-lateralized network, including prefrontal cortex, anterior insula, IPS, temporoparietal junction, and subcortical structures (Petersen and Posner, 2012; Langner and Eickhoff, 2013). These findings, however, are less likely to integrate with our results, as the specific parietal subregion we identified, left area 7A, was not part of this vigilant attention network. Finally, higher superior IPS activity may also result from increased meta-cognitive processes during task engagement, which may stabilize performance. Single-neuron recordings have revealed that neurons in monkey LIP encode the certainty about perceptual decisions (Kiani and Shadlen, 2009). Moreover, parietal damage was related to subjective confidence ratings in free-recall episodic memory (Simons et al., 2010) and trial-to-trial variability of subjective confidence ratings (i.e., meta-cognition of ongoing task performance) was highly correlated with task performance and consistent with the variable precision model (Rademaker et al., 2012; Ma et al., 2014). Higher awareness of ongoing task performance might, according to this assumption, result in stronger IPS recruitment. While this meta-cognitive account could potentially explain the absence of a direct relationship between load-dependent activation increase and mean precision, it is difficult to integrate into the observed association between neural and behavioral precision.
Whereas previous research observed a relation between load-dependent decoding of WM contents and behavioral precision for sensory regions (Emrich et al., 2013), we here additionally demonstrate this association within left superior IPS. Combined, these results suggest that increasing the load on WM results in a decline in quality of neural representations of the memoranda (see also Sprague et al., 2014; Ester et al., 2015), already during WM encoding. Inconsistencies between studies, particularly with respect to where in the brain such brain-behavior association can be observed, may be due to distinct task and design choices (e.g., set sizes 5 > 3 > 1 vs 3 > 2 > 1; classification of direction of motion vs location vs color, leave-one out approach vs generative reconstruction model vs cross-classification, sequential vs simultaneous stimulus presentation). Our results are in line with the recent demonstration that WM contents during simultaneous distraction can be decoded from superior IPS but not from occipital regions (Bettencourt and Xu, 2016). We added to these findings that superior IPS influences WM precision already during early stages of the WM task (i.e., during memory array presentation), by modulating processes of mnemonic encoding and/or perceptual processes that take place during encoding. Given the central role of variability in current models of visual WM capacity, superior IPS seems to play a critical role for WM capacity limitations, at least under conditions of increased load that reach or even exceed the individual capacity limit.
The temporal resolution of fMRI prevents a more time-resolved analysis of component processes of WM. Here, we worked with a relatively short delay length of 4 s, to keep the task as comparable as possible with the behavioral research in which the variable precision model is rooted, and to collect as many trials as possible. Thus, we cannot unequivocally distinguish brain activity reflecting encoding versus maintenance processes (even though our analyses clearly bias the results toward encoding processes). In particular, a limitation of this study, as well as of many other studies using the same stimulus presentation format, is that as WM load increases, so does perceptual load, and that the resulting neural effects are not separable using fMRI. Thus, capacity limitations arising during WM encoding may be rooted in mnemonic or perceptual processes (e.g., Palmer, 1990, Ma et al., 2014). Hence, future work should focus on temporally dissociating how different cognitive subprocesses contribute to visual WM limitations (e.g., using electrophysiological signals) (Bledowski et al., 2006, 2012).
In conclusion, our results suggest that superior IPS and V1 are critical for individual differences in WM capacity, as their load-related activity patterns are directly related to the quality of visual WM. In particular, within superior IPS, the quality of neural coding during perception and memorization of visual stimuli relates to behavioral precision, whereas the overall recruitment of this area seems to stabilize WM representations in the sense of reducing behavioral performance variability.
Footnotes
This work was supported by the Initiative for the Development of Scientific and Economic Excellence of the state of Hessen. C.J.F. was supported by The Netherlands Organization for Scientific Research VIDI Grant 45209006 and European Research Council Consolidator Grant Agreement 617891. C.B. was supported by German Research Foundation (DFG) Grant BL 931/3-1. We thank Tobias Becker for assistance during data collection.
The authors declare no competing financial interests.
- Correspondence should be addressed to Elena M. Galeano Weber, Goethe University Frankfurt, Department of Psychology, Theodor-W.-Adorno-Platz 1 (PEG), D-60323, Frankfurt am Main, Germany. galeanoweber{at}psych.uni-frankfurt.de
