Abstract
Traditionally, load sensitivity of sustained, elevated activity has been taken as an index of storage for a limited number of items in visual short-term memory (VSTM). Recently, studies have demonstrated that the contents of a single item held in VSTM can be decoded from early visual cortex, despite the fact that these areas do not exhibit elevated, sustained activity. It is unknown, however, whether the patterns of neural activity decoded from sensory cortex change as a function of load, as one would expect from a region storing multiple representations. Here, we use multivoxel pattern analysis to examine the neural representations of VSTM in humans across multiple memory loads. In an important extension of previous findings, our results demonstrate that the contents of VSTM can be decoded from areas that exhibit a transient response to visual stimuli, but not from regions that exhibit elevated, sustained load-sensitive delay-period activity. Moreover, the neural information present in these transiently activated areas decreases significantly with increasing load, indicating load sensitivity of the patterns of activity that support VSTM maintenance. Importantly, the decrease in classification performance as a function of load is correlated with within-subject changes in mnemonic resolution. These findings indicate that distributed patterns of neural activity in putatively sensory visual cortex support the representation and precision of information in VSTM.
Introduction
Although performance on many everyday cognitive tasks depends on the ability to maintain and manipulate multiple items in visual short-term memory (VSTM), it remains unclear precisely how multiple visual items are represented simultaneously in the brain. A widely accepted hallmark of VSTM storage is the presence of sustained, elevated delay-period activity (Fuster and Alexander, 1971; Funahashi et al., 1989), which is thought to underlie the “active” mechanisms of short-term maintenance. In human neuroimaging, the sensitivity of this activity to load is considered even stronger evidence for VSTM storage. For example, functional magnetic resonance imaging (fMRI) signal in intraparietal sulcus (IPS) increases with increasing memory load and asymptotes at apparent capacity limitations (Linden et al., 2003; Todd and Marois, 2004; Xu and Chun, 2006).
More recently, studies using multivoxel pattern analysis (MVPA) have demonstrated that the identity of a single item held in VSTM can be decoded during the delay period from early visual regions (e.g., V1-V4, MT), even in the absence of sustained delay-period activity (Harrison and Tong, 2009; Serences et al., 2009; Riggall and Postle, 2012). Although these more recent findings suggest that VSTM representations are coded in distributed patterns of activation in early visual cortex, it remains unclear whether such presumably sensory-based representations respond to changes in VSTM load and, critically, how these changes may reflect load-dependent changes in VSTM performance.
The present study investigated the sensitivity of sensory-cortex-based short-term memory representations to memory load by acquiring fMRI while subjects performed delayed recall of one, two, or three presented directions of motion (Fig. 1A). Pattern classifiers were trained to classify the remembered direction of motion and we examined how direction-specific classifier evidence changed as a function of VSTM load. The manipulation of load also enabled us to investigate the neural basis of precision in VSTM. Recent evidence suggests that there is a measureable decline in the level of detail or “precision” of VSTM representations as information load increases (Bays and Husain, 2008; Zhang and Luck, 2008; van den Berg et al., 2012). According to these models, this decrease in mnemonic resolution reflects a decrease in the proportion of neural resources dedicated to each representation. Accordingly, we predicted that this loss of mnemonic resolution as a function of load should be reflected in classification performance, reasoning that declines in both classification performance and behavioral precision may result from noisier neural representations. Relating classification performance to behavioral precision would be strongly consistent with the idea that patterns of activity in sensory cortex reflect the neural representation of information held in VSTM, even in the absence of sustained, elevated delay-period activity in these regions.
Materials and Methods
Participants
Ten right-handed volunteers, three female, ages 23–31 years (median = 25.5 years), from the University of Wisconsin–Madison community participated in the study for a small remuneration ($15/h). All subjects provided informed consent according to the procedures approved by the Health Sciences Institutional Review Board at the University of Wisconsin–Madison. Subjects had normal or corrected-to-normal vision, no contraindications for MRI, and no reported history of neurological disease.
Stimuli and procedure
Participants underwent fMRI while performing six runs of a delayed recall task for visual motion (Fig. 1A). A gray fixation cross (∼0.22°) was presented at the beginning of each trial on a black background and was changed to white 2 s before the onset of the memory sample. The memory sample lasted a total of 2 s (1 TR), during which 3 patterns of dots were presented. Dot patterns were circular patches (∼11° diameter) of dots (∼0.13°) with a density of ∼0.7 dots-per-square degree. Dot patterns were centered around the fixation and uniquely and randomly presented in red, green, or blue without repeating colors within a trial. Each dot pattern was presented for 500 ms and stimuli were separated by an interstimulus interval of 250 ms. On any given trial, one, two, or three of the dot patterns could be moving with 100% coherence at 2.75°/s while the remaining dots (loads 1 and 2) remained static. On 90% of trials, one of the presented directions was sampled from one of three directions (7°, 127°, 247°). These were the directions used as labels for the patterns used to train the classifier and were also selected as the probed (target) directions on those trials. Because we wanted behavioral estimates of precision to correspond directly to the neural data, the probed item was always the stimulus with one of these three directions of motion. The remaining moving patterns on these trials were sampled randomly from the 360° space. No restrictions were placed on the selected directions of these remaining items. On the remaining 10% of trials, all moving patterns were randomly selected, including the probed direction, to prevent participants from learning the target directions. The sample display was followed by an 8 s delay period, over which participants were instructed to remember the direction of motion of all moving patterns. After the delay, a probe display appeared for 5 s. The probe consisted of a colored circle (∼11° in diameter) with a line (∼4.5°) extended between the center of the screen and random location on the circumference of the circle. Participants were instructed to rotate the line so that it matched as closely as possible the direction of motion of the color-matching dot pattern. Participants rotated the line by pressing buttons that rotated the line 10° clockwise, 10° counterclockwise, or flipped it 180°. The starting position of the line segment was randomly selected, with the constraint that it was presented within a multiple of 10° of the probed sample direction. On those trials in which one of the critical items was presented, the probe color always matched the color of the critical stimulus. On the remaining 10% of trials, the probed stimulus direction was randomly selected. The temporal position of the target stimulus was also counterbalanced within each block. A 7 s intertrial interval followed the probe, during which the fixation cross changed to gray.
Participants performed 180 trials over the six runs in the scanner. Each block contained 27 critical trials, three of each load × direction × temporal position pairing, as well as three noncritical trials, one of each load. The color of the sample was always randomly selected. Experimental stimuli were controlled by the Psychophysics Toolbox (http://psychtoolbox.org; Brainard, 1997) running in MATLAB (MathWorks), presented using a 60 Hz projector (Silent Vision 6011; Avotec), and viewed through a coil-mounted mirror.
Behavioral analysis
Behavioral performance was assessed using the mixture-model developed by Zhang and Luck (2008) and extended by Bays et al. (Bays et al., 2009; Zokaei et al., 2011). This model breaks down the distribution of error (the angular deviation between the sample direction and the indicated response direction) into estimates of target accuracy [proportion of responses toward the target direction (PT)], error [nontarget responses (PNT) and guesses (PG)], and measure of precision (κ, or concentration; the variability of recall of the target and nontarget responses). That is, responses are modeled to be a mixture of a Von Mises distribution (a circular analog of the Gaussian distribution) with a concentration of κ for all target and nontarget responses plus a uniform distribution of random responses (guesses). The concentration parameter, κ, corresponds to the variability of the target (and nontarget) responses, with larger values indicating less variability. The model is defined as follows:
where θ is the target motion direction, θ̂ is the response direction on a given trial, α is the probability of reporting the target direction, and β is the probability of reporting a nontarget direction (of m nontarget directions). γ is defined as 1 − α − β, and represents the probability of responding at random (Bays et al., 2009 and Zokaei et al., 2011 contain figures depicting the relationship between the response distribution and the model parameters).
Parameter estimates for α, β, γ, and κ were obtained using maximum-likelihood estimation (expectation maximization) using MATLAB routines (available at http://www.bayslab.com). Responses were entered for each trial, and separate estimates were obtained for each participant and condition.
Imaging analysis
Functional data were preprocessed using the Analysis of Functional NeuroImages (AFNI) software package (http://afni.nimh.nih.gov; Cox, 1996). All volumes were spatially aligned to the final volume of the final run using a rigid-body realignment and corrected for slice-time acquisition. Linear, quadratic, and cubic trends were removed from each run to reduce the influence of scanner drift. For univariate analyses, data were spatially smoothed with a 6 mm FWHM Gaussian. For classification analyses, data were z-scored separately within run for each voxel. Data were not smoothed and were left in their native space.
Univariate region-of-interest analysis.
The goal of these analyses was to identify regions-of-interest (ROIs) in which the activity, either in response to stimulus presentation or during the delay period, varied monotonically with load. We did this by solving a modified general linear model (implemented in AFNI) in which regressors modeling the sample presentation and delay period were generated as 2 and 8 s boxcars, respectively, and convolved with the canonical hemodynamic response function supplied with AFNI. This approach is commonly used to discriminate transient, “stimulus-evoked” activity that returns to baseline upon the offset of the sample from activity that persists at an elevated level across the delay period despite the absence of stimulus input (Postle et al., 2000). The model also included three levels of stimulus load (1, 2, and 3).
Figure 2 illustrates regions in which “sample” and/or “delay” activity varied significantly with load. To construct ROIs for hypothesis testing MVPA analyses, the parameter estimates used to produce the map shown in Figure 2 were used to identify regions for which load sensitivity of the sample epoch was statistically greater than load sensitivity of the delay epoch (“Sample ROI”) and regions for which the converse was true (“Delay ROI”).
Anatomically defined ROIs.
In addition to the univariate ROIs, MVPA was also performed on anatomical ROIs created with the automatic parcellation routines (Fischl et al., 2008) implemented by FreeSurfer (http://surfer.nmr.mgh.harvard.edu/). These ROIs included: V1 and V2, MT, intraoccipital sulcus (including the transverse occipital and superior occipital sulci) and intraparietal sulcus (including the transverse parietal sulcus). All voxels within the anatomical ROIs were included in the MVPA analyses.
Note that we did not include frontal areas among anatomical ROIs for two reasons. First, many previous studies using fMRI, fMRI-guided repetitive transcranial magnetic stimulation, and patients with frontal lesions have either failed to find load sensitive delay-period activity in frontal cortex (Postle et al., 1999; Feredoes and Postle, 2007) or failed to find that disruption of frontal cortex affects the retention of multiple items (D'Esposito and Postle, 1999; Postle et al., 2006; Feredoes et al., 2007; Koenigs et al., 2009). Second, two recent studies using MVPA methods similar to those used here failed to find evidence for delay-period retention of stimulus-related information in frontal cortex (Christophel et al., 2012; Riggall and Postle, 2012).
Pattern classification analyses.
To examine the neural representations associated with VSTM maintenance, we trained pattern classifiers to classify each of the three critical directions of motion and examined classifier sensitivity for each direction using a leave-one-trial-out approach. Classification was performed using the Princeton Multi-Voxel Pattern Analysis toolbox (www.pni.princeton.edu/mvpa/) and custom routines in MATLAB. Preprocessed fMRI data from individual trial time points were used to train separate classifiers to classify the direction of motion (three possible directions) or the memory load (three possible loads). Classification was accomplished using L2-regularized logistic regression with a lambda penalty term of λ = 25. Training for direction was performed collapsed across all three loads; similarly, training for load was performed collapsed across all three directions. Note that all MVPA analyses were performed on data that were neither smoothed nor time-shifted in any other way. Classification was performed on the top 2000 voxels from the sample- and delay-load-sensitive general-linear-model-derived ROIs and on all voxels within the anatomically defined ROIs.
For each 2 s TR of fMRI data, the trained classifier produced a probability estimate (from 0–1) of the extent to which the observed pattern on the tested trial matched the trained pattern for each of the trained items (e.g., the three directions of motion). Classification performance was determined using leave-one-trial-out cross-validation, in which the classifier was trained on data from all but one trial and then tested on the remaining trial, rotating through all possible permutations. Training was performed using all possible trials (collapsing across the untrained dimension of load or direction). For each of the three motion directions, we computed a receiver-operating characteristic (ROC) based on the values obtained across all tested trials, and then averaged across the dimensions of interest. Therefore, this method evaluated whether, across trials, evidence for one direction of motion was higher when that direction was in fact presented on that trial relative to when it was not presented. The area under the ROC curve, averaged across the three directions of motion, provides a measure of overall classifier sensitivity, with chance performance at 0.5. Analogous methods of classification and ROC-based sensitivity analysis were used to classify memory load, collapsing across direction of motion.
To examine the dynamics of the memory representation, each classifier was trained on data from only a single time point in the trial (e.g., TR 5) and then tested on all points in the left-out trial. This resulted in a time course of decoding sensitivity for the entire trial that was determined by the weights of the single training point and provides a measure of how stable the representations were over time.
The significance of classifier performance was determined using one-tailed, one-sample t tests, testing against chance performance of 0.5. Tests performed across individual TRs were corrected for multiple comparisons using the method of Larzelere and Mulaik (1977). Within-subject correlations were computed using the method of Bland and Altman (1995) by calculating ANCOVA with the factors of classifier sensitivity and subjects as the covariates and behavioral performance as the dependent measure, and fitting the model to parallel lines.
Within-subject correlation.
Although classification performance was significantly above chance at the group level, there was significant variance between subjects in the ability to classify direction of motion. Consequently, we attempted to examine the relationship between classification performance and behavioral precision while accounting for these individual differences in classification performance. A number of studies of VSTM have typically normalized between-subject differences in neural signal by calculating pairwise differences between loads (Vogel and Machizawa, 2004; Vogel et al., 2005; Emrich et al., 2009). Here we instead attempted to examine whether changes in the relative amount of neural information across all three tested memory loads predicted the change in the quality of representations maintained in VSTM; that is, we took advantage of the variation in dependent measures across all three loads by using a within-subject correlation approach (Bland and Altman, 1995) to examine the change in both classifier performance and behavioral precision across loads. Specifically, we used ANCOVA to remove between-subject differences in classification performance and measured the remaining variation in classification performance explained by variation in behavioral precision.
Results
Behavioral results
Behavior was analyzed according to the method of Bays et al. (Bays et al., 2009; Zokaei et al., 2011). This method uses a mixture-model of the behavioral response error (the angular distance between the reported direction and the actual direction of the probed memory stimulus) to obtain maximum-likelihood estimates for the proportion of correctly recalled targets, the precision of those responses (the distribution of error around the target), and the proportion of errors (both nontarget responses and guesses; see Materials and Methods).
Examining the concentration parameter (κ, the estimate of the variability of memory precision) of this model using repeated-measures ANOVA revealed a significant effect of load (F(2,18) = 8.55, mean squared error [MSE] = 500.4, p = 0.0025; Fig. 1B). A similar effect was observed for the proportion of target responses (PT; F(2,18) = 7.97, MSE = 0.011, p = 0.0033). Therefore, consistent with previous studies examining memory for direction of motion (Zokaei et al., 2011), both the number of correctly recalled targets and the precision of those responses decreased as the amount of to-be-remembered information increased. We also observed a significant number of nontarget (i.e., “binding”) errors that increased at larger set sizes (PNT; F(2,18) = 7.15, MSE = 0.01, p = 0.0052).
Behavioral paradigm and performance. A, Schematic of the delayed-recall task performed by participants while undergoing fMRI. On every trial, participants were presented with 3 patches of dots displayed within a single 2 s TR and instructed to remember the direction of motion of the moving dots. Memory load was manipulated by varying the number of dot patterns that contained 100% coherent motion. After the 8 s delay, participants indicated the direction of the probed (color-matching) direction by rotating the angle of the line segment. B–D, Behavioral results derived from maximum likelihood estimates of the response error. B, The concentration parameter, κ, decreased as a function of memory load, indicating a decrease in mnemonic precision. C, The PT similarly decreased with increasing VSTM load. D, PNT and PG. Nontarget responses account for the majority of the errors and increased with VSTM load.
Because we wanted behavioral responses to match as closely as possible the decoded patterns of brain activity, participants were probed on the three critical directions on a majority of trials. This procedural limitation could have allowed participants to learn which directions were most likely to be probed, thereby affecting their behavioral responses. To determine whether such learning occurred, we obtained parameter estimates for the first and second half of the experiments, collapsed across load. This analysis revealed no significant changes in either the concentration of target responses (t(9) = −1.31, SD = 12.33, p = 0.22) or the likelihood of reporting the target (t(9) = −1.12, SD = 0.043, p = 0.29), which is consistent with previous studies that failed to show learning for individual items in tasks of VSTM (Olson and Jiang, 2004; Beck et al., 2008). Moreover, the estimates obtained here are very similar to those obtained in a task that used a similar design, but that selected probes at random (Zokaei et al., 2011), suggesting that participants' responses were unaffected by nonrandomzied target probes.
Classification results
ROI analysis
Before examining whether classification performance changes as a function of VSTM load, we first tested whether it was possible to successfully decode the direction of motion of one of the memory stimuli while multiple directions were held in VSTM. A previous study by our group (Riggall and Postle, 2012) reported that decoding the direction of a single item is possible within those areas that support motion perception (i.e., calcarine and extra-calcarine occipital cortex, including MT+) but not in areas that showed sustained activity during the delay (IPS, frontal areas). Accordingly, we focused our initial analysis on nonoverlapping ROIs that demonstrated load-sensitive BOLD signal intensity either in a phasic response to the memory sample (“Sample ROI”) or in a response that was sustained throughout the delay period (“Delay ROI”). In this way, we sought to determine whether information about the remembered direction of motion was coded in cortical areas sensitive to visual stimulation independently from those that exhibited load-sensitive, sustained, delay-period BOLD signal.
Examining the patterns of activity in the Sample ROI, classification of the direction of motion resulted in significantly greater-than-chance sensitivity, averaged across all three loads (Fig. 3C). This was true both when the classifier was trained on the time point that captured the peak of the sample-evoked response and when it was trained on the final time point of the delay period, a time point that would be dominated by delay-period activity. In contrast, those areas that exhibited sustained, elevated delay-period BOLD signal (i.e., the Delay ROI) showed no evidence of direction sensitivity during any point in the trial (Fig. 3D), replicating the results of Riggall and Postle (2012). As a control analysis, training the classifier on intertrial fixation activity resulted in chance-level classification performance for both ROIs.
ROIs used for classification analysis. Initial analysis was restricted to those areas that showed parametrically increasing BOLD activity as a function of load that were either transiently evoked by the initial sample or sustained throughout the delay. Subtractions were performed to isolate the Sample (green) and Delay (red) ROIs. Areas showing load sensitivity during both periods (yellow) and those not showing load sensitivity were also included in the subsequent analysis of anatomically defined ROIs. Note that this figure demonstrates group-averaged data, whereas all analyses were performed on single-subject ROIs.
BOLD signal intensity and classification performance in functional and anatomical ROIs. A, Diagram showing load-sensitive stimulus-evoked activity, but no sustained delay activity, in the functionally defined, Sample ROI. B, Diagram showing elevated, sustained load-sensitive activity in the functionally defined, Delay ROI. C, D, Classifiers were trained on data using only the TR at −2 s (gray circle), at 4 s (teal circle), or at 8 s (red circle), and then tested across the entire trial, with performance of each indicated by the waveform with the same color. TRs at which a classifier's performance exceeded chance (*p < 0.05, **p < 0.01, one-tailed, corrected) are indicated and color-coded by classification training point. Classification sensitivity is presented in for both the Sample (C) and Delay (D) ROIs. E, Mean classification across the delay period (2–12 s) for functionally defined (Sample; Delay) and anatomically defined (introcciptial sulxus [IOS]; V1, V2; MT) ROIs. Significance (one tailed, corrected) for each ROI is indicated. Shaded areas (A–D) and error bars (E) denote SEM.
It is possible that by focusing on areas in which the load-sensitive response was specific to either the sample or delay periods, this analysis overlooked important voxels that are critical to the maintenance of information in VSTM (e.g., those that show elevated levels of BOLD signal intensity to both encoding and maintenance–those that appear yellow in Fig. 2). However, further analysis restricted to several anatomically defined areas confirmed that the majority of information about direction of motion during the delay period was exhibited in calcarine and extra-calcarine occipital cortex (Fig. 3E). Classification performance was not significantly better than chance in the IPS. Therefore, although elevated delay-period activity was absent from the observed BOLD signal intensity in areas associated with visual perception (Fig. 3A), the ability to maintain the representation of multiple items in VSTM appears to be supported by distributed patterns of activity in these areas rather than in those areas that exhibit elevated delay-period activity.
Load sensitivity
A critical aim of the present study was to determine whether classification performance would decrease as a function of memory load in a manner that reflected the decline in behavioral precision. To investigate this, we trained a classifier on the final time point of the delay period for both of the Sample and Delay ROIs. Next, for each ROI, we decoded classification performance at each time point from each of the three load conditions, resulting in a classification time course that reflected the extent to which stimulus-specific neural representations were present across the entirety of the trial.
Examining the entire delay period (4–12 s) in the Sample ROI, mean classification sensitivity was significantly greater than chance for all three loads (p < 0.01 adjusted; Fig. 4A). Therefore, even in the presence of competing within-category representations, memory representations can be identified in the patterns of activity in sensory cortex. Direction sensitivity significantly decreased during the delay period at higher loads (F(2,18) = 5.63, MSE = 0.009, p = 0013), which is consistent with the decline in behavioral performance at higher set sizes. One concern with this result might be that presenting multiple stimuli in a single location might lead to a loss in classifier sensitivity at higher loads independently of VSTM; that is, by presenting multiple items at one location, the same neurons (within the same voxels) in visual cortex would be required to process the perceptual information of each of the directions of motion, resulting in a loss in sensitivity for any given direction. If this alternative account were correct, one would expect to see the same load-related decline in classification during the stimulus presentation epoch of the trial. However, no such decrease as a function of load was observed when this procedure was repeated with classifiers trained on the time point that captured the peak of the sample-evoked response (i.e., 4–6 s) and testing classification sensitivity during the sample-evoked period (i.e., 4–8 s; F(2,18) = 2.5, MSE = 0.006, p = 0.11). Therefore, our results are consistent with the interpretation that the load-dependent decrease in classification sensitivity during the delay period was due to a reduction in the proportion of neural resources dedicated to representing that information in VSTM.
Load sensitivity of patterns of neural activity. Classifiers were trained on a single TR (denoted by the colored dots) and tested across the entire trial. A, B, Classification sensitivity for direction of motion as a function of load in the functionally defined Sample (A) and Delay (B) ROIs. Averaging across the entire delay (denoted with gray shading), classification performance for all three loads is significantly greater than chance (p < 0.01, one-tailed corrected) in the Sample ROI (A), but not in the Delay ROI (B). C, Classification sensitivity for load (number of moving dot patterns) in the Sample ROI, collapsed across direction. Significance is tested at individual TRs (*p < 0.05, **p < 0.01, one-tailed, corrected). Shaded areas denote SEM for each TR.
In addition, even though delay-period BOLD signal intensity in the Sample ROI was not significantly affected by load (Fig. 2A; F(2,18) = 2.64, MSE = 0.0052, p = 0.0991), VSTM load could be successfully decoded from sensory regions (Fig. 4C). This finding provides further evidence that the patterns of delay-period activity in areas involved in visual perception reflect the representation of multiple items in VSTM independently of changes in BOLD signal intensity identified in univariate signal intensity. In contrast, despite the presence of load-sensitive BOLD signal in the Delay ROI (Fig. 2B; F(2,18) = 5.11, MSE = 0.01386, p = 0.0174), classification performance of the contents of VSTM (i.e., direction of motion) was not significantly above chance at any load (p > 0.05; Fig. 4B). Therefore, although BOLD activity in these regions (including IPS) is sensitive to VSTM load, we failed to find evidence that these areas are involved in the representation of the contents of VSTM per se. In the Discussion section, we consider other aspects of task performance to which this activity may correspond.
Relating classification performance to response precision
Does the decrease in classifier performance as a function of memory load reflect the associated decline in mnemonic precision? To examine this question, we specifically focused on within-subject changes across load (see Materials and Methods), because there may be sources of error that affect between-subject classification performance that are unrelated to differences in behavior (e.g., differences in encoding strategies, Linke et al., 2011b, Vicente-Grabovetsky et al., 2012; or individual differences in signal-to-noise ratio of BOLD signal, Tong et al., 2012). This analysis revealed that, across loads, the observed decrease in peak classifier sensitivity was significantly correlated with an individual's change in mnemonic precision (r = 0.58, p = 0.006; Fig. 5A). This was also true when the analysis was restricted to the individual TRs corresponding most closely to the encoding (TR 4; r = 0.45, p = 0.04) or delay activity (TR 6; r = 0.53, p = 0.01).
The relationship between Sample ROI classifier sensitivity and behavioral performance. A, B, Within subjects, changes in classifier sensitivity across load are significantly correlated with changes in behavioral precision (A), but not with changes in the proportion of target responses (B). Data are modeled for each subject and fit with parallel lines with ANCOVA using the method of Bland and Altman (1995).
It is possible, however, that the decrease in classifier performance reflects the decreased probability that the probed (target) direction was stored in memory at higher loads; that is, given that both the proportion of correctly recalled targets, and the precision of those responses, decreased as a function of load, the decline in classifier sensitivity could reflect the proportionally fewer trials on which the target direction was successfully encoded and maintained. Contrary to this alternative account, however, PT was uncorrelated with peak classifier sensitivity (r = 0.21, p = 0.35; Fig. 5B). In other words, the change in classifier sensitivity across memory loads cannot be predicted by a decrease in the percentage of trials on which the critical direction of motion was stored in VSTM. Therefore, the decrease in classification performance appears to reflect the decline in mnemonic resolution, rather than decreases in other aspects of behavior or random decreases that are independent of the internal representations that guide behavior (i.e., noise).
Discussion
In the present study, we sought to investigate how the neural patterns associated with the contents of VSTM change as a function of load. Consistent with several recent studies, we observed that stimulus-specific properties of information maintained in VSTM could be decoded from the patterns of activity in sensory cortex (Ester et al., 2009; Harrison and Tong, 2009; Serences et al., 2009; Riggall and Postle, 2012). We have extended these previous findings by demonstrating that the successful classification of the contents of VSTM is not an all-or-none phenomenon; instead, classification sensitivity varies as a function of VSTM load. Therefore, these findings reveal the presence of load sensitivity in the patterns of activity in sensory cortex, providing strong evidence that these patterns may represent a neural correlate of VSTM maintenance.
We also observed a significant within-subject correlation between classifier sensitivity and mnemonic precision. This finding provides some of the first evidence that variation in mnemonic precision can be extracted from the neural patterns associated with the storage of information in VSTM. A number of recent psychophysical models have established that the quality of VSTM representations decreases as a function of memory load (Palmer, 1990; Wilken and Ma, 2004; Bays and Husain, 2008; Zhang and Luck, 2008; Bays et al., 2009; van den Berg et al., 2012). Although the models that explain these data differ in their stance on whether VSTM performance is limited primarily by a finite pool of resources or if it also contains a fixed capacity of a limited number of slots, these models all emphasize that the precision of maintained representations varies with load, particularly at low set sizes (i.e., when the number of items is below any putative capacity limits). Therefore, “slots + resource” (Alvarez and Cavanagh, 2004; Buschman et al., 2011) and “slots + averaging” (Zhang and Luck, 2008, 2011; Anderson et al., 2011;) hybrid models and pure resource models (Wilken and Ma, 2004; Bays and Husain, 2008; van den Berg et al., 2012) can accommodate the findings reported here, in that all hold that the precision of maintained representations will vary with load at small set sizes. Specifically, our results are consistent with the prediction that, as the amount of to-be-remembered motion information increases, fewer resources will be dedicated toward representing a given item, resulting in less signal present in the patterns of neural activity that reflect this representation. Critically, this relationship between VSTM precision and classification performance provides evidence consistent with the idea that sensory visual cortex may play a critical role in supporting the representation of information in VSTM (Postle, 2006). These findings take the important step of linking the neural patterns associated with content-specific VSTM encoding and maintenance with the internal representations associated with behavior.
One possible concern with these results is that a decrease in decoding accuracy with increasing load, such as we present here, could also result from decoding noise that is unrelated to the mnemonic representation. For example, the neural response to multiple objects has been shown to be similar to the weighted sum (MacEvoy and Epstein, 2009) or average (Reddy et al., 2009) of individual responses. This alternative seems unlikely, however, for a number of reasons. First, each condition contained the same number of stimuli, but varied only in how many of the dot patterns contained motion. Therefore, this control reduces the stimulus differences between conditions that might affect the weighted averaging or summation of neural signal. Second, classification training was performed over all three tested loads. Accordingly, classification performance was based on patterns of activity that included the increased noise present on load 2 and 3 trials. Third, classification performance was unaffected by load when training and testing to the patterns of activity transiently evoked by the sample stimuli, indicating that classification performance did not decrease obligatorily due to a decrease in the signal-to-noise ratio at higher set sizes. Finally, a recent study examining the response patterns of single neurons in monkeys has shown that the neural information present about an item encoded and stored in VSTM decreases when multiple items are presented within the same hemifield (Buschman et al., 2011), providing evidence that this effect is not idiosyncratic to the classification technique used here. Together, these facts support our conclusion that classification performance can be accounted for by a decrease in the relative amount of information about the target stimulus present in the patterns of activity in sensory cortex.
Although our findings are consistent with studies that have failed to observe evidence for the contents of VSTM in the IPS (Linden et al., 2012; Riggall and Postle, 2012), one previous study did observe above-chance classification in posterior parietal cortex (Christophel et al., 2012). A possible explanation for these discrepant findings may be that the study by Christophel et al. (2012) used artificial stimuli that required significant training to differentiate. Therefore, decoding accuracy in posterior parietal cortex in this task may reflect processes that are not specific to VSTM maintenance per se. In addition, although the use of the data-driven MVPA approach in the current study has its advantages, there are limitations to this methodology. For example, evidence indicates there is a coarse-scale topographic map of orientation preference in human V1 that is strongly correlated with the retinotopic map (Freeman et al., 2011). Therefore, it is possible that multivariate classification methods could exploit these biases, thereby driving up classification sensitivity only in those areas that have strong topographic organization (e.g., V1 and V2). However, other studies have demonstrated that classification performance of orientation-selective activity is predicted by the average BOLD amplitude in a given region (Tong et al., 2012). Therefore, given that we observed greater BOLD amplitude during the delay period in the IPS than in sensory visual cortex, it is unlikely that our findings can be explained by differences in classification sensitivity across different regions. Nevertheless, future studies should examine load-dependent VSTM classification sensitivity on a range of stimuli (e.g., colors, faces) to determine the reliability of this effect across different features and cortical regions.
If the maintenance and precision of VSTM representations is mediated by patterns of activity in sensory cortex, what is the role of sustained, elevated, delay-period activity? Previous studies have provided evidence that delay-period activity may reflect general cognitive or attentional demands of the task (Mitchell and Cusack, 2008; Magen et al., 2009), as well as goal- or action-related information (Curtis and Lee, 2010; Riggall and Postle, 2012), rather than reflecting VSTM maintenance. One possibility is that whereas VSTM representations may be supported by patterns of activity in sensory cortex, sustained, elevated, delay-period activity (particularly in IPS) may reflect the directed attention toward a limited number of representations (Anderson et al., 2011). This mechanism could potentially account for both apparent capacity limits (i.e., those within the focus of attention), as well as a larger capacity for lower-resolution representations (Sligte et al., 2008, 2009, 2010).
The finding that the precision of VSTM representations is predicted by areas that mediate perceptual encoding is largely consistent with the evidence that VSTM performance is likely limited by encoding processes (Buschman et al., 2011; Linke et al., 2011b; Emrich and Ferber, 2012; Mazyar et al., 2012; van den Berg et al., 2012); that is, the continued recruitment of the same populations of neurons that mediate encoding and perception for VSTM will limit the precision of memory representations to that of encoding. In addition, when attention is allocated across two different memory categories and then retroactively cued to a single item, classifier evidence for the uncued item dropped to chance, suggesting that the allocation of resources in VSTM may be dynamic (Lewis-Peacock and Postle, 2012; Larocque et al., 2013). Therefore, while it is possible that the fidelity of VSTM may be further affected by subsequent processes, such as by fluctuations in attention (Van den Berg et al., 2012) or stochastic degradation (Fougnie et al., 2012), the correlation between classifier performance and VSTM precision during early stages of encoding rule out the possibility that all variation in precision reflects limitations in later stages of processing (Awh et al., 2007; Barton et al., 2009; e.g., during recall or comparison).
At the neural level, the successful classification of multiple items in VSTM from sensory cortex, as well as its relationship with mnemonic precision, further calls into question the widely held assumption that VWM maintenance of multiple items occurs in a network of frontoparietal regions (Courtney et al., 1997; Linden et al., 2003; Todd and Marois, 2004; Vogel and Machizawa, 2004; Xu and Chun, 2006; Reinhart et al., 2012). In particular, it calls into question the role of elevated, sustained, load-sensitive delay-period activity in IPS as a hallmark of VSTM maintenance (but see Christophel et al., 2012). Instead, our findings support the view that VSTM maintenance is accomplished by actively maintaining representations in “lower level” sensory cortex (Postle, 2006; Ester et al., 2009; Harrison and Tong, 2009; Serences et al., 2009; Linke et al., 2011a; Riggall and Postle, 2012).
Footnotes
This work was supported by the National Institutes of Health (Grant R01-MH064498 to B.R.P.).
- Correspondence should be addressed to: Stephen M. Emrich, Department of Psychology, Brock University, St Catharines, ON L2S 3A1, Canada. semrich{at}brocku.ca
