Decoding the Contents of Visual Working Memory: Evidence for Process-Based and Content-Based Working Memory Areas?

Introduction

The ability to keep visual information in mind after stimulus disappearance, often referred to as visual short-term memory or visual working memory (VWM), is crucial for guiding behavior. In early neuroimaging studies aimed at finding the neural correlates of VWM, it was assumed that brain areas involved with VWM would show elevated activity during memory maintenance (see Courtney, 2004; Postle, 2006 for thorough descriptions of the early literature). From these studies, it was evident that visual cortical areas, despite their involvement in stimulus encoding, did not show significant elevated activity during VWM retention, implying that VWM does not depend on activity in visual cortex. Instead, a widespread network of brain regions, most notably in parietal and prefrontal cortex, contributed to maintenance of visual information over a delayed interval (Pessoa et al., 2002). To date, the exact role of frontal and parietal areas in VWM maintenance still remains poorly understood. Yet, because these areas lack high visual selectivity and are thought to be involved in many other cognitive functions such as attention and control (Postle, 2006), it has been proposed that frontal and parietal brain areas are not involved in the actual storage of information, but rather control the top-down modulation of sensory areas, where the information itself is represented (Courtney, 2004; Postle, 2006).

These theories have recently gained credibility as VWM content does seem to be represented in visual cortex, even in the absence of elevated activity (Supèr et al., 2001; Harrison and Tong, 2009). Rather than being evident as an increase in neural activity, VWM maintenance of different stimuli evokes distinct patterns of neural activity in primary and secondary visual cortex (Harrison and Tong, 2009), and these activity patterns can be identified using multivoxel pattern analysis (MVPA). To understand how MVPA works, imagine we want to decode whether a horizontally or vertically oriented stimulus was presented to a participant, based on the pattern of BOLD activity in primary visual cortex. In the first analysis step, a classifier is trained on horizontal and vertical stimuli to find which voxels best discriminate between stimuli. Then, the classifier is presented with new trials that were not included in the training phase and the classifier is required to determine whether the observed pattern among voxels belongs to the horizontal or the vertical orientation. In this way, it is possible to discern what a subject was seeing (Kamitani and Tong, 2005), which stimulus was maintained in VWM (Harrison and Tong, 2009), and even what object a subject was imagining (Stokes et al., 2009). Importantly, this technique is effective even in the absence of an overall difference of brain activity in the region of interest.

Two studies that were recently published in this journal used MVPA to investigate whether the contents of VWM can also be decoded from higher brain areas. In the study by Riggall and Postle (2012), the contents of VWM (direction of motion and movement speed) could be decoded only from visually selective brain areas. Although no evidence was found for content-specific decoding in parietal and frontal regions, the authors could determine which task subjects were performing (i.e., deciding based on the direction or the speed of motion) from activity patterns within these higher-order areas. This finding suggests that VWM activity in higher brain areas is process-related instead of content-based. In contrast, the study by Christophel et al. (2012), in which participants had to remember a circle containing a complex mix of colors (Christophel et al., 2012, their Fig. 1), demonstrated visually specific VWM storage in both visual and parietal cortex, suggesting that the parietal cortex is involved in information storage as well. How can we reconcile these different findings suggesting a process-based role for the parietal lobe (Riggall and Postle, 2012) versus a content-based role for this region (Christophel et al., 2012)?

We propose that this apparent difference in functional roles can be traced back to a few subtle, yet important, differences in the methods between studies. First, the two studies differed in their method to select relevant voxels to include for training the MVPA classifiers. Riggall and Postle (2012) restricted their MVPA analyses to voxels that showed elevated brain activity in parietal and frontal cortices during the retention period. In contrast, Christophel et al. (2012) did not restrict their MVPA procedures to specific neural structures, but used a searchlight procedure that explores the entire brain and selects a few neighboring voxels at a time to include in the MVPA procedure. Therefore, Riggall and Postle (2012) may have missed parietal clusters that are sensitive to differences in VWM retention between stimuli despite elevated brain activity, a possibility that is excluded in the study by Christophel et al. (2012).

Second, the stimuli used in the two studies differed substantially. Christophel et al. (2012) used complex colored objects, while Riggall and Postle (2012) used moving-dot patterns. As Christophel et al. (2012) point out in their discussion, the complex colored objects might have introduced different saliency maps for each stimulus, and thus a difference in visual attentional allocation might have driven the decoding performance in parietal lobe. Alternatively, the fact that colored objects in the Christophel et al. (2012) study are different in spatial layout may cause spatially selective regions that are prominently present in parietal cortex to become selective for these items.

A last notable difference is that in the Riggall and Postle (2012) study, the classifier was trained on the activity during the presentation of the stimulus itself, whereas Christophel et al. (2012) trained the classifier on the activity during the maintenance period. Whereas it is highly likely that activity patterns during stimulus presentation reflect stimulus properties, it is plausible that activity patterns during maintenance reflect additional cognitive processes. Therefore, Christophel et al.'s (2012) accurate decoding in parietal lobe might reflect a difference in specific processes that occur during the maintenance of the stimulus, not necessarily memory content itself.

How might differences in MVPA patterns arise in higher-tier brain areas that are not related to content-based storage? This may occur because top-down projections from higher-tier areas project differentially into visually selective brain regions depending on task requirements and the stimuli presented (Ruff et al., 2006; Feredoes et al., 2011). Although this has not yet been investigated, such top-down projections might even differ between stimuli from the same stimulus category. Because MVPA is tailored to pick up fine-grained differences in neural patterns in the absence of differences in larger-scale BOLD activity, successful classification does not necessarily imply content-based representation in higher-level areas, but could relate to a difference in top-down modulation of lower-tier visual areas during active VWM maintenance.

In future work, it will be essential to elucidate the role of higher-order areas in VWM maintenance. One can speculate that parietal cortex and the frontal eye fields are more dedicated to spatial and feature-based attention that is relevant for VWM maintenance, while most prefrontal regions are involved with central attention and maintenance of task instructions (see also Courtney, 2004; Postle, 2006; see Sligte et al., 2010 for model). However, without empirical evidence, these specializations remain questionable. To further our understanding, future fMRI studies should combine different decoding techniques (searchlight/activation-based) and smart stimulus designs to dissect the role of different brain areas in VWM processes. If parietal and frontal areas are genuinely dedicated to task requirements and top-down processes important for VWM maintenance, a design in which various stimulus categories and different task sets are used in a single experiment should show a clear differentiation in function between higher-level and lower-level sensory areas. For example, manipulating the type of attention (spatial versus object) or control demands (switching task instructions versus keeping them constant) could dissociate between the roles of different areas involved in different VWM tasks, while the use of clearly distinguishable stimuli (color versus motion, faces versus houses) should reveal the role of sensory areas itself. Magnetic stimulation of higher-tier and lower-tier brain regions during similar experiments could further corroborate the functional role of different brain regions in visual working memory representation.

In summary, both Riggall and Postle (2012) and Christophel et al. (2012) provide compelling evidence that the contents of VWM cannot be decoded from frontal cortex, but are stored in visually selective areas despite a lack of significant area-wide delay activity. In contrast, the role of parietal cortex in VWM is still unclear: it might have a process-related function similar to frontal cortex or it might store visual content as well. Although the exact role of the parietal cortex in VWM storage remains uncertain, these new insights are of great importance to our current knowledge of the neural dynamics of VWM and demand a further explication of the functional roles of different brain areas in VWM storage.