Decoding complex flow-field patterns in visual working memory
Introduction
The term working memory describes the human capability to retain perceived stimuli or internally generated information for later use or mental manipulation over a limited amount of time (Baddeley and Hitch, 1974, Curtis and D'Esposito, 2003, Jonides et al., 2008, Postle, 2006). Engagement in a working memory task typically leads to increased activity in regions of the dorsolateral prefrontal cortex (Courtney et al., 1997, McCarthy et al., 1994, Ungerleider et al., 1998; for a review see Wager and Smith, 2003), originally supporting the notion that prefrontal areas maintain information during working memory (Constantinidis and Procyk, 2004, Courtney, 2004, Funahashi, 2006, Goldman-Rakic, 1995).
Instead, recent studies have shown that in humans visual contents of working memory can be decoded from sensory brain regions of visual and parietal cortex, but not from higher-order dorsolateral prefrontal cortex (Christophel et al., 2012, Riggall and Postle, 2012, Emrich et al., 2013; see Postle et al., 2003). Multivariate decoding techniques (Cox and Savoy, 2003, Haxby et al., 2001, Haynes and Rees, 2005, Haynes and Rees, 2006, Haynes et al., 2007, Kamitani and Tong, 2006, Kriegeskorte et al., 2006, Soon et al., 2008) were used to show that specific visual features like orientation, color and motion direction are encoded during memory in the respectively specialized visual areas (Christophel et al., 2012, Harrison and Tong, 2009, Jerde et al., 2012, Riggall and Postle, 2012, Serences et al., 2009, Sneve et al., 2012). These studies were consistent with earlier work showing that memory-related activity within category-specific areas can differentiate between larger categories of natural stimuli like faces and scenes (Lewis-Peacock and Postle, 2008, Lewis-Peacock and Postle, 2012, Postle et al., 2003, Ranganath et al., 2004a, Ranganath et al., 2004b).
In a previous study we demonstrated that spatial color field stimuli can be decoded from early visual and parietal brain regions across working memory retention delays (Christophel et al., 2012). The stimuli had the advantage of being at the same time complex, but not suitable for semantic interpretation or chunking, thus maximizing the demands on sensory encoding. This raises the question whether such encoding in sensory and parietal cortex is also observable for other types of visuo-spatial pattern stimuli. In particular, we chose to investigate the short-term storage of flow-field patterns. These complex motion stimuli consisted of a manifold of motion directions and varied in the distribution of motion directions across the display (Fig. 1). Subjectively, the stimuli give the impression of dynamic, swirling water patterns and were not easily described in verbal terms.
Representations of much simpler forms of motion stimuli have been demonstrated previously during working memory (Emrich et al., 2013, Riggall and Postle, 2012), as well as during attended and unattended perception (Brouwer and van Ee, 2007, Hebart et al., 2012, Hong et al., 2012, Kamitani and Tong, 2006, Serences and Boynton, 2007a, Serences and Boynton, 2007b). We used multivariate searchlight decoding (Cox and Savoy, 2003, Haxby et al., 2001, Haynes and Rees, 2005, Haynes and Rees, 2006, Haynes et al., 2007, Kamitani and Tong, 2005, Kriegeskorte et al., 2006, Soon et al., 2008) to probe all areas of the human brain for information about the stimulus encoded in memory. We hypothesized that the storage of complex flow-field patterns extends beyond MT and includes other ‘motion sensitive’ areas (Culham et al., 2001). To test for memory signals independently of the stimulus presentation we used a retro-cue method in combination with a mask to suppress perceptual signals (Christophel et al., 2012, Harrison and Tong, 2009, Sperling, 1960). Please note that this approach typically thwarts attempts to investigate signals induced by mere perception of the stimulus.
Section snippets
Participants
Twenty healthy subjects (aged 20 to 35 years) with normal or corrected to normal vision participated in the experiment. The study was granted ethical approval by the local ethics committee and participants gave informed consent. Three subjects had to be excluded from the analysis due to excessive head movements in the MRI scanner. The residual sample consisted of ten females and seven males (all right-handed, mean age: 26.3, SEM ± 0.89).
Task and stimuli
We used complex flow-field stimuli that were based upon 4 × 4
Behavioral results
The amount of positive correlation of the target motion pattern was defined during training for each participant using a staircase procedure to enforce detailed encoding. This ‘difficulty’ of the similarity task varied across subjects between 0.675 and 0.85 (mean: 0.77 SEM ± 0.01) as measured in Fisher's z transformed correlations. Behavioral performance, as measured by the proportion of trials where a given subject correctly identified the similar target stimulus, was 0.67 on average (SEM ± 0.01).
Discussion
Whether we are driving on three-lane roundabouts during rush hour or rafting in the treacherous white-waters of the Grand Canyon, motion is rarely ever unidirectional but often follows coherent motion patterns of considerable complexity. To successfully navigate such environments, we have to be capable of representing these patterns quickly and retain these patterns of coherent flow over time. The present study investigated how complex patterned flow-fields are encoded in working memory. While
Author Contributions
T.B.C., and J.-D.H. designed research; T.B.C. performed research; T.B.C. analyzed data; T.B.C., and J.-D.H. wrote the paper.
Acknowledgments
This work was funded by the Bernstein Computational Neuroscience Program of the German Federal Ministry of Education and Research BMBF Grant 01GQ0411, the Excellence Initiative of the German Federal Ministry of Education and Research DFG Grants GSC86/1-2009 and DFG Grant HA 5336/1-1. The authors would like to thank Jakob Heinzle, Bianca van Kemenade and Vera Ludwig for helpful discussion.
Conflict of interest
The authors declare that there are no conflicts of interest.
References (88)
- et al.
Unified segmentation
Neuroimage
(2005) - et al.
Working memory
- et al.
Decoding successive computational stages of saliency processing
Curr. Biol.
(2011) - et al.
Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex
Neuroimage
(2003) - et al.
Visual motion and the human brain: what has neuroimaging told us?
Acta Psychol.
(2001) - et al.
Persistent activity in the prefrontal cortex during working memory
Trends Cogn. Sci.
(2003) - et al.
A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data
Neuroimage
(2005) - et al.
Perceiving time to collision activates the sensorimotor cortex
Curr. Biol.
(2005) Prefrontal cortex and working memory processes
Neuroscience
(2006)- et al.
Areas 3a, 3b, and 1 of human primary somatosensory cortex: 1. Microstructural organization and interindividual variability
Neuroimage
(1999)
Areas 3a, 3b, and 1 of human primary somatosensory cortex: 2. Spatial normalization to standard anatomical space
Neuroimage
Cellular basis of working memory
Neuron
Spatial maps in frontal and prefrontal cortex
NeuroImage
Parietal and superior frontal visuospatial maps activated by pointing and saccades
Neuroimage
Reading hidden intentions in the human brain
Curr. Biol.
Human visual and parietal cortex encode visual choices independent of motor plans
NeuroImage
Direction-selective patterns of activity in human visual cortex suggest common neural substrates for different types of motion
Neuropsychologia
Decoding seen and attended motion directions from activity in the human visual cortex
Curr. Biol.
A touching sight: SII/PV activation during the observation and experience of touch
Neuron
Robust smoothness estimation in statistical parametric maps using standardized residuals from the general linear model
Neuroimage
Decoding the internal focus of attention
Neuropsychologia
Multisensory spatial interactions: a window onto functional integration in the human brain
Trends Neurosci.
Separating processes within a trial in event-related functional MRI: I
Method Neuroimage
Working memory as an emergent property of the mind and brain
Neuroscience
Seeking the neural substrates of visual working memory storage
Cortex
Category-specific modulation of inferior temporal activity during working memory encoding and maintenance
Cogn. Brain Res.
Feature-based attentional modulations in the absence of direct visual stimulation
Neuron
Visual short-term memory: activity supporting encoding and maintenance in retinotopic visual cortex
Neuroimage
Dynamic coding for cognitive control in prefrontal cortex
Neuron
The retinotopy of visual spatial attention
Neuron
Tactile and visual motion direction processing in hMT+/V5
NeuroImage
Visual field maps in human cortex
Neuron
Visual and spatial working memory: from boxes to networks
Neurosci. Biobehav. Rev.
Multimodal representation of space in the posterior parietal cortex and its use in planning movements
Annu. Rev. Neurosci.
Attention, intention, and priority in the parietal lobe
Annu. Rev. Neurosci.
Neural synergy between kinetic vision and touch
Psychol. Sci.
Somatosensory activations during the observation of touch and a case of vision–touch synaesthesia
Brain
Visual cortex allows prediction of perceptual states during ambiguous structure-from-motion
J. Neurosci.
Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey
Eur. J. Neurosci.
Decoding the contents of visual short-term memory from human visual and parietal cortex
J. Neurosci.
The primate working memory networks
Cogn. Affect. Behav. Neurosci.
Neural systems for visual orienting and their relationships to spatial working memory
J. Cogn. Neurosci.
Attention and cognitive control as emergent properties of information representation in working memory
Cogn. Affect. Behav. Neurosci.
Transient and sustained activity in a distributed neural system for human working memory
Nature
Cited by (48)
Consequence of stroke for feature recall and binding in visual working memory
2021, Neurobiology of Learning and MemoryCross-modal involvement of the primary somatosensory cortex in visual working memory: A repetitive TMS study
2020, Neurobiology of Learning and MemoryDecoding stimulus identity in occipital, parietal and inferotemporal cortices during visual mental imagery
2020, CortexCitation Excerpt :The recruitment of parietal regions during visual mental imagery is in line with previous studies (Formisano et al., 2002; Ganis, Thompson, & Kosslyn, 2004; Knauff, Kassubek, Mulack, & Greenlee, 2000) which also highlighted a prominent role of the left hemisphere in imagery tasks (see Winlove et al., 2018). Recent studies showed parietal regions (i.e., SPL and aIPS) to be involved during mental imagery of different hand actions (Oosterhof, Tipper, & Downing, 2012) and in the encoding of the identity of artificial stimuli during visual working memory (Christophel, Hebart, & Haynes, 2012; 2014). According to Naughtin, Mattingley, and Dux (2014), premotor cortex can host distinct representations of both identity and spatial position of stimuli in a visual working memory task during the retention delay.