Abstract
A powerful force in human memory is the context in which memories are encoded (Tulving and Thomson, 1973). Several studies suggest that the reinstatement of neural encoding patterns is beneficial for memory retrieval (Manning et al., 2011; Staresina et al., 2012; Jafarpour et al., 2014). However, reinstatement of the original encoding context is not always helpful, for instance, when retrieving a memory in a different contextual situation (Smith and Vela, 2001). It is an open question whether such context-dependent memory effects can be captured by the reinstatement of neural patterns. We investigated this question by applying temporal and spatial pattern similarity analysis in MEG and intracranial EEG in a context-match paradigm. Items (words) were tagged by individual dynamic context stimuli (movies). The results show that beta oscillatory phase in visual regions and the parahippocampal cortex tracks the incidental reinstatement of individual context trajectories on a single-trial level. Crucially, memory benefitted from reinstatement when the encoding and retrieval contexts matched but suffered from reinstatement when the contexts did not match.
Introduction
Context is the hallmark of human episodic memory, often explaining why specific events are remembered in specific situations. For example, one might suddenly recall a recently watched movie while visiting a restaurant. The underlying reason might be that the background music of the restaurant overlaps with the soundtrack of the movie. In a situation lacking contextual overlap, retrieving this memory might fail, despite all mental efforts. These effects are explained by the encoding specificity principle (Tulving and Thomson, 1973), predicting that memory success varies as a function of neural encoding patterns being reinstated at retrieval (Polyn et al., 2005; Rugg et al., 2008). Episodic memory models incorporated this reinstatement idea, assuming that the hippocampus stores distributed cortical patterns that get reinstated on retrieval (Marr, 1971; Norman and O'Reilly, 2003). However, such reinstatement should only be beneficial if the cortical patterns representing the encoding context match those present at retrieval but detrimental when these contexts do not match. Here, we investigate this yet untested prediction of the encoding specificity principle by using oscillatory pattern similarity analysis in MEG and intracranial EEG (iEEG) data in a paradigm explicitly manipulating contextual overlap between encoding and retrieval (Fig. 1A).
Reinstatement of neural encoding patterns during retrieval can be revealed by pattern similarity analyses (Kriegeskorte et al., 2008), applied in memory studies using fMRI (Johnson et al., 2009; Staresina et al., 2012; Ritchey et al., 2013), MEG (Jafarpour et al., 2014), and iEEG (Manning et al., 2011; Yaffe et al., 2014). These studies provide evidence that contextual encoding patterns get reactivated during retrieval, positively correlating with memory. Wimber et al. (2012) showed incidental reactivation of contextual patterns without overt context memory. Incidental retrieval is necessarily driven by retrieval cues, provided primarily by the context. However, it remains unclear how reinstatement of cortical patterns interacts with memory in the case of context match/mismatch. We hypothesize that reinstatement of neural patterns, representing encoding context, aids retrieval when it matches the retrieval context but interferes with memory when encoding and retrieval contexts do not match. This hypothesis is illustrated in a simple memory model shown in Figure 1, B and C.
To test this hypothesis, we used unique word–movie pairs and manipulated the overlap between encoding and retrieval. Importantly, we used the inherent temporal characteristic of dynamic movies inducing reliable context memory effects (Smith and Manzano, 2010). We applied a novel approach assessing the similarity in temporal patterns during encoding and retrieval [temporal-pattern similarity (TPSim); Fig. 2B] to exploit the temporal resolution of MEG/iEEG and capture the reinstatement of the temporally dynamic movies. We validated this approach against a more classical spatial-pattern similarity (SPSim) approach (Fig. 2B) used in EEG/MEG (Garcia et al., 2013). Central to our hypothesis, we tested whether the context match/mismatch manipulation reversed the relation between neural reactivation and memory retrieval (Fig. 2C, left). To show the feasibility of TPSim and its sensitivity to capture item-specific temporal trajectories on a single-trial basis, we then tested whether TPSim is higher for repetitions of identical movies than for different movies (Fig. 2C, right).
Materials and Methods
Participants
Eighteen healthy volunteers (11 females; mean age, 23.4 years; range, 20–32 years) were recruited from the University of Konstanz and received course credits or monetary reward for participation. All participants had German as their native language and reported normal or corrected-to-normal vision and no history of neurological disease. All of the participants gave written informed consent before the experiment, which was approved by the ethics committee of the University of Konstanz. The behavioral data from the healthy participants and the MEG data during encoding has been published previously by Staudigl and Hanslmayr (2013).
Additionally, two male patients (25 and 27 years old) with a history of drug-resistant epilepsy were recruited from the Epilepsy Center, Department of Neurology, University of Munich. Both patients, who volunteered to participate in the study, had depth electrodes implanted for diagnostic reasons. The patients gave written informed consent. The study was approved by the ethics committee of the University of Munich. The data from the patients have not been published previously.
Procedure, design, and materials
The stimuli consisted of 360 German unrelated nouns and 360 movie scenes. The movie scenes were taken from a pool of movies (provided by the Landesfilmdienst Baden-Württemberg) and included scenes with diverse topics. All of the movies were in color and included moving features. Both words and movies were grouped into three lists each, with 120 words and movies, respectively. Within lists, words were matched according to initial letters. One word and one movie list were assigned to the context-match condition. In this condition, words were paired randomly with movies (match pairs) and presented during the study phase. In the test phase, the same word–movie pair was presented. A second word and a second movie list were assigned to the context-mismatch condition. Words were paired randomly with movies during the study phase (mismatch pairs). In the test phase, each word was paired with a different movie, chosen randomly from this list. The remaining words and movies served as new pairs during the test phase. Again, words and movies were paired randomly. Across participants, assignment of word and movie lists to conditions was counterbalanced. Pairing of word lists and movie lists was also counterbalanced across participants, such that each word list was paired equally often with each movie list. Each participant was exposed to all of the conditions (within-subjects design).
Figure 1A illustrates the experimental procedure. At study, match and mismatch pairs were presented for 3000 ms in random order with the constraint that no more than five pairs of the same type (match pairs, mismatch pairs) were presented in a row. Words were displayed in white letters surrounded by a black box superimposed over the center of the respective movie scenes. The participants were instructed to focus their attention on the word and to judge whether the first and last letter of the word were in alphabetic order or not. Within each word list, half of the words had their first and last letters in alphabetic order and the other half did not. This encoding task was chosen to ensure superficial (shallow) encoding, which is known to induce particularly strong context memory effects (Hanslmayr et al., 2009). Participants gave their responses manually by pressing the specified response button on the response panel. Assignment of left and right hand to the response (alphabetic order vs non-alphabetic order) was counterbalanced across participants. A fixation cross with variable duration (750–1250 ms) preceded each word–movie presentation. All of the participants were naive about the later memory test. After the study phase, participants performed a distracter task during which they had to count backward in steps of three from a three-figure number for 45 s. This was done to prevent participants from rehearsing the items and to ensure that the recognition task tapped long-term memory. After that, the instructions for the test phase were given, followed by a short practice run.
At test, match pairs, mismatch pairs, and new pairs were presented for 3000 ms, interleaved by the presentation of a fixation cross with variable duration (750–1250 ms). Presentation of pairs was randomized, with the constraint that no more than five pairs of the same type (match, mismatch, and new) were presented in a row. As in the study phase, words were displayed in white letters surrounded by a black box superimposed over the center of the movie scenes. Thereafter, a stimulus picture depicting the response options was shown that prompted the participants to indicate their confidence on whether the word was old or new by pressing the specified button on the response panel. The stimulus picture prompting the response was shown until the participants gave a response. Participants were instructed to focus their attention on the presented word and to rate their confidence of the word being old or new using a six-point scale ranging from “very sure old” (1) to “very sure new” (6). The assignments of the buttons was counterbalanced across participants (half responded with their left hand for the ratings 1, 2, 3 for ring, middle, and index fingers, respectively; the other half used the index, middle, and ring fingers of the right hand for these buttons).
MEG acquisition and preprocessing
MEG was recorded with a 148-channel whole-cortex magnetometer (MAGNES 2500 WH; 4D Neuroimaging) in a magnetically shielded room, while participants were in a supine position. Data were continuously recorded at a sampling rate of 678.17 Hz and bandwidth of 0.1–200 Hz. The participants' nasion, left and right ear canal, and head shape were digitized before each session with a Polhemus 3Space Fasttrack. Preprocessing of the data was done using the FieldTrip toolbox (Oostenveld et al., 2011). All data were epoched into single trials, with epochs ranging from 4000 ms before the presentation of the items to 4000 ms after item presentation. This epoch length was selected to allow for artifacts at the edges of the epochs arising during time–frequency transformation. All trials were cut to non-overlapping intervals starting 1000 ms before and ending 3000 ms after stimulus onset and were visually inspected for artifacts. The noisiest trials (i.e., the trials displaying the highest signal variance) were rejected. On average, 26.4 of 240 trials (SD of 10.4) were rejected per subject. Two of the 148 channels were excluded as a result of bad recording quality. Thereafter, trials were corrected for blinks, eye movements, and cardiac artifacts using independent component analysis.
The remaining trials were sorted according to the behavioral performance of each participant's confidence judgments during the recognition test phase. Trials including old items that were confidently judged as old (responses 1–3) constituted hits, and the remaining trials including old items were classified as misses. Trials including new items that were confidently judged as being new (responses 4–6) constituted correct rejections, and the remaining trials including new items were classified as false alarms.
Time–frequency analysis
Time–frequency analysis was applied to each trial at each MEG sensor using Morlet wavelets (width of 7 for lower frequencies 1–40 Hz; width of 5 for higher frequencies 40–90; for review, see Buzsáki and Wang, 2012), following the procedure described by Tallon-Baudry et al. (1997), as implemented in the FieldTrip toolbox. Time–frequency resolution was set to 0.5 Hz and 5 ms for lower frequencies and to 1 Hz and 1 ms for higher frequencies.
Model simulating the effect of context reactivation on memory
We applied a simplified memory model to predict the effect of context match and mismatch on memory performance while simulating different levels of memory reactivation (Fig. 1B). Each item (word) and context (movie) was assigned to an individual input pattern, simulating a neural pattern representing the item and context at encoding. To retrieve the encoded items, these individual patterns need to be reactivated. During the recognition test, the presented items and contexts provide two independent memory cues possibly reactivating encoded patterns. Thereby, the item itself provides direct access to the to-be-remembered item pattern (direct links). The context also provides access to the to-be-remembered item pattern but in a more indirect manner, because it needs to coactivate an item that was presented together with the context (associative links). The reactivated pattern is a sum of direct links and associated links. The only difference between the match and the mismatch condition is that, in the match condition, direct and associative links point to the same pattern, whereas in the mismatch condition, the direct links and associative links point to different patterns. Specifically, in the mismatch condition, direct links point to the encoded pattern, but the associative links reactivate a non-matching pattern, namely the pattern that was associated originally with the context during encoding.
The overlap between the input and the reactivated pattern, i.e., the correlation between the input pattern and the reactivated pattern, is assumed to govern memory performance. This assumption is based on previous findings and theoretical work providing evidence for the notion that memory performance relies on the reinstatement of the neural activity present during the encoding of the respective item (Johnson et al., 2009, 2015; Wimber et al., 2012). Two levels of reactivation (low vs high), implemented as low and high strengths of associative links, were compared. See Notes below for a link to the Matlab code used for the simulation.
TPSim
To analyze the similarity between temporal patterns in MEG activity between encoding and retrieval, single trials corresponding to the same item in the study and the test phase were correlated in the temporal domain. A correlation value (TPSim) for each item was obtained by correlating MEG activity across time points (from 0 to 3000 ms after stimulus onset) on each sensor separately (Fig. 2B). TPSim was analyzed in three aspects of the MEG data: (1) MEG unfiltered single trials, (2) phase, and (3) power of time–frequency transformed data. To compute TPSim in preprocessed, unfiltered MEG time series, MEG activity was correlated across time points in each trial, on each sensor, separately. A total of 2036 time bins were fed into the correlation of MEG unfiltered single trials. To analyze TPSim in the phase and power of the signal, time–frequency-transformed data were obtained using Morlet wavelets as implemented in FieldTrip. The resulting phase and power estimates were analyzed separately, such that TPSim was obtained for each item, sensor, and each frequency. The number of time points fed into the correlation depended on the time–frequency resolution (see above, Time–frequency analysis). Circular correlations were used for phase estimates. Correlation values were averaged across hits and misses in the match and mismatch condition, respectively.
To obtain off-diagonal values in the correlation matrices (see Figs. 4B, 5A, 6A, 7B,E), TPSim was computed for single trials not corresponding to the same items. To account for different trial numbers in conditions, a random sample of trials from the condition with the larger number of trials was drawn, and TPSim was computed. This was repeated 100 times, and TPSim values were averaged.
For significant TPSim effects found in time–frequency data, cross-correlations between encoding and retrieval activity were computed. The cross-correlations reveal whether TPSim effects are attributable to a phase lag between encoding and retrieval activity.
SPSim
To analyze the similarity between spatial patterns in MEG activity between encoding and retrieval, single trials corresponding to the same item in the study and the test phase were correlated in the spatial domain (i.e., across sensors), such that a correlation value (SPSim) for each item was obtained by correlating MEG activity across sensors for each time point, separately (Fig. 2B). A total of 146 sensors were fed into the correlation analyses. Analogous to the TPSim analyses, SPSim was analyzed in (1) MEG unfiltered single trials, (2) phase, and (3) power of time–frequency-transformed data. To compute SPSim in preprocessed, unfiltered MEG time series, MEG activity was correlated across sensors in each trial for each time point, separately. To analyze SPSim in the phase and power of the signal, time–frequency-transformed data were obtained using Morlet wavelets as implemented in FieldTrip. The resulting phase and power estimates were analyzed separately, such that SPSim was obtained for each item, time bin, and each frequency. Circular correlations were used for phase estimates. Correlation values were then averaged across time points, and trials were averaged across hits and misses in the match and mismatch conditions, respectively.
To obtain off-diagonal values in the correlation matrices (see Figs. 4F, 8B,C), SPSim was computed for single trials not corresponding to the same items. To account for different trial numbers in conditions, a random sample of trials from the condition with the larger number of trials was drawn, and SPSim was computed. This was repeated 100 times, and SPSim values were averaged.
For significant SPSim effects found in time–frequency data, cross-correlations between encoding and retrieval activity was computed. The cross-correlations reveal whether SPSim effects are attributable to a phase lag between encoding and retrieval activity.
Statistics
To correct for multiple comparisons, randomization tests were applied as follows.
Multiple comparisons (pcorr and pbin).
In a first step (pcorr), the TPSim/SPSim effects were subjected to a randomization procedure (Hanslmayr et al., 2012b). In this procedure, proposed by Blair and Karniski (1993), t tests were conducted to investigate how many MEG sensors show significant differences in a given contrast (p < 0.05, two-tailed). Thereafter, 5000 randomization runs were performed in which the conditions were swapped randomly across participants. This procedure produces a distribution of the number of sensors exceeding a certain statistical threshold under the null hypothesis (no systematic difference between conditions) and evaluates whether a given number of sensors exhibiting a significant difference in a given contrast at this threshold can be expected by chance. If the p value of this randomization test is smaller than 0.05, <5% of the permutation runs exhibited equal or more electrode pairs with a significant difference. A second step (pbin) of statistical evaluation was applied to time–frequency-transformed data only, because the additional frequency domain increases multiple comparisons. To control for this, the number of connected frequency bins exhibiting significant differences was compared with a distribution of random differences producing connected bins in a given contrast. The distribution was generated by computing 2000 TPSim/SPSim values while shuffling trials across conditions for each frequency bin for power and phase, respectively (note that we did not shuffle data bins along the time or the frequency axis, i.e., dependencies across adjacent frequency bins remained untouched in the shuffled distribution). In a given contrast, connected bins in the frequency domain were considered to be significant if the values exceeded 97.5% (this p level was used instead of 95% because two comparisons were made for power and phase, separately) of the values in the distribution. In the case of time–frequency-transformed data, a given contrast was only considered significant if pcorr and pbin were <0.05.
Noise level (pnoise).
To further validate the statistical significance of the TPSim/SPSim effects, we tested whether the observed TPSim/SPSim values in each condition were significantly different from zero (i.e., noise level). To this end, 500 randomization runs were performed in which the order of retrieval trials was shuffled within each condition per subject, thus producing a distribution of TPSim/SPSim values for each subject based on shuffled data within each condition.
In other words, we compared the observed TPSim/SPSim values of word–movie pairs in the match condition with a random distribution of TPSim/SPSim values in that condition. For instance, for the match–hit condition, the observed TPSim/SPSim values were computed by correlating activity of corresponding word–movie pairs during encoding and retrieval, that is, correlating encoding activity during word[a]–movie[a] presentation with retrieval activity during word[a]–movie[a] presentation (Fig. 2C). The random distribution of TPSim/SPSim values was produced by correlating encoding activity during word[a]–movie[a] presentation with retrieval activity drawn randomly from all the match–hit pairs (e.g., word[b] and movie [b]). The same procedure was run for match misses, that is, TPSim/SPSim values were averaged separately for hits and misses. Five hundred such randomization runs were conducted. Likewise, observed TPSim/SPSim values of word–movie pairs in the mismatch condition (e.g., correlating encoding activity during word[e]–movie[e] presentation with retrieval activity during word[e]–movie[f] presentation) were compared with a random distribution of TPSim/SPSim values by correlating encoding activity during word[f]–movie[f] presentation with retrieval activity drawn randomly from all the mismatch–hit pairs (e.g., word[f] and movie[g]). This was done for all the items in the mismatch condition, and TPSim/SPSim values were averaged, separately for hits and misses. Five hundred such randomization runs were conducted.
Then, one TPSim/SPSim value was drawn randomly from the distribution of 500 random values for each subject and condition, separately for hits and misses, and the grand average was computed. Ten thousand of such pseudo grand averages were computed and compared with the observed TPSim/SPSim grand average. The observed TPSim/SPSim for hits and misses in the match and mismatch conditions were considered to be significant if they exceeded 95% of the values in the respective pseudo grand average distribution. This approach ensured that the computed TPSim/SPSim in a given condition was meaningful.
Source-level analyses
To identify TPSim in source space, a virtual electrode approach applying linearly constrained minimum variance spatial filtering [LCMV (Linearly Constrained Minimum Variance) Beamformer (Van Veen et al., 1997); as implemented in FieldTrip] to the sensor-level data was used. The individual structural MR images were aligned into the MEG coordinate system via NUT-MEG (Neurodynamic Utility Toolbox for Magnetoencephalography; Dalal et al., 2004), using head shapes acquired for each individual. A realistic single-shell brain model (Nolte, 2003) was constructed for each participant based on the individual structural MRI (available for 16 of 18 participants; for the remaining two participants, an affine transformation of an MNI template brain was used; Montreal Neurological Institute). Source estimates were interpolated onto the individual anatomical images. The LCMV beam former, which computes a spatial filter from the lead field of the source and the data covariance matrix of the averaged single trials at the sensor level, was used to obtain cortical source time series. The raw data were projected into source space by multiplying it with the spatial accordant filters, resulting in 1092 virtual electrodes covering the volume. TPSim for contrasts of interest was computed on the virtual electrodes. Time–frequency transformation was done using Morlet (for details, see above). Source statistics (dependent samples t tests, α = 0.05, no cluster statistics applied) were computed using FieldTrip. The resulting t map was subsequently normalized to a standard MNI brain for illustrative purposes. Significant clusters of connected voxels including a minimum of 250 voxels in cortical areas were reported.
Intracranial data
Two male patients with a history of drug-resistant epilepsy participated in the experiment and were included in the study to validate the results of the MEG source-level analyses. The patients had depth electrodes implanted for diagnostic reasons. Recordings were performed at the Epilepsy Center, Department of Neurology, University of Munich. Antiepileptic medication was discontinued before the recordings. The patients gave written informed consent. The procedure and design of the study were identical to the MEG procedure and design (see above), with two exceptions. First, only half of the material was used: 120 word–movie pairs (60 match, 60 mismatch) were shown during the study phase, and 180 pairs (120 old plus 60 new) were presented at test. This was done to compensate for inferior memory performance in a clinical setting. Second, a slightly different encoding task was used. The patients were instructed to judge whether the word had one or two syllables. Within each word list, half of the words had one syllable and the other half had two syllables. This was done to compensate for slower cognitive processing in a clinical setting. The encoding task is similar to the one used in healthy subjects in that it ensures superficial (shallow) encoding, which is known to induce particularly strong context memory effects (Smith and Vela, 2001). Patient 1 had six depth electrodes implanted, covering left temporal and parietal regions. Patient 2 had 12 depth electrodes implanted, covering the left temporal, central, and frontal regions. The locations of the electrodes were determined using coregistered preoperative MRIs and postoperative computed tomographies. Electrode locations were converted into MNI coordinates. iEEG was recorded from Spencer depth electrodes (Ad-Tech Medical Instrument) with 4–12 contacts each, 5 mm apart. Data were recorded using XLTEK Neuroworks software (Natus Medical) and an XLTEK EMU128FS amplifier, with a sampling rate set to 1000 Hz (patient 1) and 500 Hz (patient 2), with voltages referenced to a parietal electrode site. Data were re-referenced offline to the neighboring contact (bipolar reference) of each contact. Data were epoched into single trials ranging from 0 to 3000 ms after the onset of presentation of a word–movie pair, separately for the study and test phases. All trials were inspected visually for artifacts (e.g., epileptoform spikes), and contaminated trials were excluded from analyses. The encoding trials were sorted according to each participant's confidence judgments during the test phase. Trials including old items that were judged as old (responses 1–3) constituted hits; the remaining trials including old items were classified as misses. Beta (26.5–32 Hz) phase TPSim was computed as described above. We report data from two electrodes in patient 1 [MNI coordinates: (−22, −14, −22), Brodmann area (BA) 28; (−40, −20, −15), BA36] and one electrode in patient 2 [(−27, −35, −12), BA36] located in the left parahippocampal gyrus. The location of the electrodes is displayed in Figure 5B. Statistical significance was identified by a randomization procedure. After concatenating TPSim values across single trials and participants, 5000 randomization runs were performed in which the order of retrieval trials was shuffled within each condition. TPSim was computed on the shuffled trials such that a distribution of random correlations for the contrasts of interest was produced. The TPSim values for the interaction and post hoc t tests were compared with this distribution and considered to be significant if the values exceeded 95% of the values in the distribution. This approach ensured that the computed TPSim in a given contrast was meaningful.
Results
Behavioral results
We recorded MEG from 18 healthy subjects and iEEG from two epileptic patients participating in a context memory paradigm (Fig. 1A). At encoding, words were presented superimposed on movie clips. Later, word–movie pairs were presented again intermixed with new word–movie pairs in a surprise recognition test. At test, half of the words were paired with the same movie as during encoding (context-match condition), whereas the other words were paired with a different, but also old, movie (context-mismatch condition). A significantly higher hit rate in the context-match than in the context-mismatch condition (68.6% vs 60.5%, respectively; t(17) = 4.76, p = 0.00018 in healthy subjects; patients, 57.5 vs 50%) indicated a reliable context-dependent memory effect (Fig. 3). Taking into account the false-alarm rates (33.3% in healthy subjects and 21.7% in patients), the memory performance can be considered to be well above chance level.
Experimental design and behavioral data. A, At study, items were presented superimposed over movie scenes while the participants performed a shallow encoding task. At test, each word from the study phase was presented, intermixed with new words. Half of the old words were superimposed over the same movie scenes (match); the other half were superimposed over rearranged movie scenes (mismatch). Participants indicated their confidence on whether the item was old or new. B, A simplified memory model illustrating the effect of reactivation on memory performance in context-match and context-mismatch conditions. Each item (word) and context (movie) was assigned an individual pattern. Input patterns are associated with reactivated patterns via direct and associative links. Two levels of reactivation implemented as low and high strengths of associative links were compared. Memory performance is defined as the overlap of input patterns and reactivated patterns, measured by correlation. In the match condition, direct and associative links reactivate identical patterns. In the mismatch condition, associative links point to nonmatching patterns. The effects of context reactivation were simulated by varying the strength of the associative links only (i.e., the direct links were kept constant). C, Results of the model simulation. In the match condition, the simulated memory performance was higher for high levels of reactivation than for low levels of reactivation. In the mismatch condition, the opposite pattern was found, in which the simulated memory performance was lower for high levels of reactivation compared with low levels of reactivation.
Hypotheses and pattern similarity analyses. A, Presumed reinstatement of context during retrieval leads to hits (“old”) in the match condition but produces misses (“new”) in the mismatch condition. B, Pattern similarity analyses. Top row, TPSim, assessing the similarity between temporal patterns during encoding and retrieval. Middle row, SPSim, assessing the similarity between spatial patterns during encoding and retrieval. Bottom row, The correlation of encoding and retrieval activity (for both TPSim and SPSim) is high if the original context is reinstated during retrieval. C, Hypothesized TPSim/SPSim effects attributable to memory reinstatement (left and identical visual stimulation; right). Cells in the correlation matrix symbolizes simulated correlations of single-trial activity recorded during study and test. The diagonal represents item-repetition trials. Note that, in the match condition, corresponding word–movie pairs are correlated during encoding and retrieval (e.g., word[a]–movie[a] with word[a]–movie[a], abbreviated as Wa-Ma with Wa-Ma), whereas in the mismatch condition, word–movie pairs were rearranged during retrieval such that non-identical word–movie pairs were correlated (e.g., Wf-Mf with Wf-Mg). Off-diagonal cells (e.g., the correlation between Wa-Ma and Wb-Wb) within each condition were used for statistical testing separately for hits and misses (in Materials and Methods, see pnoise). Correlation values in gray areas were not analyzed.
Behavioral data. Healthy subjects showing a significantly higher hit rate in the context-match condition than the context-mismatch condition (left). Data represented as means ± SEMs. Figure reproduced with permission from Staudigl and Hanslmayr (2013). Behavioral data in patients showing higher hit rates in the context-match condition compared with the context-mismatch condition (right).
Simulated effects of context reactivation on memory
The results of the simplified memory model showed a context-dependent effect of the level of reactivation on memory performance. Figure 1C depicts the simulation results for 50 word–movie pairs. In the match condition, the simulated memory performance was higher for high levels of reactivation (high strength of associative links) than for low levels of reactivation (low strength of associative links). In the mismatch condition, the opposite pattern was found. The simulated memory performance was lower for high levels of reactivation than for low levels of reactivation. Note that the goal of this simple simulation was to illustrate on a qualitative level how context reactivation interacts with memory performance in matching and nonmatching context situations at retrieval. Based on these results, we hypothesized that hits and misses should be associated with opposite levels of reactivation in the MEG/iEEG data. In the match condition, hits (reflecting good memory performance) should show higher levels of reactivation than misses (reflecting bad memory performance), whereas in the mismatch condition, hits should show lower levels of reactivation than misses. Henceforth, interaction effects [match (hits − misses) > mismatch (hits − misses)] in pattern similarity were analyzed as an index of memory reinstatement.
The relationship between MEG pattern similarity and memory is reversed by context overlap
Memory reinstatement was analyzed by computing TPSim in MEG single trials. Following our hypothesis, interaction effects [match (hits − misses) > mismatch (hits − misses)] were used as an index of incidental memory reinstatement. No interaction effects were found when analyzing TPSim on MEG unfiltered single trials.
After time-frequency transforming the data, TPSim for phase and power was analyzed separately. A significant interaction effect in TPSim was found for beta phase (26–32.5 Hz; pcorr = 0.0298; pbin = 0.0005; Fig. 4A), with a frontal topography (Fig. 4B). Post hoc t tests revealed that hits showed significantly higher TPSim than misses (t(17) = 5.90; p = 0.00002) in the match condition (Fig. 4B). Note that this difference is necessarily driven by memory because the visual input during encoding and retrieval is identical for both hits and misses. In the mismatch condition, TPSim was significantly higher for misses than for hits (t(17) = 3.74, p = 0.0016). Again, this difference is necessarily driven by memory and not by visual input because the context is equally non-identical for both hits and misses, the item is equally identical. Notably, the TPSim for match hits and mismatch misses was significantly greater than zero (pnoise = 0.0145, pnoise = 0.0331, respectively). Unexpectedly, TPSim for match misses was significantly less than zero (pnoise = 0.0003). TPSim for the beta phase (26–32.5 Hz) was not significantly different from noise level for the mismatch hits (pnoise = 0.3992). Cross-correlation analysis revealed that TPSim during memory reactivation (match hits; mismatch misses) was highest for zero phase lags, whereas the negative TPSim value found for match misses shows a 180° phase lag between encoding and retrieval activity (Fig. 4C). This latter result suggests a discrepancy in timing between the beta phases at encoding and retrieval for match misses. Source analysis (Fig. 4D) indicated that the memory TPSim effects were localized to occipital sources (approximately BA18/19) and mediotemporal lobe regions (left parahippocampal gyrus, approximately BA28; left uncus, approximately BA38), which is in line with fMRI studies showing these regions to be involved in reinstatement of visual information (Wheeler et al., 2000) and scenes (Staresina et al., 2012). Additional sources were found in the middle and inferior frontal cortices (approximately BA10 and BA 45), which is in line with a previous fMRI study investigating encoding-retrieval similarity using pictures (Ritchey et al., 2013). No TPSim interaction effects for low-frequency power survived correction for multiple comparisons, and no interaction effects were found in the 40–90 Hz range for either phase or power.
Memory reinstatement as indexed by TPSim interaction effects. A, TPSim interaction effects for lower-frequency phases. Number of significant sensors (gray) and p level (black) are plotted. Significant interaction effects are highlighted in yellow. B, Beta phase (26–32.5 Hz) interaction effect. Left, The correlation matrix indicates the pattern similarity across conditions. Right, Bars represent means ± SEMs over significant sensors, highlighted in the topography, for each condition, respectively. C, Cross-correlations of beta phases over significant sensors show the highest correlation at a time lag of 0 ms for match hits and mismatch misses. For match misses and mismatch hits, correlation values are highest at a time lag of 15–20 ms, corresponding to ∼180° phase shift. D, Beta phase interaction effect on virtual electrodes, including left frontal, left occipital, right parietal, and left medial temporal lobe regions. Highlighted areas indicate significant voxels (dependent samples t tests, α = 0.05). E, SPSim interaction effects for lower-frequency phases. Number of significant time points (gray) and p level (black) are plotted. Significant interaction effects are highlighted in yellow. F, Beta phase (34.5–36.5 Hz) interaction effect. Left, The correlation matrix indicates the pattern similarity across conditions. Middle, Bars represent means ± SEMs over significant time points for each condition, respectively. Right, Cross-correlations of beta phases over significant time points show the highest correlation at a time lag of 0 ms for match hits and mismatch misses. For match misses and mismatch hits, correlation values are highest at a time lag of 15 ms, corresponding to ∼180° phase shift.
To compare the effects assessed by TPSim to a more classical pattern similarity approach, SPSim was assessed by correlating the spatial patterns in MEG single trials recorded at study and test (Fig. 2B). No interaction effects [match (hits − misses) > mismatch (hits − misses)] were found when analyzing SPSim on MEG unfiltered single trials. Similar to the TPSim results, a significant effect in SPSim was found for beta phase (34.5–36.5 Hz; pcorr = 0.0320; pbin = 0.0070; Fig. 4E). Post hoc t tests revealed that hits showed significantly higher SPSim than misses (t(17) = 4.6390, p = 0.0003; Fig. 4F) in the match condition, whereas the opposite pattern emerged in the mismatch condition (t(17) = 5.3328, p = 0.00005). The SPSim values for match hits (pnoise = 0.0004), match misses (pnoise = 0.0017), and mismatch misses (pnoise = 0.0001) were significantly different from zero. SPSim values for mismatch hits were not significantly different from zero (pnoise = 0.1585). Cross-correlation analysis revealed that TPSim during memory reactivation (match hits and mismatch misses) was highest for zero phase lags, whereas the negative TPSim value found for match misses shows a 180° phase lag between encoding and retrieval activity (Fig. 4F). No TPSim interaction effects for low-frequency power survived correction for multiple comparisons, and no interaction effects were found in the 40–90 Hz range for either phase or power.
Temporal pattern similarity effects in the parahippocampal gyrus in iEEG
To verify the results obtained in MEG, especially with regards to source localization for the deeper sources in the medial temporal lobe, we analyzed TPSim in data recorded directly from the parahippocampal gyrus in two patients (Fig. 5B). The results confirm our MEG source data in showing a significant interaction effect (p = 0.0012; Fig. 5A) and cross-correlations indicating that TPSim during memory reactivation (match hits and mismatch misses) was highest for zero phase lags, whereas the negative TPSim values found for match misses and mismatch hits show a 180° phase lag between encoding and retrieval activity. Hits showed significantly higher TPSim than misses (p = 0.0104) in the match condition. In the mismatch condition, TPSim was significantly higher for misses than for hits (p = 0.0194). Thus, our MEG and intracranial results add new evidence to previous findings showing visual scene processing (Epstein and Kanwisher, 1998) and visual context reinstatement (Staresina et al., 2012) in the parahippocampal gyrus.
Intracranial data. A, TPSim interaction effect (26–32.5 Hz) in intracranial data. Left, The correlation matrix indicates the pattern similarity across conditions. Middle, Bars represent means over intracranial electrodes for each condition, respectively. Right, Cross-correlations of beta phases over significant sensors show highest correlation at a time lag of 0 ms for match hits and mismatch misses. For match misses and mismatch hits, correlation values are highest at a time lag of 15–20 ms, corresponding to ∼180° phase shift. B, Recording sites in the parahippocampal gyrus of the two patients.
Pattern similarity analysis detects single repetitions of individual movies independent of memory
TPSim was assessed by correlating the temporal patterns in MEG single trials recorded at study and test (Fig. 2B). A significant main effect (match > mismatch) of TPSim was found in the MEG unfiltered single trials (pcorr = 0.0446; Fig. 6A). Within this contrast, TPSim in the match condition was significantly greater than zero (pnoise = 0.0001 for hits; pnoise = 0.0005 for misses). TPSim in the MEG unfiltered single trials was not significantly greater than noise level in the mismatch condition (hits, pnoise = 0.2430; misses, pnoise = 0.5834). Source analyses showed that the TPSim effect (match > mismatch) was localized to visual, parietal, and temporal areas, covering the ventral visual stream (Fig. 6B).
Stimulus repetition effects (match > mismatch) in MEG raw traces as indexed by TPSim. A, The correlation matrix indicates the pattern similarity across conditions. Bars represent means over significant sensors, highlighted in the topography, for each condition, respectively. Data represented as means ± SEMs. B, Main effect of TPSim in raw MEG traces on virtual electrodes cover visual, parietal, and temporal areas. Highlighted areas indicate significant voxels (dependent samples t tests, α = 0.05).
To assess the frequency characteristic of this effect, the data were time–frequency transformed, and TPSim for phase and power was analyzed separately. Theta phase (5–7.5 Hz) exhibited significantly higher TPSim in the context-match condition than in the context-mismatch condition (pcorr = 0.0036; pbin = 0.0005; Fig. 7A). Again, TPSim for the match condition was significantly greater than zero (pnoise = 0.0001 for hits and misses; Fig. 7B). TPSim for the theta phase (5–7.5 Hz) was not significantly greater than noise level in the mismatch condition (hits, pnoise = 0.7024; misses, pnoise = 0.9737). Cross-correlations revealed that TPSim was highest at zero phase lag (Fig. 7C). Source analyses revealed that the theta effect was localized in occipital and temporal areas (Fig. 7D), which resembles the source localization of TPSim analyses on the unfiltered data (Fig. 6B). A second main effect (match > mismatch) was found for the beta phase (29.5–34 Hz; pcorr = 0.0046; pbin = 0.005). TPSim for the match condition was significantly greater than zero (pnoise = 0.0498 for hits; pnoise = 0.0001 for misses; Fig. 7E). TPSim for the beta phase (29.5–34 Hz) in the mismatch condition was significantly less than zero for misses (pnoise = 0.0108) but not for hits (pnoise = 0.1917). Cross-correlations revealed that TPSim was highest at zero phase lag in the match condition (Fig. 7F). In the mismatch condition, cross-correlations was highest at a lag of 15–20 ms, corresponding to a phase shift of ∼180° in the beta frequency. Source analyses revealed that the beta effect was localized in right occipital and temporal areas (Fig. 7G). No TPSim main effects (match > mismatch) for low-frequency power survived correction for multiple comparisons. No significant main effects were found in the 40–90 Hz range for either phase or power. Together, these results show that TPSim successfully captures the temporal trajectories of dynamic visual stimuli, localized to brain regions processing such visual stimuli.
Stimulus repetition effects (match > mismatch) in low-frequency phases indexed by TPSim. A, TPSim main effects for lower-frequency phases. Number of significant sensors (gray) and p level (black). Significant main effects are highlighted in yellow. B, Theta phase (5–7.5 Hz) main effect. The correlation matrix indicates the pattern similarity across conditions. Bars represent means ± SEMs over significant sensors, highlighted in the topography, for each condition, respectively. C, Cross-correlation between encoding and retrieval theta phases. Phases averaged across 5–7.5 Hz and sensors showing a significant TPSim main effect. Note that the highest correlation values are found around time lag of 0. D, Theta-phase main effect on virtual electrodes covering visual, parietal, and temporal areas. Highlighted areas indicate significant voxels (dependent samples t tests, α = 0.05). E, Beta-phase (29.5–34 Hz) main effect. The correlation matrix indicates the pattern similarity across conditions. Bars represent means ± SEMs over significant sensors, highlighted in the topography, for each condition, respectively. F, Cross-correlation between encoding and retrieval beta phases. Phases averaged across 29.5–34 Hz and sensors showing a significant TPSim main effect. Note that the highest correlation values are found around time lag of 0 for match hits and misses. For mismatch hits and misses, correlation values are highest at a time lag of 15 ms, corresponding to ∼180° phase shift. G, Beta-phase main effect on virtual electrodes covering visual, parietal, and temporal areas. Highlighted areas indicate significant voxels (dependent samples t tests, α = 0.05). Data represented as means across participants.
As for the above analysis, we compared the effects assessed by TPSim with SPSim. No main effects (match > mismatch) were found when analyzing SPSim on MEG unfiltered single trials. A significant main effect (match > mismatch) of SPSim was found in the theta phase (6–7.5 Hz; pcorr = 0.009; pbin = 0.012; Fig. 8A). Within this contrast, SPSim in the match condition was significantly greater than zero (pnoise = 0.0001 for hits and misses; Fig. 8B). SPSim for the theta phase was not significantly greater than noise level in the mismatch condition (hits, pnoise = 0.3893; misses, pnoise = 0.7307). Again, cross-correlations revealed that SPSim was highest at zero phase lag (Fig. 8B). A second main effect (match > mismatch) was found for the beta phase (34.5–36.5 Hz; pcorr = 0.0174; pbin = 0.001). SPSim was significantly greater than zero for the match condition (pnoise = 0.0256 for hits; pnoise = 0.0009 for misses; Fig. 8C) and significantly less than zero for the mismatch condition (pnoise = 0.0174 for hits; pnoise = 0.0001 for misses; Fig. 8C). Cross-correlations revealed that TPSim was highest at zero phase lag in the match condition (Fig. 8C). In the mismatch condition, cross-correlations were highest at a lag of 15–20 ms, corresponding to a phase shift of ∼180° in the beta frequency. No significant main effects were found for low-frequency power. No significant main effects were found in the 40–90 Hz range for either phase or power.
Stimulus repetition effects (match > mismatch) in low-frequency phases indexed by SPSim. A, SPSim main effects for lower-frequency phases. Number of significant time points (gray) and p level (black) are plotted. Significant main effects are highlighted in yellow. B, Theta-phase (6–7.5 Hz) main effect. Left, The correlation matrix indicates the pattern similarity across conditions. Middle, Bars represent means ± SEMs over significant time points for each condition, respectively. Right, Cross-correlation between encoding and retrieval theta phases. Phases averaged across 6–7.5 Hz and time bins showing a significant SPSim effect. The highest correlation values are found around time lag of 0. C, Beta-phase (35.5–36.5 Hz) main effect. Left, The correlation matrix indicates the pattern similarity across conditions. Middle, Bars represent means ± SEMs over significant time points for each condition, respectively. Right, Cross-correlation between encoding and retrieval beta phases. Phases averaged across 35.5–36.5 Hz and time bins showing a significant SPSim effect. The highest correlation values are found around time lag of 0 for match hits and misses. For mismatch hits and misses, correlation values are highest at a time lag of 15 ms, corresponding to ∼180° phase shift. Data are represented as means across participants.
Together, these results show that TPSim and SPSim reveal highly similar results and confirm the high sensitivity of pattern similarity analysis to track identical visual input, even if the stimulus is only repeated once.
Discussion
The present results demonstrate that temporal patterns in human brain activity track the encoding and reinstatement of contextual memory trajectories. Importantly, we used dynamic movie stimuli, making use of their inherent temporal characteristic and inducing reliable context memory effects (Smith and Manzano, 2010; Staudigl and Hanslmayr, 2013). To capture these temporal trajectories on the neural level, we analyzed the similarity between temporal patterns of brain signals (TPSim). Confirming our hypothesis and the modeling results, we show an interaction between reinstatement of temporal and spatial encoding patterns and context overlap. Specifically, the results show that memory benefits from the reinstatement of encoding patterns only when the context between encoding and retrieval is held constant, whereas reinstatement is detrimental when the context at retrieval does not match the encoding context. This interaction is necessarily driven by memory and cannot be explained by trivial effects, because the sensory input differed drastically between the match and mismatch conditions and was modulated by memory success (hits vs miss). Especially the finding of significant pattern reinstatement during mismatch misses, being of similar magnitude as pattern reinstatement during match hits (Fig. 4B), rules out that these effects were driven by similar sensory input. Therefore, these results are a first direct proof of the encoding specificity principle (Tulving and Thomson, 1973) proposed several decades ago and go beyond previous studies investigating memory reinstatement, which so far did not directly manipulate context overlap between encoding and retrieval.
The memory reinstatement effects observed in this study arguably reflect incidental context reinstatement as opposed to conscious retrieval of the original context information for several reasons. First, subjects were not required to explicitly recall the context (movie) information. In fact, subjects were instructed to ignore the movies in the background and to just focus on the word. Second, a shallow encoding instruction was used (see Materials and Methods) that boosts incidental context memory effects (Smith and Vela, 2001) but leads to very poor recollection or explicit source memory (Rugg et al., 1998). Third, memory reinstatement was found for mismatch miss items, for which the subjects had no explicit memory. Because the background movie in the mismatch condition was different from the movie during encoding, the item (word) is the only cue that the subjects could use to retrieve the context information, which seems impossible when the subjects do not recognize the item. Therefore, in line with a previous study (Wimber et al., 2012), the reinstatement effects reported here most likely reflect the automatic, incidental reinstatement of encoding context that boosts memory when encoding-retrieval contexts match but is detrimental when they do not overlap.
Our approach of using TPSim, as opposed to SPSim, was validated in showing that temporal phase patterns in two distinct frequency bands, the theta and beta bands, in visual processing regions tracked the repetition of single items. This effect was replicated using traditional SPSim approaches (Garcia et al., 2013). The fact that beta phase tracked similarity of the visual stimulus itself and memory reactivation suggests strongly that this frequency band plays an important role in both the coding of information in the sensory and memory domains. This would be an interesting explanation for why beta oscillations show a similar behavior (i.e., power deceases) during encoding and retrieval of episodic memories (Hanslmayr et al., 2012a, 2014). A likely neurophysiological mechanism behind those results could be phase coding, i.e., information coding reflected in the timing of neural spikes with respect to the phase of the local field potential. For instance, the memory reinstatement effect in the beta phase is consistent with previous work showing that items in working memory are coded in the beta phase (Siegel et al., 2009). These results are also consistent with a recent account suggesting that oscillations in the alpha/beta range carry content-specific information in the episodic memory system (Hanslmayr et al., 2012a). In contrast to beta band, the theta band tracked perceptual similarity only but not memory reactivation. In general, this finding is in line with that of Montemurro et al. (2008) who found sensory information coding to be dependent on low-frequency phases. These effects were found in the primary visual cortex while monkeys were presented with movies. The results from our source analyses provide a similar picture of the coding of information presented in form of movies.
In general, context-memory effects are difficult to evoke reliably, but our own work (Staudigl and Hanslmayr, 2013) and recent behavioral work (Smith and Manzano, 2010) suggest that using movies as contexts yield very reliable effects. We suggest that the temporal trajectory induced by these movies triggers the formation of context-dependent episodic memories and that the match between reinstated trajectories and trajectories present at retrieval is underlying context-memory effects. To compare the TPSim results with a more commonly used method, we also applied SPSim to the MEG data. Although TPSim and SPSim analyses produced mostly overlapping results, analyzing pattern similarity in the temporal domain has several practical advantages compared with spatial pattern analyses. TPSim uncovers the spatial origin of neural reinstatement, e.g., in source space, without the necessity of applying additional methods, e.g., search light approaches. Moreover, if spatial coverage is limited, as it is for example often the case in intracranial recordings, TPSim is theoretically capable of tracking reinstatement within just one recording site.
To summarize, the results of this study demonstrate that the reinstatement of neural encoding patterns is not always beneficial for memory retrieval, as previous studies suggest. Manipulating the overlap between encoding and retrieval contexts, we here show that memory benefits of the reinstatement of neural encoding patterns only when retrieval and encoding contexts match, whereas neural reinstatement is detrimental when there is a discrepancy between encoding and retrieval contexts. Furthermore, our results also show that both temporal and spatial patterns of oscillatory brain activity in the beta phase track such incidental reinstatement of dynamic episodic memory trajectories.
Notes
Supplemental material for this article is available at http://psychologie.uni-konstanz.de/cognition-and-oscillations-lab/files/simulation. MATLAB code was used for the simulation. This material has not been peer reviewed.
Footnotes
This work was supported by German Research Foundation Grant HA 5622/1-1 and the Zukunftskolleg at the University of Konstanz. We thank all of the participants, in particular the patients, for taking part in the study and the technical staff at the Epilepsy Center Munich for their support. We thank Janin Weichert and Ann-Kristin Rombach for their help with data collection, Tzvetan Popov for input on the source analyses, and Maria Wimber, Ehren Newman, and Howard Bowman for helpful comments.
The authors declare no competing financial interests.
- Correspondence should be addressed to either of the following: Tobias Staudigl, Department of Psychology, University of Konstanz, Postfach ZPR, D-78457 Konstanz, Germany, tobias.staudigl{at}uni-konstanz.de; or Simon Hanslmayr School of Psychology, University of Birmingham, B15 2TT Birmingham, UK. s.hanslmayr{at}bham.ac.uk
