Abstract
Cognitive psychologists have long hypothesized that experiences are encoded in a temporal context that changes gradually over time. When an episodic memory is retrieved, the state of context is recovered—a jump back in time. We recorded from single units in the medial temporal lobe of epilepsy patients performing an item recognition task. The population vector changed gradually over minutes during presentation of the list. When a probe from the list was remembered with high confidence, the population vector reinstated the temporal context of the original presentation of that probe during study, a neural contiguity effect that provides a possible mechanism for behavioral contiguity effects. This pattern was only observed for well remembered probes; old probes that were not well remembered showed an anti-contiguity effect. These results constitute the first direct evidence that recovery of an episodic memory in humans is associated with retrieval of a gradually changing state of temporal context, a neural “jump back in time” that parallels the act of remembering.
SIGNIFICANCE STATEMENT Episodic memory is the ability to relive a specific experience from one's life. For decades, researchers have hypothesized that, unlike other forms of memory that can be described as simple associations between stimuli, episodic memory depends on the recovery of a neural representation of spatiotemporal context. During study of a sequence of stimuli, the brain state of epilepsy patients changed slowly over at least a minute. When the participant remembered a particular event from the list, this gradually changing state was recovered. This provides direct confirmation of the prediction from computational models of episodic memory. The resolution of this point means that the study of episodic memory can focus on the mechanisms by which this representation of spatiotemporal context is maintained and sometimes recovered.
Introduction
Episodic memory refers to our ability to remember vividly specific events from our own experience. The vividness of episodic memory, along with the specificity of the memory to a particular place and time, has led researchers to characterize episodic memory as “mental time travel” (Tulving, 1972; Hassabis et al., 2007; Schacter et al., 2007). This verbal description has been operationalized in computational models of episodic memory in which the flow of time is described by a slowly and gradually changing state of temporal context (Howard and Kahana, 2002; Sederberg et al., 2008; Polyn et al., 2009; Howard et al., 2015). In these models, episodic memory retrieval is accompanied by the recovery of a prior state of temporal context, a jump back in time, which accounts for the behavioral contiguity effect (Kahana, 1996; Schwartz et al., 2005; Howard et al., 2008; Unsworth, 2008).
This computational hypothesis makes two predictions that can be tested neurally (Fig. 1). First, in addition to stimulus-evoked activity, the activity of some neurons involved in episodic memory should also change gradually over time. This prediction aligns with a large body of animal work showing that neural ensembles in the hippocampus, amygdala, and prefrontal cortex change slowly over time scales up to at least tens of minutes (Manns et al., 2007; MacDonald et al., 2011; Hyman et al., 2012; Mankin et al., 2012; Rubin et al., 2015; Cai et al., 2016; Rashid et al., 2016; Salz et al., 2016; Howard, 2017). Second, during retrieval of an existing memory, the prior state (temporal context) associated with an episodic memory should be restored. Although some prior studies have attempted to measure this hypothesized reinstatement (Manning et al., 2011; Howard et al., 2012; Yaffe et al., 2014), due to methodological limitations of those studies, there is presently no definitive study linking recovery of a gradually changing temporal context to episodic memory in humans.
A neural signature of retrieved temporal context. a, While experiencing a sequence of stimuli A–B–C, the brain is hypothesized to maintain information about the recent past at each moment. Because the recent past changes gradually, so too should this brain state. That is, the brain state after G should resemble the brain state after F more so than the brain state after C. This gradually changing representation is hypothesized to form a temporal context for the study items. b, Retrieved temporal context models hypothesize that an episodic memory is accompanied by recovery of the temporal context at the time that memory was encoded. When the participant remembers a particular event such as C, this reinstates the temporal context when C was experienced. This predicts that the brain state after memory for C should resemble the brain state during experience of the neighbors of C. The similarity should fall off with distance from C in both the forward and backward directions.
Episodic memory is often studied in the laboratory using the item recognition task. In item recognition, participants are presented with a study list of novel stimuli to remember (here, we used pictures). After study, participants are provided with a set of probe stimuli one at a time, some of which were on the study list and some of which were not. The participants' task is to distinguish probe stimuli that were on the list from probe stimuli that were not on the list. Many investigators have hypothesized that recognition memory is supported by two processes, recollection and familiarity (Yonelinas et al., 2002; Eichenbaum et al., 2007; Wixted, 2007; Staresina et al., 2013). According to this viewpoint (which, it should be noted, is not universally accepted; Squire et al., 2007) recollection corresponds to vivid episodic memory in which details of the study experience is recovered. When an old probe is recollected, triggering retrieval of an episodic memory, this is believed to lead participants to endorse the probe as old with high confidence (Yonelinas et al., 2002; Diana et al., 2007). Regardless of one's position on two-process theory, it is clear that highest confidence old responses are often associated with the recovery of detailed source information about the context in which a probe was studied (Slotnick and Dodson, 2005; Hautus et al., 2008; Onyper et al., 2010), with a behavioral contiguity effect (Schwartz et al., 2005), and with the activation of neurons in the medial temporal lobe (Rutishauser et al., 2015), properties that we would ordinarily associate with an episodic memory. In this study, we operationalized highest confidence old responses as a marker of probes that were more likely to have triggered an episodic memory.
Materials and Methods
In this study, epilepsy patients performed an item recognition task rating their confidence that probes were presented on a six-point scale (Fig. 2). During both study and retrieval, single units were recorded from microelectrodes implanted in the medial temporal lobe (MTL). Population vectors were measured across units; consistent with previous results, the population vectors changed gradually during study of the list. Comparing the population vector in response to an old probe at testing time with the population vectors during the study enabled us to evaluate whether temporal context is recovered. We tested the hypothesis that probes that triggered a strong episodic memory, here operationalized as probes that received a highest confidence response, are accompanied by greater recovery of temporal context than probes that did not trigger a strong episodic memory, here operationalized as probes that did not receive a highest confidence response.
Behavioral task. During a study (learning) phase, participants were asked to learn set of pictures. To ensure that the patients were attending to the picture, they responded to an orienting task after each item. After a 30 min delay, participants were presented with a test list that included both stimuli from the study session and also new probes. For each, they indicated whether they thought they had seen an item before or not on a six-point confidence scale.
Patients
Fifty-four recording sessions were made from 35 patients of either sex who were evaluated for possible surgical treatment of epilepsy using implantation of depth electrodes. All patients volunteered for the study and gave informed consent. Protocols were approved by the institutional review boards of the Cedars–Sinai Medical Center, Huntington Memorial Hospital, and the California Institute of Technology. Of the 54 recording sessions, 44 were previously reported by Rutishauser et al. (2015) and 10 were not. The dataset used in the present study is a subset of a publicly available dataset that has been published, along with another study that describes the methods in detail (Faraut et al., 2018). The complete dataset includes 65 sessions, some of which became available after the analyses in the current study were started. Five sessions were rejected because memory accuracy was not sufficiently high (d′ < 0.5). The remaining 49 sessions were from 33 patients, of whom 24 were male and 9 were female.
Electrophysiology and spike sorting
The recording methods and single-unit data analyses for this dataset have been described in detail previously (Rutishauser et al., 2015; Faraut et al., 2018). Briefly, the recordings analyzed here were obtained from depth electrodes implanted bilaterally within the hippocampus and amygdala (eight microwires each, 32 channels per patient in total).
Broadband extracellular recordings were filtered at 0.1 Hz to 9 kHz and sampled at 32 kHz (Neuralynx). Electrodes were localized based on postoperative MRI images.
Electrode locations were chosen according to clinical criteria alone. Spikes were detected and sorted as described previously (Rutishauser et al., 2006).
Behavioral task
The task (Fig. 2) consisted of two parts: a study (learning) phase followed by a test phase. During study, patients viewed a list of 100 photographs of natural scenes.
There were 25 instances each from five different visual categories (animals, people, cars/vehicles, outdoor scenes/houses, and flowers/food items; see Fig. 6 for examples). The list was assembled randomly such that categories were not clustered. Each image appeared on the screen for 1 s, followed by a blank delay of 0.5 s, followed by an orienting task in which participants answered whether the image they had just seen contained an animal or not. The method used in this behavioral task is the same as that used in Rutishauser et al. (2015)
A delay that ranged in duration from ∼15 min to ∼30 min intervened between study of the last stimulus and the beginning of the test list. During the test phase, subjects were shown 100 images, half of which were identical to those seen previously (“old”) and half of which were novel (“new”). After each image, subjects indicated whether they saw the image before or not together with how confident they were about their decision on a scale of 1 to 6 where 1 = “new, certain”; 2 = “new, probably”; 3 = “new, guess”; 4 = “old, guess”; 5 = “old, probably”; and 6 = “old, certain.” There was no response deadline.
Artifact rejection
We excluded 96 units that contributed no spikes to the firing rate vectors or that had a bimodal firing rate distribution. This left a total number of 1286 units used in this report. In addition, we excluded trials during which there was an abrupt signal loss in several simultaneously recorded units (Fig. 3). Such loss is likely attributable to recording problems and we thus excluded such periods of time. To achieve this, time periods during which a fraction of ≥0.25 of the units ceased firing for ≥0.05 of the total trial duration were classified as artifacts. We identified such artifacts in two study sessions and four test sessions.
Artifact rejection. The raster plots show the activity of each unit (row) as a function of time. The method for artifact rejection described in the text identified the red squares as an artifact. The rejected units were all located in the same brain region.
Figure 3 shows examples of artifacts that were rejected by the artifact rejection algorithm. There were a total of 1286 units across sessions, each potentially present in 200 events; of these, a total of 664 points were rejected. These artifacts were found and rejected from six of 49 sessions.
Population vectors
Population vectors were computed from the average firing rates within a 2 s window starting at stimulus onset. To control for changes in baseline firing rate for different units, the mean firing rates for each unit were z-scored with respect to the average firing rate of that unit across all events. After z-scoring, all statistics reported were computed across all recorded units across all sessions. Trials with reaction times that exceeded 2.5 SDs of the reaction time distribution of a given patient were excluded (136/10439 trials were excluded based on this criterion).
Recency analysis
To evaluate whether the ensemble changed gradually over time, we analyzed how the similarity between population vectors changed as a function of recency, the difference between the serial position of the two events. For instance, the comparison between the population vector from presentation of the seventh stimulus in the list to the population vector from the fourth stimulus in the list is associated with a recency of −3 (Fig. 4a). A normalized inner product of z-scored population vectors (the inner product normalized by the number of units) was used to characterize the similarity between the ensemble response between a pair of events as a function of recency. To avoid any possible confounding influence of a primacy effect on the analysis, only events after the first 20 item presentations were included. In doing statistics on effects of recency and contiguity (described below), recording session was treated as a random variable.
Schematic for definition of recency and contiguity. Analyses in this study compute the neural pattern similarity between pairs of events. These similarities are averaged over the experiment and aggregated as a function of recency or contiguity. a, Recency is defined as the difference in the serial positions at which two events took place. b, Contiguity is measured in units of lag. When a stimulus is presented as a recognition probe, lag is defined as the difference in serial positions between the original presentation of the probe stimulus. Comparison of a probe with the original presentation of that stimulus is associated with a lag of zero.
Neural contiguity analyses
To determine whether memory for an event caused reconstruction of the gradually changing neural state during study of that event, we compared ensemble similarity as a function of lag, defined as follows (Fig. 4b). Given an old test probe that was originally (during study) presented at serial position i and a study event presented at serial position j, lag is defined as j − i. To be concrete, consider the population vector from the test of an old probe originally presented at serial position seven. We compared this population vector with each of the population vectors from a study event. The lag associated with comparison of the test event to the study event at serial position seven, the same stimulus, is zero. The lag associated with the comparison to the population vector from study at serial position eight, which immediately followed study of the stimulus, is +1; the lag associated with the comparison with the event from serial position six is −1. For each old probe, lag defines a number line across the study serial positions with zero at the original study location of the probe stimulus. Ensemble similarity between each pair of events consisting of an old test probe and a study stimulus was aggregated as a function of lag.
Note that the number of data points entering into the contiguity analyses changes as a function of lag. For instance, there are many more combinations of serial positions that result in a lag of +1 than there are combinations that lead to a lag of +50. For statistical tests, we restricted our attention to lags between −30 and +30. Because lag zero is a special case (similarity could be boosted simply because the same visual features present during study and test of the same stimulus), lag zero was not included in statistical analyses of lag.
Isolating a neural signature of episodic memory–memory advantage index
A goal of this work was to identify the neural correlates of episodic memory. Here, we used confidence ratings to compare between memories with large versus small episodic contributions. A large body of work has argued that responses to old items that do not receive the highest confidence response rely on familiarity (and perhaps weak recollection), whereas highest confidence old responses rely on a mixture of familiarity and strong recollection (Yonelinas et al., 2002; Diana et al., 2007). To isolate the contribution due to episodic memory, we computed a difference between old probes that received a highest confidence response and old probes that did not receive a highest confidence response. Subsequent analyses subdivided old probes that received a lower confidence old response (4–5 on the six-point scale) from old probes that received a new response (1–3 on the six-point scale). Note that these analyses can only identify neural signatures of episodic memory performance that manifest as consistent changes as a function of temporal lag; there may well be other neural signatures of episodic memory that are invisible to this analysis.
To compute this difference due to memory, we started by taking the product of z-scored firing rates for each pair of stimuli that entered into the contiguity analysis aggregated by lag. However, rather than averaging over units, as in taking the normalized inner product, we computed a matrix with each possible lag corresponding to the columns and each unit corresponding to the rows. Separate matrices were computed using the similarity for low confidence and high confidence trials. To estimate a difference attributable to episodic memory, we took a paired t test (over units) for each lag as a measure of “memory advantage.” The use of the paired t-statistic minimizes variability due to difference across the units.
The t-statistic can be used to evaluate the null hypothesis directly (values >1.96 are statistically different from zero), but it can also be compared for different lags. If there was no recovery of temporal context, then the memory advantage would be the same across lags; a systematic change in the memory advantage as a function of lag must reflect recovery of some form of information that was present during study of the list.
Permutation analysis.
The assumptions of the traditional parametric tests used in the contiguity analysis are violated. For instance, z-scored firing rates are in general non-normal. To eliminate concerns that the conclusions were simply the result of inappropriate parametric statistics, we supplemented those analyses with a permutation test. In this permutation analysis, the stimulus identities during the test session were shuffled randomly, thereby removing any actual link to the actual study events. We separately permuted the identity of all of the old probes that were remembered with highest confidence (6) and all other confidences (1–5) independently among themselves.
We then recomputed the statistics reported in the contiguity analysis for each of 1000 random permutations. This procedure preserves all of the marginal distributions.
However, it disrupts the actual temporal relationships between study and test. If the observed effect of a temporal variable (e.g., |lag|) exceeds the distribution of the permuted data, then this supports the conclusions of the parametric statistics.
Separate contiguity analyses for hippocampus and amygdala.
To gain further insight into the anatomical origins of the contiguity signal, we examined contiguity effects separately for gross anatomical regions. We computed contiguity effects separately for units recorded from the amygdala and hippocampus, collapsing over hemispheres. Note that there are more units recorded from the amygdala (849) compared with the hippocampus (533).
Additional analyses of lower confidence responses.
We also conducted an analogous set of analyses in which we compared three types of responses: highest confidence old responses (6), lower confidence old responses (4–5), and misses, old probes that received a new response (1–3).
Visually selective (VS) units
To determine whether the gradually changing temporal context representations examined using the recency and contiguity analyses were distinct from visual representations, we repeated these analyses restricting the analysis to VS units. VS units are those that responded differently to the different categories of images, as assessed by an ANOVA on their firing rate. Methods for identifying VS units were identical to those reported in detail previously (Rutishauser et al., 2015). A total of 213/1286 units were classified as VS.
Results
Behavioral results were consistent with episodic memory for some old probes
Patients judged each item presented during the test phase as either old (seen before) or new (not seen before) together with a confidence rating (Fig. 2). The behavioral results from patients were broadly consistent with canonical behavioral results from control participants (Kahana, 2012). Patients used all confidence ratings and used the highest confidence old response approximately five times more often for old probes than for new probes (Fig. 5a). In contrast, the lower confidence old responses (4–5) were less effective in discriminating old probes from new probes. Patients were able to differentiate weak from strong memories using subjective confidence ratings.
Behavioral results. a, Participants successfully distinguished repeated probes from new probes. Shown is the probability of each response (1–6) conditional on the ground truth; that is, whether the stimulus is old (blue) or new (red). Note that responses (1–3) for new (red) stimuli and responses (4–6) for old (blue) stimuli are correct, whereas the others are incorrect. Patients had good memory, as demonstrated by using the highest confidence rating (1 or 6) for about half of the new and old probes, respectively. Error bars indicate SEM across n = 49 sessions. b, Behavioral ROC curves for each participant included in this study (gray lines) and the average ROC (heavy line). The ROC plots hit rate as a function of false alarm rate for each possible criterion; chance performance would be along the diagonal. These ROC curves are typical of item recognition studies, with a reliable asymmetry characteristic of episodic memories (see text for details). c, The 30 min delay between study and test successfully eliminated behavioral recency effect. The hit rate, here, the probability of an old probe receiving a highest confidence response, is shown as a function of each probe's binned serial position during study. The slope of the regression line is not significantly different from zero. Error bars indicate the 95% confidence interval.
We next quantified each patient's behavior using a receiver operating characteristic (ROC) curve (Fig. 5b). The ROC shows the hit rate, the probability of a yes response to an old probe, as a function of false alarm rate, probability of a yes response to a new probe, for each possible confidence criterion. To the extent that the ROC points lie above the diagonal, memory is above chance. The ROC curves were asymmetric (Fig. 5b). To quantify this, we computed the slope of the z-transformed ROC curve for each session and compared the values with one. The average slope 0.73 ± 0.04 (mean ±SEM) was reliably less than one (t(46) = 6.32, p < 0.001) and the slope was less than one for 40 of 49 sessions (in two sessions, the patient only used the two extreme response keys for the new probes, making it impossible to measure a slope). A slope of the z-ROC curve less than one is commonly taken as a signature of episodic memory accompanied by successful recollection (Yonelinas, 2002; Fortin et al., 2004; Wixted, 2007).
It is important for the later neural analysis that the probability of recognizing a stimulus as old does not depend on how long ago it was seen; some previous attempts to measure a neural jump back in time (Howard et al., 2012) were confounded by a large behavioral recency effect. In this study, the study and test period were separated by a delay of ∼30 min. This successfully eliminated the recency effect: the hit rate (probability of a yes response) was independent of the position an item was shown in during study (Fig. 5c). There was no significant effect of serial position, as indicated by a regression coefficient (2 ±7) × 10−4, that was not reliably different from zero. This shows that the delay between study and test was effective in eliminating the recency effect at test. These behavioral results are consistent with another study using the same task (Fig. 1, Rutishauser et al., 2018).
Population vector during study changed gradually over at least a minute
A key requirement for contextual reinstatement to occur is that neural activity changes gradually across multiple stimulus presentations during learning. Because they are not imposed by the stimuli, which are randomized in order, such gradual changes could be a signature of temporal information. We thus first tested whether neurons within the MTL exhibit signatures of a gradually changing temporal context. We constructed population vectors from the mean firing rate in a 2 s window after stimulus onset for all recorded units (see Materials and Methods for details). We then tested whether the pairwise similarity between population vectors from study events differed systematically as a function of time between those events. We found a gradual increase in similarity for pairs of study events closer together in time (Fig. 6a); the regression coefficient was 0.00123 ± 0.00008 (F(1,78) = 221.6, p < 0.001). A similar recency effect was also evident during the test phase (Fig. 9); the regression coefficient was 0.00089 ± 0.00006 (F(1,78) = 188.7, p < 0.001). We thus found significant temporal context effects during both study and test.
Neural jump back in time. a, Neural recency effect. Top, Schematic describing the definition of recency. For each presentation of a stimulus, a population vector was computed for the 2 s after presentation of the stimulus. This vector was then compared with the population vector from all preceding stimulus presentations and the similarity was aggregated as a function of the recency between the comparisons. Bottom, Population vector showing a recency effect changing (conservatively) to at least recency −30 during study, corresponding to ∼2 min. Smoothed curves are from a LOESS regression. b, Neural contiguity effect showing a jump back in time. Top, Schematic of the lag variable. For a test probe, similarity of the population vector after the test probe is compared with the population vectors of each study event. The similarity is aggregated as a function of lag, the difference between the original presentation of the probe stimulus and the other list stimulus; the lag to the repeated stimulus is zero. Bottom, To isolate the effect due to episodic memory, we calculated the difference between the similarity for pictures receiving a highest confidence response and pictures that were not well remembered (see Materials and Methods for details). This “memory advantage” is in units of a paired t-statistic. For clarity, a sliding binning procedure was used to plot the results for lags other than zero. Critically, the memory advantage is peaked around zero, falling off gradually in both the forward and backward directions, indicating a neural jump back in time associated with successful episodic memory retrieval.
We next tested whether this neural recency effect was also visible for specific subsets of VS units (Rutishauser et al., 2015). As Rutishauser et al. (2015) described previously, VS units respond shortly after the onset of a stimulus conditional on the visual category of a stimulus. For example, a subset of “animal-selective” VS units change their firing rate only when the image contains an animal. A total of 213 (of 1286) recorded units qualified as VS units (see Materials and Methods) and the analysis that follows is restricted to these units. If VS units only carry information about the stimulus that was just presented, then their activity should not vary gradually and would thus not show a temporal context effect. Contrary to this prediction, we found that VS units also exhibited a robust neural similarity effect similar to that observed to all recorded units (Fig. 8a). This was true both during study, 0.0013 ± 0.0001 (F(1,78) = 104.6, p < 0.001), as well as test, 0.0009 ± 0.0001 (F(1,78) = 84.64, p < 0.001). Therefore, the response of VS units is modulated by temporal context in addition to visual input. This suggests that feedforward visual input is modulated by temporal context, a critical prediction of the temporal context model.
Episodic memory is associated with the recovery of temporal context
We next computed the similarity in neural response between pairs of test and study items (see Fig. 6b, top, for an illustration). The contextual reinstatement model predicts that the neural response to a recollected old probe that was originally presented at position i will be similar to the neural response to study events that were presented shortly before or after position i. The variable lag describes the difference between the serial position of the original presentation of an old probe and a study item; lag zero corresponds to the comparison between an old probe at test and its original presentation during study (when it was new). We would expect the neural pattern similarity at lag zero to be large to the extent the response is determined by visual input, which is similar for study and test of the same stimulus.
Raw contiguity analyses
We first computed the neural similarity as a function of lag separately for recollected probes and unrecollected probes, operationalized as probes that did and did not receive a highest confidence old response (Fig. 7). Negative values for the measure of neural similarity in the contiguity analysis do not imply that the vectors are anticorrelated. To see why, start with a skewed random variable and take the z-score of this variable. The expected value of the product of independent draws from this z-scored random variable is negative. Negative values of the contiguity analysis are a consequence of the skew of the distribution of firing rates.
Neural similarity as a function of lag for old probes that received a highest confidence yes response (filled circles) and old probes that did not receive a highest confidence yes response (gray open circles). Statistical analyses confirm that there was a contiguity effect (inverted-V centered around zero) for remembered probes but an anti-contiguity effect (V-shaped centered around zero) for unremembered probes. All data points except lag zero were binned. A LOESS curve was fitted for each dataset. a, All units. b, Analysis restricted to units categorized as VS.
The similarity at lag zero was much higher than other lags for highest confidence responses for both all units taken together and for VS units taken alone. For all units (Fig. 7a), the similarity value at lag zero for probes that received a highest confidence response (Fig. 7a, filled circles) was greater than for other lags (sign test, p < 0.001). Similar results were found for VS units taken alone (Fig. 7b). For old probes that did not receive a highest confidence response (Fig. 7a, open circles), there was no discernible advantage for lag zero for all units taken together. For VS units (Fig. 7b, open circles), there was a reliable advantage for lag zero over other lags (p < 0.001 by a sign test), although the numerical value was much smaller than for probes that received a highest confidence response. This effect at lag zero is unsurprising and would be expected to hold for any neural process that responds to the features of the stimuli.
In addition, similarity tended to decrease as a function of |lag| for old probes that received a highest confidence response. That is, for old probes that received a highest confidence response, the population vector at test was more similar to the population vector for study events close in time in both the forward and backward direction to the time at which that probe stimulus was originally studied. In contrast, there was a tendency for similarity to increase as a function of |lag| for probes that were not recollected. That is, for probes that did not receive a highest confidence response, the population vector at test was less similar to the population vectors for items studied close in time to the probe stimulus than for items studied further away. This suggests that the degree of contextual reinstatement predicted the success or failure of episodic retrieval.
We quantified these results by performing an ANOVA with |lag| as regressor (excluding lag zero) and direction (backward or forward) as a categorical variable separately for old probes that did and did not attract a highest confidence old recognition judgment. For simplicity, we will refer to these as recollected and unrecollected probes. We did this ANOVA for both all units (Fig. 7a) and restricting our attention to VS units (Fig. 7b). Both set of analyses led to similar conclusions. Considering all units, the neural similarity for recollected probes showed a significant effect of |lag| (F(1,56) = 8.09, p < 0.01), no effect of direction (F(1,56) = 2.15), and no interaction (F(1.56) = 1.40). For unrecollected probes, the effect of |lag| was again significant (F(1,56) = 9.59, p < 0.005) and there was neither an effect of direction (F(1,56) = 1.35) nor an interaction (F(1.56) = 1.79). However, the effect of |lag| was in different directions for recollected and unrecollected probes. For recollected probes, the effect of |lag| was positive in both the forward and backward directions (an inverted V); for unrecollected probes, the effect of |lag| was negative in both the forward and backward directions (a V-shaped curve). For recollected old probes, the effect of lag on neural similarity in the forward direction reached significance (−0.46 ± 0.15 × 10−3, p < 0.005) and the contiguity effect in the backward direction did not reach significance (0.19 ± 0.18 × 10−3, p > 0.2). For unrecollected probes, there was a reliable negative effect of lag in the backward direction (−0.52 ± 0.17 × 10−3 (p < 0.005). There was a similar trend in the forward direction, although the trend did not reach significance (0.21 ± 0.16 × 10−3).
Furthermore, there was an interaction such that recollected probes showed a different dependency on |lag| than unrecollected probes. Using |lag| as the regressor and recollection as a categorical variable, an ANOVA showed a significant interaction term (F(1,116) = 17.27, p < 0.001).
Subdividing probes that did not receive a highest confidence response
The preceding analysis compared the neural contiguity effect to old probes that received a highest confidence old response to the neural contiguity effect to all other old probes. In that analysis, old probes that received a lower confidence old response (4–5 on the six-point scale) were collapsed with misses, meaning old probes that received a new response (1–3 on the six-point scale). To determine whether there were reliable differences in the neural contiguity effect for probes that received a 4–5 response, we conducted further analyses comparing these three categories of responses with one another.
There was no evidence of an effect of |lag| for old probes that received a response of confidence 4–5 (F(1,58) = 1.615). The contiguity effect for probes that received a 4–5 response was reliably different from the neural contiguity effect for probes that received a highest confidence response; an ANOVA with |lag| as a regressor and confidence level (4–5 vs 6) as a categorical variable showed a significant interaction term between confidence level and |lag| (F(1,116) = 5.65 p < 0.05). Further, we investigated whether there is a significant difference between the neural contiguity effect caused by familiar old items (4–5 on the six-point scale) and to probes rated as new (1–3 on the six-point scale). An ANOVA similar to the above did not show a significant interaction between the two groups (F(1,116) = 0.1646). In light of these results, our subsequent analyses only compared old probes that received highest confidence responses (6 on a six-point scale) with all other old probes (1–5 on a six-point scale).
Episodic memory was associated with a neural jump back in time
To isolate the contribution to neural pattern similarity attributable to episodic memory, we calculated the difference between the neural pattern similarity as a function of lag for probes that received a highest confidence response and those that did not (see Materials and Methods for details). In the following, we refer to this difference as “memory advantage,” which is measured in units of a t-statistic comparing recollected with unrecollected probes. By examining the memory advantage as a function of lag, we can simply assess whether episodic memory retrieval is associated with a neural contiguity effect as predicted by retrieved context models.
The memory advantage index showed a robust contiguity effect (Fig. 6b). This was also true for VS units considered alone (Fig. 8b). The memory advantage at lag zero was significant (t = 4.75, p < 0.001). The effect at lag zero, however, does not indicate reinstatement of temporal context. This is because similar visual features were present during study of stimulus i and test of stimulus i. To test for a neural jump back in time, we investigated whether the memory advantage changed systematically as a function of lag. A jump back in time requires that the repeated image presentation triggers a retrieval of previous context; that is, the reinstatement of the neural ensemble activity present before the first encounter with the probe stimulus. This would be expected to manifest in a decrease in the neural memory advantage as a function of lag in both the forward direction (lag increasing from zero) and in the backward direction (lag decreasing from zero).
Visual-category-sensitive units showed neural recency and contiguity effects. a, Visual-category-sensitive units showed a neural recency effect. b, Visual-category-sensitive units showed a neural contiguity effect. Format is as in Figure 6, but with analyses restricted only to units that differentiated the category of the currently presented image during study.
Parametric analyses.
To evaluate this prediction, we performed an ANOVA on memory advantage with |lag| as regressor (excluding lag zero) and direction (backward or forward) as a categorical variable. There was a significant effect of |lag| (F(1,56) = 16.4, p < 0.001), but no effect of direction (F(1,56) = 0.003) and no interaction (F(1,56) = 0.01). The effect of |lag| means that the similarity of the population vector after recovery to the population vectors close together in time to the original presentation of the probe stimulus predicted whether the probe triggered an episodic memory (i.e., attracted a highest confidence old judgment). This is as predicted by the hypothesis that episodic memory is accompanied by recovery of a gradually changing state of temporal context in the human brain.
A decrease in the memory advantage extending to lags near zero but restricted to the forward direction (positive values of lag) could correspond to persistence of stimulus-specific features in memory such as in a short-term memory buffer. In contrast, a jump back in time would cause reconstruction of the pattern of activation before initial presentation of the probe. Therefore, a jump back in time would manifest as an advantage for lags near zero in both directions. To quantify the effect, we performed linear regressions of memory advantage onto lag separately for each direction. We found reliable regression coefficients for both the forward (lags 1–30) and backward (lags −30 to −1) direction (forward: −0.06 ± 0.02, F(1,28) = 7.69, p < 0.01; backward: 0.07 ± 0.02, F(1,28) = 8.694, p < 0.01). Therefore, we found an effect of lag on the memory index separately in both the forward and backward directions as predicted by a neural jump back in time hypothesis.
Permutation analysis.
To determine whether these results were simply due to violation of one or more assumptions of the parametric tests, we also performed permutation tests to evaluate the probability of obtaining these results by chance. The results of the permutation analysis were consistent with the conclusions from the parametric statistics. The observed regression coefficient for |lag| (0.06) was more extreme than the regression coefficient for 997/1000 permutations. The permuted regression coefficients were approximately normal (−0.004 ± 0.02). In addition to the regression of |lag|, we also computed the regression coefficients for the forward and backward regressions on the permuted data. The observed value for the forward and backward regression coefficients (−0.06 and 0.07, respectively) were more extreme than 970/1000 and 974/1000 of the values from the permuted data. Again, the permuted statistics were approximately normally distributed (−0.001 ± 0.03 and 0.008 ± 0.03, respectively).
VS units showed a jump back in time
We found similar evidence for a neural jump back in time when considering only VS units (Fig. 8b). The memory advantage at lag zero was significant (t = 5.05, p < 0.001). An ANOVA with |lag| as regressor (excluding lag zero) and direction (backward or forward) as a categorical variable showed a significant effect of |lag| (F(1,56) = 18.9, p < 0.001), but no effect of direction (F(1,56) = 0.07) and no interaction (F(1,57) = 0.91).
Considering the forward and backward directions separately, we found a regression coefficient of −0.06 ± 0.03 (F(1,28) = 5.4, p < 0.03) for the forward direction (lag 1 to lag 30) and 0.09 ± 0.02 (F(1,28) = 15.03, p < 0.001) for the backward direction (lag −30 to lag −1). Therefore, a signal compatible with contextual reinstatement was visible even when only considering VS units that were sensitive to the category of the visual stimulus presented during study.
The conclusions of these parametric analyses of contiguity restricted to the VS units were also supported by the results of the permutation analysis. The true regression coefficient for |lag|, −0.08, was more extreme than the coefficients from all 1000 permutations. The regression coefficients for forward and backward lag (−0.06 and 0.09, respectively) were greater than the values for 966/1000 and 993/1000 permutations.
Although the effect of contiguity was significant for the VS units taken in isolation, there was not a reliable difference between the contiguity effect in the memory advantage index for VS units and the contiguity effect in the memory advantage index for non-VS units. An ANOVA with |lag| as a regressor and group (VS/not-VS) as a categorical variable showed main effects of |lag| (F(1,116) = 25.27, p < 0.001) and group (F(1,116) = 39.96, p < 0.001), but no interaction of |lag| and group (F(1,116) = 2.72).
Contiguity signal separated by brain region
Separating the analysis according to brain regions, we found that units in the amygdala and hippocampus displayed a contiguity effect taken in isolation. In addition, there was no evidence that the contiguity effect differed between the two brain regions, supporting its existence independently in both areas.
An ANOVA of the memory advantage in the amygdala showed a reliable effect of |lag| (F(1,56) = 17.14, p < 0.001) without an effect of direction (F(1,56) = 0.558) or an interaction between |lag| and direction (F(1,56) = 0.0009). The main effect of |lag| was substantiated by a permutation analysis; the observed regression coefficient was larger than 1000/1000 values from permutations of the trials. Furthermore, the effect of decreasing similarity is evident in both directions, with the regression analysis in the forward direction yielding a significant coefficient in both the forward (−0.07 ± 0.03, p < 0.05) and backward (lag = 0.07 ± 0.02, p < 0.001) directions. These parametric results within the amygdala were supported by permutation analyses; the observed values were more extreme than 980/1000 and 986/1000 values from shuffled data.
Considering the hippocampus in isolation, we also observed a contiguity effect on the memory index. There was a main effect of |lag| (F(1,58) = 12.13), a weakly significant effect of direction (F(1,58) = 4.93, p < 0.05), and no interaction between direction and lag (F(1,58) = 0.4217). The observed regression coefficient for |lag| was greater than 978/1000 values from permuted data. Although the forward direction yielded a reliable regression coefficient (lag = −0.06 ±0.02, p < 0.01), the regression coefficient in the backward direction did not reach significance (0.04 ± 0.02, p > 0.09). The observed regression coefficient in the forward direction exceeded 942/1000 shuffled values. The observed regression coefficient in the backward direction exceeded 887/1000 shuffled values.
Most importantly, we did not find strong evidence that the effect of contiguity on the memory index was different across regions. An ANOVA with |lag| as regressor and brain region (amygdala vs hippocampus) as categorical variable showed a significance of brain region (F(1,116) = 7.1163, p < 0.01), but not a significant interaction (F(1,116) = 0.7259). A permutation analysis showed that in 547/1000 cases. the permuted data showed an interaction term this large.
These results are consistent with the hypothesis that the amygdala and the hippocampus have the same contiguity effect, but that it is more difficult to measure in the hippocampus than in the amygdala with these data. Indeed, the recordings yielded ∼1.6 times as many units from the amygdala than in the hippocampus. Of the 1286 units used for the analyses, 800 were located in the amygdala and 486 in the hippocampus.
Additional evidence for a jump back in time
If some old probes caused a jump back in time, then we would expect this to result in greater pattern similarity between pairs of test vectors and pairs of study vectors. If two test probes recover information from the temporal context during presentation of those items, then this would be expected to result in additional similarity between these two test events if those two test probes were close together in the list. Consistent with this hypothesis, although both study and test lists showed a reliable recency effect, for the test list, the similarity stabilized at a higher baseline value. At recency less than ∼−20, the test similarity was reliably higher (Fig. 9) at almost all values of recency.
Enhanced neural similarities during test. Neural recency effects for study (filled circles) and test (open circles). The neural similarity was higher between test events than between study events over a wide range of values of recency. This advantage is consistent with the predictions of a retrieved temporal context model. Smoothed curves are from a LOESS regression.
As a simple quantitative measure, the similarity for test is greater than the similarity for study at 13/20 of the points for recency −1 through −20. In contrast, for recency −21 through −80, the similarity is greater for 55/60 of the values. These two proportions differ from one another (χ2(1) = 6.41, p < 0.05).
Discussion
We found that the population activity of human MTL units changed gradually over minutes (Fig. 6a). This replicates in humans prior evidence for a gradually changing temporal context signal found previously in the MTL of animals (Manns et al., 2007; MacDonald et al., 2011; Naya and Suzuki, 2011; Hyman et al., 2012; Mankin et al., 2012; Rubin et al., 2015; Cai et al., 2016; Rashid et al., 2016) and humans (Manning et al., 2011; Howard et al., 2012; Hsieh et al., 2014; Yaffe et al., 2014; Hsieh and Ranganath, 2015). Crucially, VS category cells also showed this gradual change (Fig. 8a). This is important because it suggests that the population vector does not merely change gradually over time, but also carries information about the identity of the stimuli presented.
The critical new insight that the present study contributes is a first demonstration that the retrieval of human episodic memory is associated with the recovery of a gradually changing state of temporal context: a neural jump back in time. This analysis measures the similarity between a population vector caused by an old probe and population vectors during study of the neighbors of the original presentation of the probe stimulus. The difference between the population similarity calculated for probes that triggered an episodic memory (operationalized as a highest confidence old response) and the population similarity calculated for probes that did not trigger an episodic memory was greater for the neighbors of the original presentation and fell off reliably with distance from the original presentation of the probe stimulus in both directions (Fig. 6b). A robust contiguity effect associated with memory was observed when considering population vectors constructed from only the visual selective units (Fig. 8b).
We did not merely observe contiguity effect for recollected probes, but also an anti-contiguity effect for old probes that were not recollected (Fig. 7). Methodologically, this means that, had we averaged over all old probes, it would have been very difficult to observe a neural contiguity effect. Had we observed that the probes that did not evoke a highest confidence response resulted in a contiguity effect that was weaker, this would suggest a continuity between recollection and familiarity. Because there was not such an effect and, indeed, there was a tendency toward an anti-contiguity effect, this suggests that, in this study, successful retrieval of preceding temporal context is only observed for probes that received a highest confidence response. Put another way, if the degree of reinstatement of the neural population causes a high confidence response, then we would expect that high confidence probes would correspond to a high degree of reinstatement and probes that received a lower confidence response correspond to a low degree of reinstatement. This latter property predicts an anti-contiguity effect.
In this study, we observed that the brain state in the moments after retrieval of an episodic memory resembled the gradually changing temporal context at the time that memory was encoded. However, this does not imply that this recovered context persists long after the recollection of that probe. If context retrieved by a probe persisted long after the presentation of the probe, then one would expect the rate of contextual drift during the test list to be very different from the rate of drift during study. Although there was some difference (Fig. 9), the discrepancy was modest compared with what one would expect if retrieved context persisted long after the presentation of a probe. One possibility is that retrieved context only becomes available for a short time after presentation of the probe and then dissipates. This is analogous with “awake replay” events in the rodent hippocampus (Carr et al., 2011; Pfeiffer and Foster, 2015), in which hippocampal place cells briefly fire as if the animal is in a remote location during sharp-wave-ripple events. Perhaps the neural jump back in time is a transient discontinuity in the stream of temporal context much like the transient discontinuity in the representation of position.
Methodological advantages of the present study compared with previous attempts to measure a neural jump back in time
This study avoids methodological pitfalls of previous studies addressing whether human episodic memory is associated with a jump back in time. A previous study with human single units (Howard et al., 2012) used continuous recognition, in which probes are intermixed with study items in a continuous stream of experience. In that study, there was a robust behavioral recency effect. Because recency is confounded with lag in the backward direction, it was necessary to decouple recency statistically from contiguity in that study. Here, the 30 min delay between study and test and the absence of a behavioral recency effect eliminated any confound due to recency. Note also that we would expect a neural recency effect to be present for old probes that were not recollected as well as old probes that were recollected. Because the neural contiguity effect was observed in the difference between these suggests that it is not due to a confound between recency and contiguity.
Another prior study used autocorrelated features from electrocortigraphy (ECoG) in a free recall study (Manning et al., 2011) and found that the features during recall of the word studied at serial position i in the list resembled the features during study of nearby list items. However, because free recall is extended in time and exhibits a robust behavioral contiguity effect (Kahana, 1996; Sederberg et al., 2010), that finding does not establish that recall of word i is associated with recovery of a gradually changing temporal context. Because of the behavioral contiguity effect, the recall of word i is likely to have been preceded by neighbors of word i, so that the neural contiguity effect could have been due to the persistence of item representations from previous recalls. Similar concerns apply to a human single unit study that argued for recovery of spatial context during a free recall task (Miller et al., 2013). Another ECoG study used cued recall to establish that successful recovery of a word was associated with recovery of temporally varying features from the list (Yaffe et al., 2014). Although this study was able to establish a correlation between successful memory and a contiguity effect, the analyses included lag zero in the measurement of the neural contiguity effect in the backward direction. Because the neural contiguity effect in the backward direction is a distinctive signature of a jump back in time, whereas similarity at lag zero may be attributed to repeated items, the analyses reported in that study did not clearly establish a neural jump back in time.
Recent studies showing the importance of temporal context in human memory
This study adds to a growing body of work from human cognitive neuroscience suggesting that that a gradually changing state of temporal context affects memory in a range of tasks. A free recall study using fMRI showed that the content of lingering item representations during study predicted free recall transitions during retrieval (Chan et al., 2017). A recent fMRI study showed that the amount of drift in the right entorhinal cortex between two events in a radio program predicted participants' judgment of the duration between the two events (Lositsky et al., 2016). Similarly, when participants rate the relative recency of two probes, hippocampal pattern similarity predicted the order judgment (DuBrow and Davachi, 2014, 2016). Moreover, in that same study, successful judgments were associated with reinstatement of stimuli that intervened between the two probes. A recent study with patients with MTL damage showed that patients were impaired at their ability to perform temporal ordering, as if an intact MTL was required for recovering temporal context (Dede et al., 2016).
Finally, a pair of recent studies suggest that the recovery of temporal context that we observed in the laboratory could also reflect a mechanism for memory in more natural settings. In natural settings, the visual features the participant experiences are autocorrelated in both time and space, unlike the randomly assembled list of visual stimuli experienced in a fixed location used in the present study. A recent study of natural memory automatically recorded pictures as participants went about their daily lives for several weeks (Nielson et al., 2015). After a delay, participants were brought into the scanner and shown images from their own lives. The pattern similarity between pairs of images that were well remembered was computed. The pattern similarity in the anterior hippocampus predicted the distance in both time and space between pairs of remembered images, on the scale of hours to weeks for time and tens of meters to kilometers for space. Another recent study adding to work in virtual reality environments (Chadwick et al., 2010; Copara et al., 2014) observed similar results for episodic memory in a well controlled virtual environment in which spatial and temporal proximity could be deconfounded (Deuker et al., 2016). In light of this growing body of evidence and modeling work suggesting a deep connection between temporal context and spatial context (Howard et al., 2014; Howard and Eichenbaum, 2015), the present study suggests that recovery of a gradually changing state of spatiotemporal context is an essential aspect of human episodic memory that depends crucially on the function of the MTL.
Implications for theory of hippocampus and episodic memory
We found that the activity of populations in the human MTL changed gradually over at least a minute, adding to a large and growing body of evidence that neural states change gradually in the MTL (Manns et al., 2007; Mankin et al., 2012; Hyman et al., 2012; Cai et al., 2016; Rashid et al., 2016; for review, see Howard, 2017). The present results suggest that these gradually changing states are recovered during retrieval of an episodic memory. This provides a challenge for traditional models of memory retrieval that rely on autoassociative pattern completion (Hopfield, 1987). Attractor models of content-addressable memory are notoriously sensitive to overlap in the stored patterns and the capacity of attractor networks plummets if the stored patterns are correlated with one another (Amit et al., 1985). Traditionally, the way that this inherent tension has been addressed is to assume that the pattern completion stage performed by the attractor network is preceded by a pattern separation stage that decorrelates the patterns before they are stored (Marr, 1971; Levy, 1989; McClelland et al., 1995). This hypothesis has been extremely influential and has inspired a wealth of empirical work, especially cognitive neuroimaging work (Bakker et al., 2008).
If pattern similarity was a big problem for the computational mechanism used to recover a memory, then one might have expected the brain to avoid pattern similarity at all costs. However, even in a randomly assembled list of pictures, the brain induces robust overlap, introducing pattern similarity to the neural states extending at least a minute. One possibility is that the CA3 field of the hippocampus is decorrelated, suggesting that populations in CA3 would not change gradually. Although we were unable to identify the subfields of the hippocampus in this study, animal work has provided mixed results addressing this point. Mankin et al. (2012) observed less temporal drift in CA3 than in CA1 over long periods of time during open-field foraging tasks. However, Salz et al. (2016) observed robust sequences of time cells in the CA3 that were indistinguishable from those observed in CA1 in the same experiment.
It is also possible that pattern separation followed by pattern completion is not necessary because the hippocampus does not rely on an autoassociative content-addressable memory (Teyler and DiScenna, 1985, 1986). Rather than relying on content-addressable pattern completion, time and space could function as pointers for an address-addressable memory (Howard et al., 2018). In this view, the hippocampus contains a map of indexes to the content that can be found at different temporal and spatial addresses. Singh et al. (2017) measured the amount of time to access memory for a picture as a function of how far in the past that it was experienced in a continuous recognition experiment. They found that the time to access a memory went up linearly with the logarithm of the recency of the probe stimulus. These results are as one would expect if memory accessed depended on scanning across a logarithmically compressed timeline of the past (Howard et al., 2015).
Footnotes
This work was supported by the National Institutes of Health (Grants R01EB022864 and R01MH112169 to M.W.H. and Grants R01MH110831 and U01NS103792 to U.R.), the National Science Foundation (CAREER Award BCS-1554105 to U.R.), and a Memory and Cognitive Disorders Award from the McKnight Foundation for Neuroscience (U.R.). We thank Nigel Stallard, Inder Singh, Zoran Tiganj, Amy Criss, and Rosie Cowell for helpful discussions.
The authors declare no competing financial interests.
- Correspondence should be addressed to Marc Howard, Department of Psychologial and Brain Sciences, Boston University, 610 Commonwealth Ave., Boston, MA 02215. marc777{at}bu.edu
