Time for Memories

Time, neural sequences, oscillations, and memory

The survival and prosperity of an animal are afforded by the regularities and consistencies of the niche in which the animal lives. These affordances allow for emulation and internalization of world events by generalization from the past to the current situation to optimize future behavior. Thus, the concepts of past, present, and future are intricately intertwined. For historical reasons, we give them special names, including memory, timing, and planning, and search for their allegedly distinct brain mechanisms. We suggest that the organization of brain dynamics offers a mechanism to explain how the brain interprets both the temporal and spatial relationships among objects and events in the world. Neuronal networks constantly generate neuronal sequences or trajectories. These internal sequences, embedded into hierarchically organized brain rhythms, are well suited to emulate the multimodal spatial and temporal relationships afforded by external events (Cisek, 2007), as well as the sequential structure of episodic memory (Fig. 1A) (Buzsáki and Tingley, 2018; Buzsáki et al., 2022; Buzsáki, 2010; Tsao et al., 2018; Clewett et al., 2019; Heusser et al., 2016).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

A, Brain mechanism-based view of episodic memory: the hippocampus as a sequential multiplexed pointer. Indices that point to cortical modules for different inputs are sequenced by evolving hippocampal activity patterns, thus preserving the directed ordinal structure over which experience occurred. Semantic content resides in cortical modules that are concatenated by the hippocampus during both encoding and recall. Aud, Auditory; Olf, olfactory; Som, somatosensory; Vis, visual. Adapted with permission from Buzsáki and Tingley (2018). B, Sequential activation of neuronal assemblies in an episodic memory task. Middle, The rat was required to run in a running wheel for 15 s before choosing either the left or the right arm of the maze based on the remembered last arm choice. It obtained a water reward if it chose the opposite of the previously chosen arm. Color-coded dots represent spike occurrences of simultaneously recorded hippocampal neurons. Left, Normalized firing-rate profiles of neurons during wheel running, ordered by the latency of their peak firing rates during left trials. Each line indicates a single cell. Right, Normalized firing rates of the same neurons during right trials. An observer can infer the run duration (and distance) in the wheel as well as the future choice of the rat from the same sequential firing patterns of the neurons. Adapted with permission from Pastalkova et al. (2008). C, Time detection on single trials using a time prediction model fit from all other trials. In each time bin, elapsed time in the running wheel is inferred either from the population firing rate vector (red) or the firing phases of active cells with respect to the theta oscillation (purple). In each case, the prediction approximates well the true time (black). D, Error of time estimation from population vector of neuronal activity in 3 rats. Rat 1 had <50 recorded neurons; Rats 2 and 3 had >50 neurons. Note reliable estimation of running duration from neuronal activity. Adapted with permission from Itskov et al. (2011).

Time and space are postulated to play special roles in episodic memory (Tulving, 1972; Squire, 1986). Episodic events have a duration and take place somewhere. An episode is an unfolding storyline in Newtonian space-time coordinates. In contrast to this time-directed and segment-defined type of memory, declarable semantic facts are abstracted punctate events. Semantic memories are gradually formed from multiple overlapping episodes with common items (junctions) among the episodes (Buzsáki, 2005; Buzsáki and Moser, 2013). Through repetitions, the “what” becomes invariant to the temporal and spatial conditions of the individual episodes. For example, neurons participating in hippocampal neural sequences have been designated as “place cells” (O'Keefe and Nadel, 1978) and “time cells” (Itskov et al., 2011; Eichenbaum, 2014). Yet, despite their appeal, these terms can be misleading because evolving neuronal sequences always also serve other functions rather than just mapping locations or keeping time.

Multiplexing WM and elapsed time in neural sequences

Broadly speaking, the need to learn the temporal relations between external events and anticipate them, engage in temporal discounting and temporal wagering, conceptualize time, perform mental time travel, and represent time in episodic memory, have been referred to as different components of temporal cognition (Sosa et al., 2021). Critically, many of these tasks require that neural mechanisms be in place to tell time.

It is helpful to distinguish between tasks that require explicit versus implicit timing (Coull and Nobre, 1998; Nobre and van Ede, 2018). Explicit timing tasks require observers to make judgments about the duration of events or intervals separating them, for example, by estimating, comparing, bisecting, or reproducing temporal durations or intervals (Coull and Nobre, 1998; Grondin, 2010; Ameqrane et al., 2014). In contrast, implicit timing occurs when observers learn the temporal structure in a task, such as the interval between stimuli, although the task requires no direct temporal judgment. Although not directly relevant, these learned temporal relationships influence performance, for example, by improving stimulus detection or discrimination (Nobre and van Ede, 2018).

The mechanisms underlying how the brain performs computations akin to a timer continue to be debated. But it is now widely accepted that in many cases the brain uses its own internal neural dynamics; that is, evolving spatiotemporal patterns of neural activity that change in some reproducible way to encode elapsed time (Buonomano and Mauk, 1994; Merchant et al., 2013; Paton and Buonomano, 2018; Issa et al., 2020). Notably, the same neural signatures that have been implicated in timing have also been implicated in WM and decision-making (Stokes, 2015; Kozachkov et al., 2022; Gold and Shadlen, 2007; Shadlen and Shohamy, 2016; Taxidis et al., 2020; Cueva et al., 2020). Influential early studies suggested that WM is encoded in steady-state persistent neural activity (Fuster and Alexander, 1971; Funahashi et al., 1989; Fuster et al., 2000), a fixed-point attractor in the language of dynamical systems. More recent studies, however, have suggested that WM may also be encoded in time-varying patterns of neural activity (Stokes, 2015; Lundqvist et al., 2018).

The neural signatures that have been implicated in timing and WM include ramping activity and neural sequences (Fujisawa et al., 2008; Pastalkova et al., 2008; Mello et al., 2015; Stokes, 2015; Paton and Buonomano, 2018; Issa et al., 2020). Additionally, both WM and timing have been proposed to rely on short-term synaptic plasticity (Buonomano and Merzenich, 1995; Buonomano, 2000; Mongillo et al., 2008; Stokes, 2015; Kozachkov et al., 2022), sometimes referred to as relying on the hidden neural state in the context of the timing literature (Buonomano and Maass, 2009), or as activity-silent mechanisms in the of WM literature (Stokes, 2015).

The convergence of the models of WM and timing are a direct consequence of their shared properties: they require transiently storing information, retrospective information in the case of WM and prospective information in the case of timing and decisions (e.g., when a delayed reward will occur). To explore this commonality directly, Zhou et al. (2023), following previous work (van Ede et al., 2017), developed a variant of the standard delay-match-to-sample (DMS) WM task that imposes an association between WM content and implicit timing. In its simplest form, a DMS task presents either of two cues (A or B) and, following a delay period, either of the two stimuli is presented again, resulting in four conditions (AA, AB, BA, BB). In the differential-delay-match-to-sample (dDMS) task, the identity of the first stimulus predicted the delay duration (Fig. 2A): AA and AB trials were associated with a 1 s delay, and BA and BB trials with a 2.2 s delay. These delays are irrelevant to the performance of the task itself, which simply required differential responses during match and nonmatch trials. To determine whether participants implicitly learned the temporal structure of the task, the cue-delay contingency was reversed in 20% of the trials. Compared with standard trials, the inverse efficiency (reaction time/accuracy) was worse during reverse trials (Fig. 2B), indicating that not only did participants implicitly learn the task-irrelevant cue-delay associations, but that violations of the expected delay impaired WM performance.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

A, Schematic of the dDMS task (left). Inverse efficiency (RT/accuracy) of human subjects on the dDMS task across Standard (cyan) and Reverse (orange) trials. The short and long delays correspond to the actual delay epochs. There was a significant main effect of Standard versus Reverse trial (n = 27, F_1,26 = 9.05, p < 0.01). B, Schematic of the RNN architecture and the inputs and target outputs for the dDMS task during the control and reverse conditions. C, Neurograms during the AA (top row) and BA (bottom row) conditions (A, red line above neurogram; B, green), sorted according to the peak time during the short (left) or long (right) delays (standard trials). The self-sorted neurograms (top left and bottom right) are cross-validated (average of even trials sorted on average of odd trials). The overlaid white and gray lines indicate the “motor” unit (right y axis) and “temporal expectation” output unit, respectively.

To gain insights into the potential trade-offs between the diverse neural signatures associated with both WM and timing, Zhou et al. (2023) trained recurrent neuronal networks (RNNs) to perform the dDMS task. Since the psychophysical data established that participants learned the cue-delay association, the RNNs were trained to drive two outputs: one that performed the match/nonmatch response and one that anticipated the delay duration, representing implicit timing (Fig. 2C). Interestingly, the dominant neural dynamic regimen that emerged in the RNN were neural sequences; specifically, the A and B cues elicited distinct sequential neural trajectories. Because each cue elicited distinct time-varying trajectories, the dynamics of the network effectively multiplexed the encoding of WM content (the first cue) and elapsed time.

It is relevant to note that, while neural sequences were the dominant dynamic regimen observed in the RNNs, some model parameters led to ramping dynamics. Thus, it is important to understand the potential computational advantages and disadvantages of these apparently distinct dynamic regimens. Low-dimensional, largely linear regimens, such as ramps, provide a code that is more amenable to generalization to novel delays; it is relatively easy for output neurons to learn to decode the cue from the ramping activity during any stage of the ramp (Cueva et al., 2020). In contrast, neural sequences require more sophisticated decoding or training to generalize to all delays. In the RNNs simulated in Figure 2, generalization to novel delays was robust because more than one delay was presented during training (standard and reversal trials). Because of their low dimensionality, however, ramps are not well suited to generate arbitrary temporal output patterns, including a simple step function. In contrast, neural sequences are ideally suited to generate arbitrary temporal output patterns, even while using biologically plausible decoding mechanisms, because the dynamics is high-dimensional and quasi-orthogonal; essentially, there is little overlap in the “time fields” of different units (Zhou et al., 2020, 2023). Interestingly, analyzing the evolution of the RNNs dynamics over the course of training on the dDMS task (Fig. 2) revealed that the dynamics transitioned from a regimen dominated by low-dimensional ramping to high-dimensional neural sequences. These observations suggest that these apparently distinct neural signatures may be able to transition smoothly from one to another depending on the extent of training and the computational requirements of the task at hand. These results thereby extend the proposal of the utility of neural sequences to include providing a shared neural dynamic regimen for encoding time and WM, in addition to the role in navigation and episodic memory discussed above (and below).

Memory and attention in time

We are active agents immersed in dynamically unfolding environments. Although the boundless incoming stream of information is ever new, not all of it is unpredictable. Consistent relationships among the attributes of events carry predictions that guide proactive sensory and motor preparation in the brain. The same holds for the temporal structuring of events. Recurring temporal structures enable proactive and temporally selective preparation for anticipated relevant events. The field of selective attention has uncovered many mechanisms by which the brain anticipates and selects relevant events to guide adaptive perception and action — for example, the filtering of competing sensory inputs, the upregulation of firing rates, and interareal synchronization. At the core of most of these mechanisms is the modulation of neuronal activity and of neuronal communication based on receptive-field properties, leading to the prioritization of items that occur at the relevant location or contain a relevant feature. Thus, we have come to a relatively advanced understanding of still frames, or snapshots, of attention. However, how the brain can use predictable temporal structure to anticipate and select relevant events immersed in the continuous flow of stimulation remains puzzling. As attention is, by definition, a dynamic process, we need to add “time” to understand attention fully. Similarly, our views of memory have also been dominated by fairly static frameworks, but recent studies demonstrated that not only are attention and memory dynamic processes, but they recursively influence each other (Nobre and Stokes, 2019).

Incidental, implicit learning of temporal structures within WM tasks results in temporal expectations that guide prioritization and access to WM contents at moments when they are most relevant (van Ede et al., 2017; van Loon et al., 2017; Zokaei et al., 2019; Jin et al., 2020). van Ede et al. (2017) provided a powerful example of this by embedding temporal structure in the expected recall times of a visual WM task (Fig. 3A). Two randomly oriented lines of different colors (orange or blue) were simultaneously presented and, after a delay period, participants had to report the angle of the line that matched the color of a probe cue. In 80% of the trials, the probe to reproduce the orientation of one of the colored lines (e.g., orange) appeared after a short delay (1.25 s) and the probe to reproduce the orientation of the other colored line (e.g., blue) appeared after a long delay (2.5 s). In a 20% minority of trials, this relationship was reversed, and a probe appeared at an improbable interval. Temporal expectations had a profound influence on WM performance, leading to faster access times as well as more accurate orientation recall for items that were probed at expected times. Furthermore, this dynamic prioritization was associated with the temporally specific attenuation of contralateral α (8-14 Hz) oscillations that, moreover, predicted WM access times on a trial-by-trial basis. These experiments demonstrated that attentional prioritization in WM can be dynamically steered by internally guided temporal expectations and that the neural mechanisms for accessing items from WM can be time-varying.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Results showing flexible temporal anticipation for WM retrieval (A) and based on long-term memory (B). A, Task schematic and behavioral results in the study by van Ede et al. (2017). Participants encoded two randomly oriented lines presented for 250 ms on the left and right of fixation. Lines were colored light orange and blue, and their side was random. After a short (1250 ms) or long (2500 ms) delay, a central probe prompted participants to reproduce the orientation of one of the lines. Color of the probe handle represents the item to be reproduced. The main manipulation was that the probability of being probed about the orange or blue item varied over time. Items presented in one of the colors was more likely (80% probability) to be probed after the short delay, and items in the other color were more likely to be probed after the long delay. Schematic represents the case in which the orange item is likely to be probed earlier and the blue item to be probed late. Participants had unlimited time to activate the reproduction response by clicking the mouse but then had limited time (2500 ms) to complete the response, thus yielding a decision response time and an angular error for the response in each trial. Decision times indicated that reproduction responses started earlier when participants were probed about the temporally expected items (red placeholders) at both short and long intervals. Angular errors were also smaller when retrieving the temporally expected item (red placeholders). Note the flexible reprioritization of items, such that colored items yielded slower and less accurate responses when unexpectedly probed early but fast and accurate responses when probed later, when expected. B, Task schematic and results from Cravo et al. (2017, their Experiment 1). In a learning session, participants viewed scenes in which a placeholder cartoon bomb appeared after 1500 ms. For a given scene, the bomb changed color after either a short (800 ms) or long (2000 ms) interval. Participants made a speeded response if the target turned blue (80% of trials, go target) and withheld from responding if the target turned red (20%, no-go targets). Over five blocks, participants learned the implicit temporal associations between scenes and intervals, with responses becoming faster and more accurate over blocks. A temporal-orienting task was performed after learning. The interval between placeholder and target matched that in the learning task on the majority (67%) of the trials (valid memory cues) but was reversed in the remaining trials (33%, invalid memory cues). Response speed and sensitivity were significantly improved when targets occurred at their learned long-term memory intervals. Both reaction times and perceptual sensitivity were better for targets following Valid (V) than Invalid (I) memory cues. The grand-averaged CNV potential was localized over the frontal temporal scalp and was strongly modulated by the temporal association. The CNV developed earlier and was steeper after scenes associated with short placeholder-target delays (red line) compared with long placeholder-target delays (gray line).

The interaction between temporal attention and WM goes beyond the dynamic prioritization for access to items in WM. Learned temporal structures can also help shield WM items from sensory interference (Gresch et al., 2021). Protecting WM content from distracting external sensory inputs and intervening tasks is a ubiquitous demand in daily life. Gresch et al. (2021) manipulated the temporal predictability of interfering items that occurred during the retention period of a visual WM task. Specifically, while participants maintained two colored lines in WM for an orientation-reproduction task like the one described above, an interfering oriented line was presented at a fixed or variable interval, depending on the experimental block. Participants could either ignore the interfering line (distraction) or had to discriminate its orientation (interruption) depending on the task condition. The predictable temporal structure of the fixed-interval blocks improved WM performance, as indicated by smaller angle errors in the fixed compared with variable blocks. To determine whether the protective effects of temporal expectations rely on distractor suppression or involve shielding of internal representations, the authors compared effects after interfering distractor stimuli that could be ignored versus after interfering interrupter stimuli that required a response. Whereas distractor suppression may be sufficient to confer protection from predictable distractors, any benefits after interruption are likely to involve memory shielding. Similar benefits of temporal expectations were observed after both types of interference, suggesting that implicitly learned timings can dynamically shield WM contents.

The interaction between time and memory is not limited to the short-term scale of WM. The fundamental role that our long-term memories play in guiding perception is increasingly recognized, but again these interactions have focused primarily on static spatial relationships and have often neglected their temporal and dynamic nature. To fill this gap, Cravo et al. (2017) asked whether long-term memories can guide attention proactively and dynamically based on learned temporal associations. In two studies, participants learned to detect or discriminate target events within unique scenes (the change of color of a placeholder cartoon bomb) and make a speeded response. During a training session, the target event consistently occurred after either a short or long delay (800 or 2000 ms) within any given scene (Fig. 3B). To determine whether participants implicitly learned and used the interval associations to guide performance, a subsequent testing phase included trials in which the target event occurred at the predictable, learned interval (valid memory cue; 67%) or the other, nonlearned interval (invalid memory cue; 33%). Participants were significantly faster and more sensitive to targets occurring at the previously learned intervals.

This memory-guided orienting of attention in time was further corroborated by concurrent EEG recordings of a classic electrophysiological marker of temporal anticipation (the contingent negative variation [CNV]), which developed more steeply when targets were expected after a short interval than when they are expected after a long interval. Importantly, how well a memory was stored influenced not only the benefit in performance but also the CNV amplitude. While the neural correlates of the CNV are not fully understood, they may reflect neural ramping activity observed in some timing and WM experiments (Brody et al., 2003; Leon and Shadlen, 2003; Macar and Vidal, 2003; Pfeuty et al., 2005). Future concerted efforts between animal, human, and computational approaches should further characterize the neural dynamic regimens associated with CNVs, and the degree to which timing, anticipatory attention, WM, and recall of long-term memories exhibit shared neural signatures, such as ramping and neural sequences.

These and other findings (Nobre and van Ede, 2018; van Ede and Nobre, 2023) emphasize the ecological role that memories play not only in storing information but also in anticipating future events and preparing perception. Specifically, rather than emphasizing their reflective and retroactive role of reconstituting, or re-membering past events, they highlight the proactive role they play in predicting and preparing perception dynamically by “pre-membering” anticipated events. Furthermore, long-term memories underlie our ability to learn temporal relationships between events in the external world, and use these memories both to anticipate future events and to prioritize access to or protection of memory content at the expected moments. As sensory experience unfolds, the sequences of changes in the incoming sensory stimulation and their temporal relations interact with internal states to create the episodic events that ground our autobiographical memories. Timing and memories also exhibit important and interesting interactions at this broader, more extended timescale.

Episodic memory affects the perception of time, and the perception of time affects episodic memory

The everchanging outside world is accompanied by an orchestra of internal neural trajectories that respond to and anticipate external environmental features. How do these trajectories come to mark the beginning, middle, and end of episodic events? How does the brain know how to organize items in the world into causal units or events? While we do not have the answer to this question, we have learned a lot about what kinds of change the hippocampus and cortex are most sensitive to and how these “change detectors,” if you will, have reliable and robust effects on the organization of our memories.

Identifying how the brain extracts and represents an overarching structure from experience is fundamental to our understanding of episodic memory. But what defines an “episode” in episodic memory? Through the use of a novel paradigm, referred to here as the Ezzyat-DuBrow-Davachi Paradigm (Fig. 4A), it has repeatedly been shown behaviorally that context, broadly defined as shifts in goal states, influences how we perceive and remember discrete events. Context shift or boundaries can be imposed by changing location, switching from viewing images of people to objects, or even rewording a narrative from “… moment later …” to “… a while later ….” These context shifts influence whether items appearing on either side of an event are bound together or not, affecting memory for their order (Fig. 4B) (Ezzyat and Davachi, 2011, 2014; DuBrow and Davachi, 2013, 2014, 2016). Critically, context shifts also modulate our retrospective perception of time, leading to subjective time dilation (Ezzyat and Davachi, 2014; Lositsky et al., 2016).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

A, Ezzyat-DuBrow-Davachi paradigm for studying the effect of event boundaries on episodic memory. B, Memory for the correct temporal order of items is significantly better when judging within context pairs (right). When asked to judge whether pairs of items were close or far apart, participants were more likely to rate items separated by the same distance as close when the pair was within the same context, and as “far” across contexts (right). Adapted with permission from Ezzyat and Davachi (2011); DuBrow and Davachi (2013, 2014, 2016); Heusser et al. (2016); Clewett et al. (2019).

Neuroimaging studies using fMRI reveal how event boundaries shape memory representations in the brain. This work has identified neural measures of event organization that predict the temporal organization of events in long-term memory (Ezzyat and Davachi, 2014; DuBrow and Davachi, 2016). The hippocampus, medial temporal cortical regions, and the PFC play essential roles in memory formation and integration (Eichenbaum, 2004; Howard and Eichenbaum, 2013). The gist of this work is that, while context is somewhat stable, more slowly changing, the items we encounter are maintained in similar, more slowly changing, neural states. Indeed, we see a network of regions, including medial temporal lobe regions and mPFC, which shows that univariate activation increases across items within an event and then drops off at context shifts, mirroring the ebb and flow analogy of the contexts themselves (Ezzyat and Davachi, 2011). Within these more stable events, furthermore, MEG has shown theta-γ coupling that is present only when participants remember the correct temporal order of presented items in the Ezzyat-DuBrow-Davachi Paradigm (Heusser et al., 2016). Finally, in addition to activity increasing and decreasing over time, the stability in hippocampal multivoxel patterns across time is correlated with temporal integration in memory (Ezzyat and Davachi, 2014).

The recruitment of different neuronal ensembles appears to have a strong bearing over whether memories become segregated or integrated. Recent rodent and human work also shows that information may become integrated through overlapping contexts and neural ensembles. If events are experienced close enough or in similar contexts (e.g., two halves of a movie), they can become integrated into a unified memory representation (e.g., representation of entire movie) (Cai et al., 2016; Lositsky et al., 2016; Tsao et al., 2018). On the other hand, the acquisition of more distinct hippocampal timestamps may contribute to temporal pattern separation processes that differentiate memories of repeated exposures to the similar spatial/perceptual context. Temporal context signals thereby inform the separation or integration of events depending on whether an individual's goal is to build a coherent memory regardless of when things are learned or to minimize memory interference between perceptually overlapping events.

In sum, the structure of our episodic memories is strongly influenced by WM processes that are present during our experiences. What we attend to, and in what order, shape how we remember the past. Reversing the order by the brain leads to false memories. Importantly, however, these processes interact with an underlying representation of the overall context and the affordances each context offers. These processes influence our memory for the temporal order of events and may also contribute to memory-based temporal anticipation effects in attention. In addition, they also contribute to more subjective aspects of temporal memory, such as how long an event took. We remember items from the same event as having occurred closer in time compared with items from across event boundaries, even if the same amount of “clock” time has passed. Somewhat counterintuitively, however, experiences containing a lot of change lead to the perception of time passing quickly and shorten the actual duration of events in memory (Sherman et al., 2023). Thus, across the ongoing work on episodic event segmentation, results have yielded valuable insights about the close interrelation between temporal processing and memories over different time scales to integrate and separate the continual flux of sensory experience into the meaningful events. The sensory, mnemonic, and temporal factors building the events also interact closely in shaping the subjective nature of the resulting episodic memories.