Abstract
The what, where, and when components of episodic memory can be differentiated based on their distinctive domain-specific underlying neural correlates. However, recent studies have proposed that a common neural mechanism of conceptual mapping may be involved in the coding of cognitive distance across all domains. In this study, we provide evidence that both domain-specific and domain-general processes occur simultaneously during memory retrieval by identifying distinctive and common neural representations for mapping what (i.e., semantic distance), where (i.e., spatial distance), and when (i.e., temporal distance) using scalp EEG from 47 healthy participants (age 21-30, 26 male and 21 female). First, we found that all three components commonly showed a positive correlation between cognitive distance and slow theta power (2.5-5 Hz) in parietal channels. Meanwhile, fast theta power (5-8.5 Hz) specifically represented spatial and temporal distance in occipital and parietal channels, respectively. Additionally, we identified a unique correlate of temporal distance coding in frontal/parietal slow theta power during the early phase of retrieval. All of the above neural markers of cognitive mapping, both domain-general and specific, were associated with individual differences in what, where, and when memory accuracy.
SIGNIFICANCE STATEMENT The Cognitive Map Theory was originally founded to explain how we remember and organize the immense amount of spatial information that we face when we navigate. However, memory research has recently trended in the direction of emphasizing the generalizability of cognitive mapping mechanisms to information in any domain, represented as distances in an abstract conceptual space. In a single study, we show that both common and unique neural coding of semantic distance (i.e., what), spatial distance (i.e., where), and temporal distance (i.e., when) simultaneously support episodic memory retrieval. Our results suggest that our ability to accurately distinguish between memories is achieved through an integration of domain-specific and domain-general neurocognitive mechanisms that work in parallel.
Introduction
Episodic memories are recollections of past events that allow us to mentally reconstruct those events in time (Tulving, 1972, 1993) and can be defined by a binding of what, where, and when information (Tulving, 1972, 1993; Nyberg et al., 1996; Clayton and Dickinson, 1998). Past studies have suggested that these three components are processed and remembered through different neurocognitive mechanisms (Nyberg et al., 1996; Hayes et al., 2004; Holland and Smulders, 2011; Kwok and Macaluso, 2015). The what component can be thought of as the contents (e.g., objects and agents) of an event, while the where and when components organize the contents into a spatial and temporal framework. The where component (i.e., spatial memory) has been most heavily investigated in relation to episodic memory, due to its detailed neural representation in the medial temporal lobe (Burgess et al., 2002; Maguire et al., 2006; Iglói et al., 2010; O'Keefe, 2014) and its comorbidity in memory disorders such as dementia (Pai and Jacobs, 2004; Plancher et al., 2012; Serino et al., 2015; Coughlan et al., 2018). Neurocognitive models of episodic memory have traditionally focused on the distinction between what and where pathways and their convergence in the hippocampal formation (Mishkin et al., 1983; Knierim et al., 2014). However, recent studies have emphasized the importance of temporal processing in episodic memory (Slotnick, 2010; Suh et al., 2011; Kyle et al., 2015); indeed, when memory (or event sequence) has been shown to be especially vulnerable in Alzheimer's Disease compared with the other memory components (Park and Lee, 2021).
The cognitive map is an internal representation of the environment based on distance relationships among remembered locations (Tolman, 1948; O'Keefe and Nadel, 1978). An increasingly popular theoretical view posits that the brain uses a generalized distance coding mechanism for using cognitive maps in memory. Such views are founded on neuroimaging results that show a common neural representation of distances, whether they be spatial, temporal, perceptual, semantic, or social, not only in the hippocampal formation (Doeller et al., 2010; Tavares et al., 2015; Constantinescu et al., 2016; Aronov et al., 2017; Bellmund et al., 2018) but also in the orbitofrontal cortex (Schuck et al., 2016; Park et al., 2021) and parietal cortex (Yamakawa et al., 2009; Parkinson et al., 2014; Bottini and Doeller, 2020). These results are also supported by recent intracranial EEG studies reporting that hippocampal theta (3-8 Hz) power codes various cognitive distances. For example, Solomon et al. (2019) demonstrated that hippocampal theta power coded both semantic and temporal distance (TD) in a verbal free recall task. Herweg et al. (2020a) found that theta power across the medial temporal lobe was correlated with spatial distance between recalled locations in a place-word association task. A recent study showed that the neural correlates of cognitive distance coding are observable from scalp EEG in the 2-11 Hz range in a temporal and spatial distance estimation task (Liang et al., 2021).
Distinguishing two different memories based on the discriminability of their spatial, temporal, or conceptual components is critical for retrieval of detailed episodic memory. And mechanisms both specific to each component and general to cognitive mapping may be called upon to make this possible. However, most past studies tested memory in a single cognitive domain or focused solely on identifying a common representation of distance. Furthermore, based on recent reports that slower theta (∼1-5 Hz) and faster theta (∼6-10 Hz) are differentially engaged for memory and spatial navigation (Jacobs, 2014; Pastötter and Bäuml, 2014; Bush et al., 2017; Goyal et al., 2020), there may exist frequency-dependent specificity in the neural coding of cognitive distances in the theta band.
In reconciling domain-general distance coding with domain-specific information processing into a comprehensive episodic memory framework, one possibility is that, although domain-specific mechanisms suffice for perceiving and recognizing relevant features within a class of entities (e.g., spatial locations, social conspecifics, etc.), when faced with the demanding task of comparing among many entities that vary along multiple dimensions, the brain may adaptively recruit general cognitive mapping processes. According to this view, it may be possible to find unique neural correlates of what, where, and when, on one hand, and a shared representation of distance mapping for all components, on the other. To this end, we aimed to find common and unique markers of semantic, spatial, and temporal distance during memory retrieval using scalp EEG, with its high temporal resolution for detecting the occurrence of multiple processes over a quick retrieval period.
Materials and Methods
Subjects
Forty-seven healthy college students (26 male and 21 female) participated in this study and were given a small monetary compensation (∼$25 U.S.). This study received the approval of the institutional review board of Korea Advanced Institute of Science and Technology and Seoul National University and was conducted in accordance with ethical guidelines for research on human subjects.
Scene-based episodic memory task
A scene-based episodic memory task was executed using MATLAB 2019b. The whole task consisted of four sets consisting of 10 scenes each, and each set was used for testing all three conditions (i.e., what, where, and when) in a blocked design (see Fig. 1A). One of the four sets consisted of real-life indoor pictures and was identical to the one used in the previous study (Park and Lee, 2021). The other three sets were virtual scenes newly generated using Unreal Engine (Epic Games), and each set had a different theme (apartment, outdoor, and museum). The order of the set, the condition, and their combination were all randomized (e.g., where-set2 → what-set4 → when-set1 → …). In each block, subjects were asked to remember 10 scenes during the encoding phase, and every scene was shown on the screen for 3 s. Following the encoding phase, subjects were informed of the test condition (what, where, or when) for that block, and then the retrieval phase began. Retrieval consisted of 10 forced-choice questions. In the what condition, subjects chose between a lure object and the correct object that was in a particular scene during encoding (see Fig. 1B, left). In the where condition, subjects chose between two scenes: one shown during the encoding and the other with an object displaced from the original location (see Fig. 1B, middle). In the when condition, subjects were shown two scenes and asked to choose the one that was seen first during the encoding phase (see Fig. 1B, right). Subjects were given 5 s to respond at will. After one run of three test blocks (what, where, when, randomly ordered, see Fig. 1A) was completed, subjects were given 1 minute to stare at a fixation cross at the center of the screen, and the EEG signal from this period was used as the baseline for the following three blocks. Subjects performed a total of 4 runs (12 blocks), testing what, where, and when memory four times each. Data were collected using two slightly different stimulus sets (e.g., size and location of objects), with 15 subjects for the first set and 32 subjects for the second (no significant age or sex differences between groups). The accuracy scores of the first set were scaled to match the mean and standard deviation of the second.
EEG signal acquisition and preprocessing
We recorded EEG signals using 30 passive electrodes mounted in an elastic cap (Neuroscan Grael): FP1, FP2, F11, F7, F3, FZ, F4, F8, F12, FT11, FC3, FCZ, FC4, FT12, T7, C3, CZ, C4, T8, CP3, CPZ, CP4, P7, P3, PZ, P4, P8, O1, OZ, O2. Signals from four channels (F11, F12, FT11, FT12) were discarded because of frequent motion artifacts. A reference electrode was positioned between CZ and CPZ, and a ground electrode was placed at a position between FCZ and FZ. Two additional electrodes were placed on the left and right mastoids. EEG signals were collected at 1024 Hz sampling rate, and bandpass filtered between 0.1 and 100 Hz. All sites were referenced online by the reference electrode and rereferenced offline to the algebraic average of the left and right mastoid electrodes. Impedance of all electrodes was kept <10 kΩ. Vertical EOG was recorded from a bipolar pair of electrodes placed above and below the left eye and bandpass filtered between 1 and 50 Hz. Eyeblinks were detected by vertical EOG signal above the threshold (150-200 μV), and the artifacts induced by the eyeblinks and movements were detected and suppressed by removing the first principal component after applying the PCA algorithm from the Curry 8 software (Neuroscan). EEG signals were further inspected visually, and time periods where large artifacts existed were manually removed. Finally, remaining artifacts were removed by rejecting independent components labeled as horizontal eye movement, excessive muscle noise, and skin potentials from the EEGLAB toolbox (Pion-Tonachini et al., 2019).
Calculating spectral power from EEG signals
We calculated neural power by applying the Morlet wavelet transform between 0.5 and 100 Hz and averaged the power into 29 bins where the center frequency was 2(−0.75+k/4) Hz for the kth bin to account for the fact that power becomes similar to nearby frequencies in the high-frequency domain. The temporal resolution of the moving window was set to 31 ms (32 Hz) for the whole experiment, including fixation, encoding, and retrieval periods. We calculated average power across two frequency bands of interest: slow theta (2.5-5 Hz) and fast theta (5-8.5 Hz). Slow and fast theta ranges were divided based on recent studies suggesting a functional dissociation of slow and fast theta power in human memory (Bush et al., 2017; Miller et al., 2018; Goyal et al., 2020). Power values were baseline z-scored with respect to the preceding fixation period. Trial-level rejection was applied for outliers >3 scaled median absolute deviations away from the median value across the 40 trials of each condition. Response time was normalized from retrieval stimulus onset to the response and divided in half to calculate the average band power in early and late retrieval (see Fig. 3A).
Band power comparison across timing and conditions
First, we compared neural activity in the early and late retrieval periods across the three task conditions (see Fig. 3). A repeated-measures ANOVA compared channel-averaged power in the two frequency bands (slow/fast theta power) separately across the three conditions (what, where, and when) at 80 time points equally interpolated across the retrieval onset and response. Only correct retrieval trials were considered. We found a clear early-late separation in theta power differences across what-where-when conditions (see Fig. 3B), so all subsequent analyses were separately performed for the first and second halves of the retrieval period (from stimulus onset to response). Another three-way repeated-measures ANOVA compared slow/fast theta power across the three conditions (what, where, and when) in the early/late periods. Post hoc t tests were Bonferroni-corrected.
Defining semantic, spatial, and temporal distance
To compute the semantic distance (SD), which has been previously reported to affect the mnemonic process (Cann et al., 2011; Naspi et al., 2021), we used the Wordnet-based Wu & Palmer similarity algorithm (Wu and Palmer, 1994) to leverage its superior performance in describing semantic relationships between objects compared with another commonly used Word2vec algorithm (Saedi et al., 2018). The similarity score ranged from 0 to 1 (e.g., 0.47 for slippers & fish and 0.85 for a bowl and a cup), with a value of 1 assigned to two objects belonging to the same semantic category (e.g., trumpet and horn). We calculated semantic similarity scores for each pair of objects presented in the what retrieval test (similarity scores ranged from 0.22 to 1 with an average of 0.712), then assigned SDs by their rank across the 10 trials within a block (i.e., SDs ranging from 1 from 10) to prevent potential problems arising from the skewed distribution (skewness = −0.483). Therefore, SD of 1 indicated the two objects with the highest lexical similarity (short distance) in the set, while 10 indicated the pair with the lowest similarity (long distance). Based on previous studies showing a critical role of spatial distance in memory retrieval (Lee et al., 2018; Herweg et al., 2020a), Euclidean spatial distance (ED) was calculated by the distance from the original to the moved position (lure) of an object in the scene. EDs were ranked by the EDs in an ascending order (i.e., ED 1: the shortest distance and ED 10: the longest distance). Because TD is known to directly affect performance in temporal order judgment (Jacques et al., 2008), we defined TD by a difference in the order of two scenes during the encoding. It ranged from 1 to 9, since a total of 10 scenes were shown during encoding. A range of raw distances (i.e., before ranked) of 10 scenes (=1 set) was balanced across the four stimuli sets (one-way ANOVA SD: F(3,36) = 1.096, p = 0.363; ED: F(3,36) = 1.045, p = 0.385; TD: F(3,36) = 0.12, p = 0.948). In addition, we verified that the variance across the three distances, normalized by the max-min difference, was not significantly different according to the Bartlett test (χ2(46) = 0.639, p = 0.727).
Perceptual distances during what retrieval
We measured two types of perceptual distance between the target and lure objects during what retrieval to explore their effects on behavior and neural activities: (1) shape and (2) color. To compare shape-related effects, we created a histogram of oriented gradients (HOG), a distribution of intensity gradients or edge directions (Dalal and Triggs, 2005). Color difference was measured by calculating the root-mean-squared difference in the RGB ratio histogram between each pair of objects tested (Swain and Ballard, 1991). As with the other distance metrics in our study, we defined HOG distance (HOGD) and color distance (CD) by their rank across the ten trials of a single block.
Correlation between cognitive distances and retrieval accuracy
We tested whether cognitive distances influenced episodic memory performance by calculating Kendall's tau correlation coefficients between SD, ED, TD, and accuracy in the what, where, and when conditions, respectively. Statistical significance of the averaged correlation coefficient from every subject (n = 47) was tested by performing one-sample t tests against zero after applying the Kolmogorov–Smirnov test for normality (see Fig. 2C,F,I). Similarly, we calculated Kendall's tau correlation coefficients between HOGD, CD, and the what retrieval accuracy for lower SD (≤5) and higher SD (>5) trials separately to find the effects of the perceptual distances.
Correlation between cognitive distances and neural spectral power
To find neural correlates for cognitive distance coding during episodic memory retrieval, we calculated Kendall's tau correlation coefficients between each distance and the corresponding neural spectral power. Correlation coefficients were computed with band power between 2.5 and 70 Hz (for all correct retrieval trials) in each subject, for each cognitive distance (SD, ED, or TD) across 80 normalized time points evenly distributed between retrieval onset and the response (see Fig. 4A–C). To compare early and late retrieval periods, correlations between each cognitive distance and slow theta (2.5-5 Hz) and fast theta (5-8.5 Hz) power were calculated for early and late retrieval periods using channel-averaged power values (see Fig. 4D–I). Statistical significance of the correlation for channel-averaged power was tested by computing correlation coefficients (Kendall's tau) from every subject (n = 47) and comparing it against zero using a one-sample t test (after the Kolmogorov–Smirnov test for normality).
Channel-wise cluster-based permutation test
To assess the channel-wise significance of the correlation between cognitive distances and spectral band power, we used cluster-based permutation tests for the 2D correlation maps (Time × Frequency) in each channel (Maris and Oostenveld, 2007; Damsma et al., 2021; Wen et al., 2022), and then applied Bonferroni correction for multichannel comparisons. First, the largest cluster among the sums of subject-level t statistics (α = 0.05) was identified in four different combinations of time and frequency ranges of interest on the 2D correlation maps, based on the early and late retrieval periods across the two frequency bands of investigation (slow and fast theta). Second, a distribution of surrogate clusters was created in the same manner but using randomly shuffled (1500 times) power and distance pairs. Finally, we compared the surrogate distribution and the largest cluster using t tests, Bonferroni-corrected for the 26 channels (see Fig. 4D–I, topoplot).
Conditional/timing/frequency/regional specificity of neural distance coding
To validate generality or distinctiveness of neural distance coding, correlations between theta power and each distance averaged from the significant channels were compared across three distances (i.e., SD, ED, and TD), two frequency ranges (slow and fast theta), and two halves of the retrieval phase (early and late). Data from all channels were used if significant channels were not found or the average from all channels was not significant. We also tested for regional specificity, with nine parietal EEG channels (C3, CZ, C4, CP3, CPZ, CP4, P3, PZ, and P4) and three occipital EEG channels (O1, OZ, and O2).
Individual differences in distance-dependent retrieval performance and neural distance coding
We investigated how the cognitive distances affect retrieval performance and its individual differences by correlating the behavioral pattern with the neural distance coding. Subjects were divided into two groups based on their accuracy difference between low and high distance trials (above the median: distance-dependent group [n = 23] vs. below the median: distance-independent group [n = 24]). The choice of the median-split method was based on previous episodic memory studies, which divided subjects into two groups by applying the median-split method according to their behavioral performance (Doppelmayr et al., 2005; De Pascalis et al., 2012; Brodt et al., 2016; Auger et al., 2017; Maidenbaum et al., 2018). Then, the neural distance coding, as quantified by Kendall's tau correlation between theta power and the cognitive distances, was compared between the two groups (see Fig. 6). One-sample t tests against zero for each group were performed for the three neural distance markers: (1) domain-general neural distance coding mediated by slow theta power during the late retrieval period, (2) domain-specific spatiotemporal neural distance coding mediated by fast theta power during the late retrieval period, and (3) distinctive TD coding mediated by the early slow theta power.
Results
Study overview
A scene-based episodic memory task (Fig. 1A) based on previous studies (Babb and Johnson, 2010; Park and Lee, 2021) was used to independently test the what, where, and when components of episodic memory. In the encoding period, subjects were asked to remember 10 scenes, each displayed for 3 s. During what retrieval, subjects were asked to choose from two objects the correct one that was in a particular scene; where retrieval required subjects to choose from two scenes in which the spatial location of an object in the scene differed; and when retrieval required subjects to choose from two scenes the one that came first during encoding. They were not informed about the condition in advance. To investigate the neural coding of SD (what), ED (where), and TD (when), we correlated behavioral accuracy and EEG spectral power during retrieval.
What, where, and when episodic memory task. A, B, Scene-based episodic memory task design (Fixation – Encoding – Retrieval). Subjects were asked to memorize all scenes during the encoding and to choose the correct responses based on what, where, or when during the retrieval. C, A schematic diagram of cognitive distances (what: semantic distance; where: spatial Euclidean distance; and when: temporal distance) in episodic memory.
Memory accuracy was correlated with cognitive distance
We examined the functional importance of cognitive distance in the what, where, and when conditions separately. First, we calculated SD (see Materials and Methods) between the pairs of objects given during what retrieval (Fig. 2A). A positive correlation between accuracy and SD was observed in most subjects (Fig. 2B,C, 36 of 47 subjects, average Kendall's tau across subjects = 0.254, one-sample t test vs. zero: t(46) = 7.113 and p < 10−3), which indicates that a longer SD between the test pair increased their discriminability during memory retrieval. For spatial distance, we first found that our initial measure of ED (see Materials and Methods) between the original and moved object locations in the scene did not show a correlation with accuracy (average Kendall's tau across subjects = 0.026, one-sample t test vs. zero: t(46) = 0.626 and p = 0.534). However, considering previous studies which reported that scene-related brain regions were not sensitive to an object's left-right symmetry (Dilks et al., 2011), once we adjusted subjects' tendency to confuse between symmetric locations in the scene, we found that accuracy was strongly correlated with ED in trials in which the object was not moved to the opposite (left-right) side of the scene (Fig. 2D, 21 among 40 scenes) (Fig. 2E,F, average Kendall's tau across subjects = 0.139, one-sample t test vs. zero: t(46) = 3.055 and p = 0.004). This result indicates that, in general, subjects found closer object locations (between the original and test stimuli) more difficult to distinguish, provided that there was no mirror image confusion. Lastly, TD (Fig. 2G) between the two given scenes at retrieval was positively correlated with when accuracy (Fig. 2H,I). A positive correlation was observed in almost every subject (46 of 47 subjects, average Kendall's tau correlation across subjects = 0.519, one-sample t test vs. zero: t(46) = 19.001 and p < 10−3), which indicated that remembering the temporal order of two scenes that occurred closer together in time was more difficult. Overall, the behavioral facilitation of memory across all three distance conditions can be attributed to an increase in representational discriminability in the brain. Next, we investigated which neural markers of episodic memory retrieval were involved in distance coding and how they varied across the what, where, and when components.
Cognitive distance affects memory performance. A, Definition of SD. The distances calculated from 10 scenes (1 block) were converted into ranks in an ascending order (for details, see Materials and Methods). B, Retrieval accuracies across SD. Error bars indicate SE. C, Kendall's tau correlation coefficients between the SD and accuracy were calculated for each subject and compared against zero using one-tailed t tests. D, Definition of ED along with an example of same-side and opposite-side displacement during where retrieval. E, The median of retrieval accuracies across ED for same-side trials only. F, Kendall's tau between the ED and accuracy for same-side trials only. G, Definition of TD. H, Retrieval accuracies across TD. I, Kendall's tau between the TD and accuracy. **p < 0.01. ***p < 0.001.
EEG power differences between what, where, and when components during the memory retrieval
Based on previous research implicating the importance of theta power during episodic memory (Nyhus and Curran, 2010; Lega et al., 2012, 2016), we measured spectral power in the two theta bands (slow theta: 2.5-5 Hz, fast theta: 5-8.5 Hz) during the retrieval of what, where, and when (correct trials only). Also, we divided the retrieval phase into early and late periods based on normalized response times (Fig. 3A), given that a difference across the conditions (i.e., what, where, and when) in both slow and fast theta power was characterized by two peaks in the early and late periods (Fig. 3B, repeated-measures ANOVA at each of the 80 time points interpolated between the retrieval onset and response, see Materials and Methods). An ANOVA comparing the 2 frequency bands (slow and fast theta) in the 2 halves of the retrieval (timing; early and late periods), across the 3 conditions (i.e., what, where, and when), revealed a main effect of frequency band (F(1,46) = 101.59, p < 10−3), with slow theta power significantly larger than fast theta (t(46) = 10.079, p < 10−3). Next, we found significant interactions between timing and frequency band (F(1,46) = 57.075, p < 10−3) and between condition and frequency band (F(2,92) = 3.696, p = 0.029). ANOVAs performed separately for each theta band showed that slow theta power was not significantly different across what, where, and when conditions (Fig. 3C, F(2,92) = 2.648, p = 0.076) but was significantly larger in the late half of the retrieval period compared with the first half (Fig. 3E, early vs. late F(1,46) = 11.678, p = 0.001). The increase in slow theta power was significant only in the where and when conditions (paired t test early vs. late, what: t(46) = 1.622, p = 0.336; where: t(46) = 2.985, p = 0.014; when: t(46) = 3.262, p = 0.006, Bonferroni-corrected). The recruitment of slow theta across the retrieval period confirms previous studies that suggest its involvement in general episodic memory processes (Lega et al., 2012; Goyal et al., 2020). The significant increase in slow theta across the retrieval period in the where and when conditions compared with the weaker increase in the what condition (Fig. 3E) may reflect a more extended memory retrieval process for where and when information.
Comparisons of neural activities across what, where, and when conditions in two frequency bands in the 2 halves of the retrieval. A, The averaged response time was 2.4 s out of a maximum of 5 s. We divided the retrieval phase into early and late periods based on normalized response times from the retrieval stimulus onset to the response. B, The difference among the conditions in slow and fast theta power across the retrieval period was compared (repeated-measures ANOVA); the F values showed two separate peaks in the early and late periods. C, D, Band power comparisons in both slow (2.5-5 Hz) and fast (5-8.5 Hz) theta ranges averaged across all channels. Asterisks indicate significant pairwise comparisons. Error bars indicate SE. ***p < 0.001. E, F, A comparison of slow and fast theta power averaged across all channels between early and late etrieval periods in the what, where, and when conditions. Red-filled circles represent significant differences (p < 0.05, pairwise t tests) between early and late periods. Error bars indicate SE.
Fast theta power differed significantly across the three conditions (F(2,92) = 17.660, p < 10−3), with when and where conditions larger than the what condition (Fig. 3D, paired t tests, Bonferroni-corrected, where vs. what: t(46) = 4.579, p < 10−3 and when vs. what: t(46) = 5.187, p < 10−3). Fast theta power also decreased overall in the second half of the retrieval period (Fig. 3F, early vs. late F(1,46) = 5.901, p = 0.019) but was mainly driven by the drastic decrease in power for the what condition (condition-timing interaction: F(2,92) = 5.639, p = 0.005; paired t test early vs. late, what: t(46) = −3.901, p < 10−3; where: t(46) = −1.173, p = 0.741; when: t(46) = 1.928, p = 0.180, Bonferroni-corrected). The significantly larger fast theta power and its maintenance across late retrieval in the where and when conditions converge on previous findings suggesting its importance in navigation and spatial memory (Bush et al., 2017; Miller et al., 2018; Goyal et al., 2020). Our finding that when and where retrieval shows greater recruitment of both ranges of theta in our study may indicate the engagement of long-range communications among brain regions, as reported in other studies (Von Stein and Sarnthein, 2000; Nyhus and Curran, 2010; Hasselmo and Stern, 2014; Herweg et al., 2020b).
Slow theta power codes domain-general cognitive distance
Given its prevalence during memory retrieval, we hypothesized that slow theta oscillations may be involved in coding cognitive distance. Looking at the three distance conditions (what, where, and when) separately in a wide frequency range (2.5-70 Hz) across the retrieval period (Fig. 4A–C), we found that theta power during late retrieval was indeed significantly correlated with all three distances in positive direction. When divided into slow and fast theta bands, we found this effect to be true particularly in slow theta range, averaged across all EEG channels (Fig. 4G–J) (SD: average Kendall's tau = 0.056, t(46) = 3.603, p = 0.001; ED: average Kendall's tau = 0.030, t(46) = 2.091, p = 0.042; TD: average Kendall's tau = 0.024, t(46) = 1.714, p = 0.093). When we applied Bonferroni-corrected cluster-based permutation tests in each EEG channel, a significant correlation between slow theta and all three cognitive distances was predominantly found in the parietal channels (Fig. 4G–I,M) (paired t test comparing parietal vs. other channels on the distance-slow theta correlation averaged for all distances: t(46) = 2.649, p = 0.011, Fig. 4K). An ANOVA showed no differences across distances in their correlation with slow theta power (F(2,92) = 0.555 and p = 0.576), and all correlations were significantly larger in the late retrieval period than in the early period (late vs. early SD: t(46) = 3.4493 and p = 0.001; ED: t(46) = 2.643 and p = 0.011; TD: t(46) = 5.976 and p < 10−3, Fig. 4J). In summary, we found a common neural correlate for what, where, when distance coding mediated by slow theta in the parietal channels during the late retrieval period, indicating a potential neurophysiological mechanism for domain-general distance coding. In contrast to these abstract cognitive distances (SD, ED, and TD), perceptual features such as object shape (HOGD) and color (CD) did not show any neural distance coding by theta power (Fig. 5E–H, HOGD: t(46) = 1.654, p = 0.105 (slow-early), t(46) = −0.241, p = 0.81 (slow-late), t(46) = −0.539, p = 0.593 (fast-early), t(46) = 1.016, p = 0.315 (fast-late); CD: t(46) = −0.314, p = 0.755 (slow-early), t(46) = 1.1, p = 0.277 (slow-late), t(46) = −0.441, p = 0.661 (fast-early), t(46) = −0.248, p = 0.806 (fast-late)), despite their relationship with what retrieval performance in trials with SD <6 (HOGD: averaged Kendall's tau = 0.331, t test against zero: t(46) = 9.156, p < 10−3; CD: averaged Kendall's tau = −0.27, t test against zero: t(46) = −6.753, p < 10−3, Fig. 5A,C).
A correlation between EEG band power and cognitive distance. A-C, Time-frequency plots of Kendall's tau correlation coefficients between band power and each cognitive distance from the retrieval stimulus onset to the response. Black contour lines indicate the time-frequency ranges (>10 units wide) of significant correlations (p < 0.05, one-sample t test vs. zero). D-F, Slow and fast theta power in the early retrieval period across SD, ED, and TD. Circles represent averaged power across all EEG channels. Error bars indicate SE. Colored circle represents p < 0.1. Red trend line indicates p < 0.05. Channel-wise significance for the Kendall's tau correlation coefficients based on t tests against zero is indicated on the topoplots. Filled circle represents p < 0.05, corrected by cluster-based permutation tests with Bonferroni correction. G-I, Averaged slow and fast theta power for SD, ED, and TD in the late retrieval period and topoplots with marked significant channels. J, Bar plot represents results of direct comparisons of neural distance coding across conditions, frequencies, and periods. K, A comparison of domain-general distance coding averaged for all distances between parietal and other channels. L, A comparison of spatiotemporal coding across channels (parietal and occipital) and conditions (ED and TD). M, A schematic overhead view of the configuration of 26 EEG channels. Each channel was allocated to one of four cortical regions (frontal, parietal, temporal, or occipital).
Two types of perceptual distances (shape and color) during what retrieval and their effects on behavior and neural activities. A-D, Retrieval accuracies across HOGD and CD in both lower SD (SD ≤ 5) and higher SD (SD > 5) trials. A significant correlation between distance and retrieval accuracy was observed for HOGD and CD only in lower SD trials. Error bars indicate SE. Red trend lines indicate significant correlations (p < 0.05). E-H, Neural coding of perceptual distances (HOGD and CD) was measured by calculating Kendall's taus between slow/fast theta power and the distances in the early and late retrieval periods. Unlike SD, which showed a positive correlation with slow theta power in the late retrieval period (Fig. 4G), neither HOGD nor CD was correlated with slow/fast theta in the early/late retrieval periods.
Distinctive neural correlates of cognitive distance coding
In addition to the common neural representation of cognitive distance as discussed above, we also found distinctive theta-modulated correlates for each of the what, where, and when conditions. First, fast theta power was significantly correlated with ED and TD but not with SD in the late retrieval period (Fig. 4H,I, SD: average Kendall's tau = 0.011, t(46) = 0.916 and p = 0.364; ED: average Kendall's tau = 0.035, t(46) = 2.104 and p = 0.041; TD: average Kendall's tau = 0.038, t(46) = 2.554 and p = 0.014). A comparison of correlations across distances (ANOVA) was significant (Fig. 4J, F(2,92) = 3.589 and p = 0.032, averaged from significant channels) and larger for ED and TD compared with SD (paired t test ED vs. SD: t(46) = 2.862 and p = 0.006; TD vs. SD: t(46) = 2.099 and p = 0.041). We found a significant interaction effect between region and distance (Fig. 4L, F(1,46) = 7.789 and p = 0.008); compared with TD coding, ED coding by fast theta was found in the occipital rather than the parietal channels (occipital vs. parietal t(46) = 2.094 and p = 0.042). The lack of significant correlation in the what condition was consistent with the finding of significantly weaker fast theta power in the what condition compared with the where and when conditions (Figs. 3D, 4G). Meanwhile, fast theta power was not only larger for where and when retrieval (compared with what) but also correlated with their cognitive distance (ED and TD). These results show, for the first time, that fast theta is not only induced in spatial information processing, as reported in previous studies (Goyal et al., 2020), but that it codes detailed information about both SD and TD.
Second, we identified a unique neural correlate of TD by slow theta power in the early retrieval period (Fig. 4F, average Kendall's tau = −0.051, t(46) = −3.338 and p = 0.001), which was significantly larger (in terms of the absolute value given the negative correlation) than SD and ED (Fig. 4J, paired t tests, TD vs. SD: t(46) = 3.453 and p = 0.001; TD vs. ED: t(46) = 3.046 and p = 0.004). In a few of the channels, there was a significant negative correlation between TD and fast theta power in the early retrieval period, but the all-channel averaged fast theta power was not significantly correlated with TD (Fig. 4F, average Kendall's tau = −0.021, t(46) = 1.366 and p = 0.178). The other two distances (SD and ED) did not show any correlated neural activity in the early retrieval period (Fig. 4D,E, SD: average Kendall's tau = 0.022, t(46) = −1.572 and p = 0.123; ED: average Kendall's tau = −0.005, t(46) = −0.339 and p = 0.736). This exclusive TD coding in the early retrieval period may reflect a unique set of cognitive computations involved in the temporal organization of memory.
Distance-dependent performance was associated with stronger distance-dependent theta power
To test whether the engagement of theta-modulated distance coding is indicative of distance-dependent behavioral performance, we divided subjects into two groups based on their accuracy difference between high and low distance trials for each of the what, where, and when conditions (Fig. 6A,B; median split into distance-dependent and distance-independent groups, Fig. 5C). When the theta-modulated distance markers were compared between the two groups (Fig. 6D–L), we found that subjects with a greater tendency for distance-dependent performance showed a greater engagement of domain-general neural distance coding (late slow theta power) across all three conditions (Fig. 6D–F, one-sample t tests against zero SD: t(24) = 3.634, p = 0.001; ED: t(24) = 2.197, p = 0.038; TD: t(24) = 1.835, p = 0.079). Furthermore, the spatiotemporal distance coding in the fast theta range was associated with distance-dependent performance in the where and when conditions only (Fig. 6H,I, one-sample t tests against zero ED: t(24) = 2.21, p = 0.037; TD: t(24) = 2.713, p= 0.012). Lastly, the distinct neural correlate of TD (early slow theta power) was associated specifically with distance-dependent when accuracy (Fig. 6L, TD: t(24) = −2.818, p = 0.01). On the other hand, subjects whose behavioral performance was distance-independent did not show neural distance coding in any domain (Fig. 6). In summary, we found that the engagement of a particular cortical neural marker of cognitive distance also showed corresponding distance dependence in memory performance.
Distance-dependent performance was associated with neural coding of cognitive distance. A-C, Each subject's retrieval accuracy difference between high and low distance was used to divide subjects into two groups: a distance-dependent group (above median, n = 23) and a distance-independent group (below the median, n = 24). D-L, A comparison between distance-dependent and distance-independent groups for general distance coding (D-F), spatiotemporal coding (G-I), and distinct TD coding (J-L) represented by Kendall's tau correlation between cognitive distances and late slow theta, late fast theta, or early slow theta power. First row (D,G,J): SD; second row (E,H,K): ED; and third row (F,I,L): TD. †p < 0.1. *p < 0.05. **p < 0.01.
Discussion
For the first time in a single study, we investigated the existence of both shared and unique neurocognitive representations of semantic distance, spatial distance, and temporal distance during memory retrieval. We reported both domain-general and domain-specific characteristics of neural correlates for distance coding mediated by theta power. We found that slow theta power was commonly induced for what, where, and when components during retrieval, while fast theta power was specifically induced for where and when components (Fig. 3A,B).
Our finding that slow theta power was positively correlated with all three cognitive distances may reflect a process of domain-general distance coding during memory retrieval. Previous studies reported that hippocampal slow theta (∼3 Hz) power increased with spatial distance traveled during virtual navigation (Vass et al., 2016; Bush et al., 2017). Similarly, in our task, longer cognitive distance induced larger theta power in the parietal EEG channels (Fig. 4G–I). The fact that these effects were localized to the parietal channels converges on decades of research on shared representations of magnitude (e.g., space, time, number) in the parietal lobe (Dehaene and Brannon, 2011). We thus propose that these results may be an electrophysiological correlate of parietal magnitude representations that contribute to the formation of a generalized cognitive map proposed recently in neuroimaging studies (Epstein et al., 2017; Behrens et al., 2018; Spiers, 2020).
The specific recruitment of fast theta for where and when components of episodic memory in our study is largely reminiscent of past findings that it is involved in spatial processing (Miller et al., 2018; Goyal et al., 2020). We extend such claims to include temporal information and suggest that fast theta may either be involved in contextual processing (compatible with a generalized view of hippocampal function) (Eichenbaum et al., 1999, 2007; Deuker et al., 2016) or, alternatively, is specific to navigation which is spatiotemporal in nature (Burgess et al., 2002). Furthermore, not only is fast theta simply increased in power during such processes, it also codes information about spatial ED and TD. Unlike the domain-general slow theta distance coding that was commonly found in the parietal channels, spatial and temporal fast theta distance coding was found in distinctive regions in the occipital and parietal channels, respectively. This may be related to the involvement of the occipital cortex in the visual processing of spatial information (Kamps et al., 2016) during where retrieval and the parietal region for temporal order processing (Foudil et al., 2020) during when retrieval. The activity of these domain-specific cortical mechanisms may be coordinated by the hippocampus for accurate memory recall (Jacques et al., 2008; Baldassano et al., 2015).
The unique neural correlate of TD in the early retrieval period may be related to the fact that when memory, compared with the what and where memory, requires an entirely contextual (i.e., relative sequential order) process rather than detailed processing of perceptual features. The negative correlation between TD and theta power may be explained in two ways: The first is that larger theta power was required on more difficult trials (i.e., closer distance trials), which required a higher resolution of the temporal cognitive map stored in memory. However, since the comparison between correct trials (relatively easy trials) and incorrect trials (relatively hard trials) did not show any difference in slow theta power during the late retrieval (what: t(46) = 0.358 and p = 0.722; where: t(46) = 0.667 and p = 0.508; when: t(46) = −0.068 and p = 0.946), it is not likely to be driven by a level of difficulty. The second potential interpretation is that, given numerous previous studies showing increased theta power for hippocampal associative memory retrieval (Miller et al., 2018; Kota et al., 2020) and negative correlation between hippocampal theta power and TD (Solomon et al., 2019), larger theta power in the short distance trials in our study might be because of a higher similarity of neural representation between closely associated events. Similar patterns of theta TD coding in frontal and parietal channels may therefore be indicative of a hippocampus-driven cortical memory reinstatement in the early stages of retrieval (Sestieri et al., 2017; Staresina and Wimber, 2019).
Neural distance coding may be a mechanism that facilitates the efficient retrieval of multiple memories that vary widely along a set of conceptual or perceptual features. However, not every subject showed better performance in the high distance trials. How did subjects in the distance-independent group retrieve memory to a high level of accuracy? One possibility is that subjects distinguished between scenes based on multiple features, in addition to the ones included in the design of our study. The episodic memory task in this study did not restrict subjects to use a specific retrieval strategy, meaning that subjects could have used more than one cognitive distance (i.e., what condition based on perceptual and semantic distances). We actually found that the color and shape features correlated with performance in what retrieval when SD was lower, in which case the utilization of semantic information for separating a lure from the target object was not effective (Fig. 5A,C). A significant correlation between perceptual distance and retrieval accuracy was not observed in high SD trials (Fig. 5B,D). Domain-general cognitive mapping may occur with respect to a variety of cognitive distances. Other cognitive distances may have still contributed to domain-general cognitive mapping, presumably also with the potential for slow-theta modulation.
A limitation of this study is that scalp EEG signals mostly stem from cortical neural activities and thus do not provide access to deep brain regions. Further studies correlating these EEG signals with hippocampal activity may help overcome these limitations. Another limitation of this study was that, despite the existence of individual differences in both domain-general and domain-specific correlates of episodic memory, we did not conduct such tests in the present analysis. Future studies will explore the potential usefulness of these neural markers in predicting individual strengths and weaknesses in memory ability. In particular, the distinctive neural correlates of spatial and temporal memory may serve as a potential marker for detecting early memory impairments or susceptibility to neurological disorders, such as Alzheimer's disease.
Footnotes
This work was supported by National Research Foundation of Korea 2021M3E5D2A01023891 and 2020K1A3A1A19088932; and Seoul National University Creative-Pioneering Researchers Program. We thank Jin-hyuck Park, Heeso Kim, and Dongho Shin for experimental assistance.
The authors declare no competing financial interests.
- Correspondence should be addressed to Sang Ah Lee at sangahlee{at}snu.ac.kr
