Abstract
The hippocampus plays a central role as a coordinate system or index of information stored in neocortical loci. Nonetheless, it remains unclear how hippocampal processes integrate with cortical information to facilitate successful memory encoding. Thus, the goal of the current study was to identify specific hippocampal–cortical interactions that support object encoding. We collected fMRI data while 19 human participants (7 female and 12 male) encoded images of real-world objects and tested their memory for object concepts and image exemplars (i.e., conceptual and perceptual memory). Representational similarity analysis revealed robust representations of visual and semantic information in canonical visual (e.g., occipital cortex) and semantic (e.g., angular gyrus) regions in the cortex, but not in the hippocampus. Critically, hippocampal functions modulated the mnemonic impact of cortical representations that are most pertinent to future memory demands, or transfer-appropriate representations. Subsequent perceptual memory was best predicted by the strength of visual representations in ventromedial occipital cortex in coordination with hippocampal activity and pattern information during encoding. In parallel, subsequent conceptual memory was best predicted by the strength of semantic representations in left inferior frontal gyrus and angular gyrus in coordination with either hippocampal activity or semantic representational strength during encoding. We found no evidence for transfer-incongruent hippocampal–cortical interactions supporting subsequent memory (i.e., no hippocampal interactions with cortical visual/semantic representations supported conceptual/perceptual memory). Collectively, these results suggest that diverse hippocampal functions flexibly modulate cortical representations of object properties to satisfy distinct future memory demands.
Significance Statement The hippocampus is theorized to index pieces of information stored throughout the cortex to support episodic memory. Yet how hippocampal processes integrate with cortical representation of stimulus information remains unclear. Using fMRI, we examined various forms of hippocampal–cortical interactions during object encoding in relation to subsequent performance on conceptual and perceptual memory tests. Our results revealed novel hippocampal–cortical interactions that utilize semantic and visual representations in transfer-appropriate manners: conceptual memory supported by hippocampal modulation of frontoparietal semantic representations, and perceptual memory supported by hippocampal modulation of occipital visual representations. These findings provide important insights into the neural mechanisms underlying the formation of information-rich episodic memory and underscore the value of studying the flexible interplay between brain regions for complex cognition.
Introduction
Suppose we encounter a giant panda during a zoo visit, which combines perceptual details—such as the panda's black-and-white coat and posture—with our conceptual knowledge—such as pandas’ bamboo diet. These disparate forms of information collectively form our “panda encounter” episode, whose subsequent recollection depends on an organized interaction of a network of brain regions. The hippocampal memory indexing theory (Teyler and DiScenna, 1986) posits that the hippocampus stores links to heterogeneous informational content represented by disparate cortical regions, and reactivates those cortical representations for the subsequent recollection of a complete episode. Consistent with this model are two well-known findings: (1) the observation that the hippocampus shows greater activation during the encoding of stimuli that were subsequently remembered than forgotten (Davachi et al., 2003; Ranganath et al., 2004) and (2) the observation that a range of cortical representations revealed by multivariate pattern analysis support a hierarchy of object representation, from early occipital areas representing low-level visual details like orientation and color (Devereux et al., 2018; Martin et al., 2018) to more anterior regions representing abstract conceptual information like taxonomy and semantic relatedness (Clarke and Tyler, 2014; Martin et al., 2018). Meanwhile, multivariate activity patterns in the hippocampus often demonstrate low sensitivity to visual and semantic details (LaRocque et al., 2013; Liang et al., 2013) and low ability to decode stimulus identity (Diana et al., 2008; Huffman and Stark, 2014). However, some studies did find that hippocampal activity patterns (Davis et al., 2014, 2021; Naspi et al., 2021; Poh et al., 2022) are predictive of subsequent memory. Given evidence of both cortical representations for diverse stimulus details and mnemonic relevance of various measures of hippocampal engagement at encoding, it is likely that the interactions between hippocampal processes and cortical representations of informational content, on top of individual components, are essential to successful memory encoding. Such interactions are the focus of the current study.
We hypothesized that such hippocampal–cortical interactions follow a general transfer-appropriate heuristic, such that the relationship between these interactions and subsequent memory will depend on the relevance of represented information to the memory decision. According to the transfer-appropriate processing principle, memory performance is improved when the encoding and retrieval tasks involve similar cognitive processes (Morris et al., 1977). An open question, however, is whether the hippocampus indexes cortical representations in a manner appropriate to their subsequent retrieval. Therefore, we focus on hippocampal–cortical interactions that exhibit transfer-appropriate representations; specifically, stronger cortical representations of semantic features (e.g., “eats bamboo”) benefiting conceptual memory, and stronger cortical representations of visual features (e.g., black-and-white coat) benefiting perceptual memory.
We predicted that the hippocampus would interact in parallel with multiple cortical regions indexing transfer-appropriate representations, and that these interactions would be influenced by the strength and specificity of the hippocampal response. We therefore assessed a range of univariate and multivariate measures of hippocampal function, all of which have been found to demonstrate subsequent memory effects, including univariate activation level (Davachi et al., 2003; Ranganath et al., 2004), neural pattern similarity (Davis et al., 2014; Poh et al., 2022), and representational strength for visual or semantic features (Davis et al., 2021; Naspi et al., 2021). Notably, even though the multivariate measures may not robustly discriminate encoding contents, they do appear to be related to encoding success, suggesting that the informational contents of our experiences are perhaps centrally organized by the hippocampus. Therefore, we applied these measures to predict subsequent conceptual and perceptual memory performance at the level of individual trials using two sets of linear mixed-effects regression models to explore how patterns of hippocampal–cortical interactions supporting memory of everyday objects in a transfer-appropriate manner.
Materials and Methods
Experimental design
Details of the experimental paradigm were first reported in Davis et al. (2021). Here, we report data from 19 neurologically typical adult participants (7 females, age = 23.08 ± 2.73 years, range = 19–29, all native English speakers), in accordance with a protocol approved by the Duke University Health System Institutional Review Board. We used object images spanning across 12 categories of both living (e.g., birds, mammals) and nonliving (e.g., buildings, furniture) things (see Fig. 1C for a subset of images) selected from a published stimulus set (https://mariamh.shinyapps.io/dinolabobjects/; Hovhannisyan et al., 2021). The 12 object categories were balanced for frequency based on the Corpus of Contemporary American English (Davies, 2008).
A, Experimental paradigm. On Day 1 in the scanner, participants covertly named 360 real-world objects spanning across 12 categories (bird, building, clothing item, food, fruit, furniture, mammal, musical instrument, street item, tool, vegetable, and vehicle). On Day 2, participants completed two memory tests serially. During the Conceptual Memory test, participants determined whether the presented word refers to objects that were previously shown on Day 1 or novel. During the Perceptual Memory test, participants determined whether the presented image stimuli were identical to what were presented on Day 1 or novel, non-identical exemplar images of previously shown objects (e.g., the white bowtie image, a novel panda image). Oldness rating responses comprised: 1 = “definitely new”, 2 = “probably new”, 3 = “probably old”, and 4 = “definitely old”. B, Behavioral results. Mean Oldness ratings of old and new objects during the Conceptual (top) and Perceptual (bottom) Memory tests are plotted for each participant. Participants are ordered based on their mean rating difference for old and new objects on the Conceptual Memory test—here indicated by the length of the black vertical line linking mean responses for old and new objects. This difference is consistent across subjects in the Perceptual Memory test. Density distributions of participant-level means are plotted to the right. C, Similarity structures of example stimuli. Nonmetric multidimensional scaling (MDS) plots illustrating the similarity structures of a subset of 60 objects (five per category) according to their semantic properties (top) quantified based on encyclopedic features ascribed to each concept by human participants in a separate study (Hovhannisyan et al., 2021) and visual characteristics (bottom) quantified based on an early convolutional layer from the pre-trained VGG16 neural network model.
The two-session experimental paradigm aimed at examining participants’ memory for both the concept and the exemplar image of real-world objects after a delay of 20–28 h (Fig. 1A). During the first session, participants completed two encoding runs whereupon they covertly named 360 real-world images of objects in total. All objects were presented once without repetitions, and all categories were evenly distributed between runs. Each encoding run lasted approximately 13 min and comprised 180 trials: in each trial, a fixation cross was shown for 500 ms, followed by a single letter (e.g., “P”) for 250 ms, an image (e.g., panda) for 500 ms, and a blank response screen for 2–7 s (i.e., inter-trial interval). Participants were instructed to press a button if the preceding letter did not match the initial letter of any label of the depicted object, which was the case on a total of 60 catch trials. These catch trials were excluded from subsequent analyses. Participants also indicated via keypress when they did not retrieve the object name, and we further excluded from subsequent analyses trials in which participants incorrectly judged the letter as a “mismatch” and trials in which participants did not retrieve the object name (mean 25.1 out of 300 trials across participants, or 8%). During the second session, participants were tested on their memory for both the object concepts and image exemplars (Davis et al., 2021). The Conceptual Memory test took place in the scanner while participants were presented with word labels for old (n = 300) and new objects (n = 100) not previously seen. After the scan, participants completed the Perceptual Memory test, in which they were presented with object images that either was previously seen (n = 210) or was a novel exemplar of an old object concept (n = 90); as this later test was explicitly focused on memory for perceptual detail, no new concepts were presented. For both Conceptual and Perceptual Memory tests, participants had 3 s to respond (1 = “definitely new”, 2 = “probably new”, 3 = “probably old”, and 4 = “definitely old”), and timing parameters were identical for both memory tests. To correct for potential bias related to the unequal proportions of old and new objects, responses on this 4-point scale were adjusted by first estimating a “false alarm tendency” value for each participant (average response to all new objects on the 4-point scale), and subtracting this value from each participant's response to the old objects. Adjusted responses were treated as continuous variables and mean-centered at the group-level average. Separate corrections were performed for Conceptual and Perceptual Memory tests.
MRI acquisition
All scan sessions occurred at the Duke University Brain Imaging and Analysis Center with a 3T GE MR 750 scanner. Anatomical scans were acquired using a 3D T1-weighted echo-planar sequence (68 slices, 256 × 256 matrix, in-plane resolution 2 × 2 mm2, 1.9 mm slice thickness, TR = 12 ms, TE = 5 ms, FOV = 24 cm). On each run, following a brief calibration, functional images were acquired using an inverse spiral sequence (37 axial slices, 64 × 64 matrix, in-plane resolution 4 × 4 mm2, 3.8 mm slice thickness, flip angle = 77°, TR = 2,000 ms, TE = 31 ms, FOV = 24 cm). Participants wore MRI-compatible lenses to correct vision when necessary.
Data preprocessing was performed using SPM12 and custom MATLAB scripts. Functional images were realigned to the first image of the first run using rigid-body transformation with six motion parameters. Functional images were then corrected for slice acquisition time (reference slice = first slice) and linear signal drift, temporally smoothed using a high-pass filter of 190 s, segmented into different tissue types (gray matter, white matter, and cerebrospinal fluid), and normalized to the MNI152 standard space. Automatic ICA-denoizing was then applied to remove artifacts due to motion or susceptibility artifacts.
For each gray matter voxel, the activity estimate on each encoding trial was obtained by constructing a first-level general linear model with a separate regressor for each trial. The effects of head motion, button presses, white matter signals, and cerebrospinal fluid signals were regressed out from the models.
Statistical analysis
Visual and semantic similarity
Visual similarity between object pairs was determined by correlating activation vectors from the second convolutional layer of VGG16 (Simonyan and Zisserman, 2015), a deep convolutional neural network pre-trained for image recognition. The second convolutional layer was chosen to extract low-level visual features from our images, consistent with several past studies of visual representation (Bone et al., 2020; Davis et al., 2021; Naspi et al., 2023). Principal component analysis was conducted to reduce the influence of non-informative channels (e.g., those associated with peripheral pixels in the white background); component scores for each object were correlated using Pearson's correlation weighted by the eigenvalues of principal components to create a 300-by-300 visual similarity matrix. Semantic similarity of object pairs was based on a normative database (Hovhannisyan et al., 2021) of 448 unique encyclopedic features (e.g., “a sombrero is from Mexico”, “a manatee does live in the ocean”), which cannot be directly observed from the exemplar pictures and thus reflect prior knowledge of the objects from some form of learning. Pairwise similarity between objects was computed using cosine similarity of the semantic feature vectors, resulting in a 300-by-300 semantic similarity matrix (Fig. 2A). Nonmetric multidimensional scaling plots were visually inspected to verify that these two matrices indeed reflect the similarity of objects based on visual and semantic properties (Fig. 1C). For each participant and our regions of interest, we then computed three types of measures at the level of individual experimental trials, i.e., corresponding to a unique stimulus object.
Analytical methods. A, Illustration of the computation of activation level, neural pattern similarity, and visual and semantic representational strengths for the hippocampus. The pairwise correlations were low. B, Example linear mixed-effects model predicting subsequent memory of individual exemplar images (Perceptual Memory), with fixed effects of three measures of hippocampal (Hipp) engagement (activation level, b; neural pattern similarity, r; visual representational strength, ρ), visual representational strength in the ventromedial occipital cortex (vMOC), and all two-way hippocampal–cortical interactions. Conceptual Memory was included as a regressor of no interest. Participants and objects were included as crossed random effects. C, To aid interpretation of possible effects of hippocampal and cortical effects on subsequent memory performance, with cortical representation plotted on the x-axis (width), hippocampal engagement on the y-axis (depth), and subsequent memory performance on the z-axis (height). Three plausible forms of hippocampal–cortical effects are proposed. Left, additive effects; middle, hippocampal gating; right, hippocampal–cortical coordination. Black dashed lines illustrate the effect of cortical representation at low levels of hippocampal involvement and grey dotted lines highlight the effect at high levels of hippocampal involvement.
Trial-level activation level
Masks for different brain regions were created using the Brainnetome atlas (Fan et al., 2016), with 46 cortical regions of interest (ROIs) and one bilateral hippocampus; in a follow-up analysis, we also examined the left and right hippocampus separately (Table 1-1). For each participant, trial-level activation level was computed as the average of activity estimates (betas) for a given object stimulus on a single trial across voxels within an ROI mask (Fig. 2A, red).
Visual and semantic representational strengths in cortical and hippocampal regions
Table 1-1
Cortical and hippocampal regions of interest. Download Table 1-1, DOCX file.
Trial-level neural pattern similarity
For each region, in each participant, multi-voxel activity (beta) patterns for each trial were used to compute the pairwise similarity between objects using Spearman's rank correlation, resulting in a square correlation matrix with its row and column dimensions equal to the number of objects; this matrix is also symmetrical along the diagonal of 1 s reflecting maximal similarity between an object and itself. Of note, diagonal entries were omitted in all subsequent analyses. Given generally high temporal autocorrelation in hemodynamic responses, trials that occurred closer in time may have inflated similarity estimates than trials that were farther apart. As such, we computed the temporal proximity (in seconds) between all pairs of objects presented within the same run and removed this effect from the raw similarity estimates using a linear regression; no such correction was performed for the pairwise similarity estimates between objects presented in different runs since they were not affected by temporal autocorrelation. The resultant neural similarity matrix reflects the extent to which the multi-voxel activity patterns in a given brain region of a participant are similar for any pair of object stimuli. Then, a trial-level measure of neural pattern similarity was computed for the hippocampus as the average of all pairwise similarity values between one object and all other objects, i.e., the row-wise average of all off-diagonal entries in the neural similarity matrix (Fig. 2A, green).
Trial-level representational strength
A trial-level representational strength measure was calculated following the approach outlined in Davis et al. (2021). For each object, we extracted its corresponding row vectors from the visual and semantic similarity matrices, as well as from the neural similarity matrix of a given brain region of a participant. Then, the neural similarity vector was correlated with the semantic and visual similarity vectors, separately using Spearman's rank correlation. These correlation coefficients thus reflect the extent to which a given brain region's neural activity pattern reflects the semantic (Fig. 2A, tan) or visual properties of the object (Fig. 2A, blue) and are thus termed representational strength.
Single-region analyses
Brain regions that represented visual or semantic properties of the stimuli during encoding were localized by fitting a series of linear mixed-effects models using the “lme4” package (Bates et al., 2015) in R (version 4.3.1). Each model predicted either visual or semantic representational strength in a given brain region with fixed effects of trial-level Conceptual and Perceptual Memory scores, as well as their product term. Participants and stimuli were included as crossed random intercepts. The fitted intercept reflected the representational strength of visual or semantic information independent of subsequent memory effects. We used a threshold of p < 0.005 for a rather liberal selection of representational regions.
Hippocampal–cortical interactions
To test the hippocampal–cortical interaction effect on subsequent memory of individual stimuli, as well as to adjudicate among different measures of hippocampal modulation in that process, we performed a series of analyses on each of the candidate regions that represented visual or semantic properties during encoding (Table 1 and Fig. 3), as specified in the previous section. Two separate sets of linear mixed-effects models were fitted for each candidate region: first, a set of transfer-appropriate models (i.e., using semantic information represented in a semantic region predict Conceptual Memory, or using visual information represented in a visual region to predict Perceptual Memory) was tested. For instance, one model predicted subsequent Conceptual Memory of individual objects using the semantic representational strength in right angular gyrus (AnG), as well as all of its two-way interactions with each hippocampal measure (i.e., activation, pattern similarity, and semantic representational strength; Fig. 2B). Additionally, because participants’ performance on the two memory tests were positively correlated, the subsequent Perceptual Memory of each object was included as a regressor to ensure that the estimated effects of brain measures were specific to Conceptual Memory. Second, a set of transfer-incongruent models were evaluated, in which the performance measures on two memory tests were swapped from the transfer-appropriate models. This second set of models addresses the possibility that visual information may have contributed to subsequent Conceptual Memory, or that semantic information may have contributed to subsequent Perceptual Memory. In all transfer-appropriate and transfer-incongruent models, all brain measures were standardized, and random intercepts were estimated for participants and for objects. To mitigate Type I errors, all significant fixed effects were further verified using a residual-based permutation test framework (Winkler et al., 2014; Buzkova, 2016), which consists of five procedures: (i) fit the full model to predict the original dependent variable, Y, and obtain the actual test statistic, t-stat, associated with the effect of interest, X; (2) fit a reduced model (leaving out X) to predict Y and obtain the residuals; (3) fit the full model to predict randomly permuted residuals and obtain the t-stat associated with the effect of interest; (4) repeat the previous step 1,000 times to generate an empirical distribution of t-stats under the null hypothesis; and (5) compare the actual t-stat to said distribution to compute an empirical p-value. Of note, in step (3) the residuals were permuted within participants in step (3) to preserve individual variability that renders the observations not independent (Winkler et al., 2015).
Cortical regions representing visual and semantic properties of the stimuli at encoding, independently of subsequent memory. Semantic representational regions shown in cool colors: left inferior frontal gyrus, left posterior superior temporal sulcus, left and right fusiform gyrus, left and right angular gyrus, right perirhinal cortex, right parahippocampal gyrus, and lateral occipital cortex. Visual representational regions shown in warm colors: ventromedial occipital cortex, lateral occipital cortex, and left fusiform gyrus.
The results of the above regression models focus on the role of specific hippocampal–cortical interactions in support of subsequent memory, interactions which may take multiple theoretically relevant forms. We speculate three plausible outcomes. First, cortical representations and some hippocampal function (e.g., activation) may make independent contributions to subsequent memory formation, such that memory would be stronger for objects in a linear additive fashion (Fig. 2C, left). Second, the hippocampus may “gate” the encoding of cortically represented information, such that strong cortical representations would lead to enhanced memory only in the context of particular hippocampal states (Fig. 2C, center). Finally, hippocampal and cortical representations might need to be “coordinated” in order to optimize memory encoding: some degree of visual or semantic information is necessary for subsequent memory, but it is possible that object information may also be detrimental, especially if not coordinated with the hippocampus. This view would be consistent with a “saddle-shaped” interaction (Fig. 2C, right) in which successful subsequent memory can be expected so long as both the hippocampus and representational regions are engaged in information coding or processing to the same extent.
Results
Behavioral results
Overall, memory for both the object concept and exemplar image was good, such that the oldness ratings were significantly higher for previously seen concepts or exemplar image, respectively, than novel ones (Conceptual test: meanold = 3.11, SEold = 0.08, meannew = 2.24, SEold = 0.07, meanold-new = 0.87, SEold-new = 0.04, df = 398.27, t = −21.67, p < 0.001; Perceptual test: meanold = 2.89, SEold = 0.05, meannew = 2.70, SEold = 0.06, meanold-new = 0.20, SEold-new = 0.05, df = 299.13, t = 3.84, p < 0.001). However, we noticed a wide range of performances on new objects across participants (Fig. 1B), and therefore adjusted for individual difference in their response bias (see Methods). We examined the mean adjusted memory scores for each object on both Conceptual and Perceptual Memory tests; these two scores were moderately correlated (Spearman's ρ = 0.33, p < 0.001), suggesting that some objects may be intrinsically easier to remember than others across test formats. Therefore, in all subsequent analyses where either type of memory of each object (e.g., memory of individual object concepts) was the dependent variable we included the other type of memory (e.g., memory of the corresponding object images) as a covariate of no interest.
Cortical representation of visual and semantic information
We first localized visual and semantic representation of real-world objects during encoding by testing which brain regions showed positive trial-level representational strength. Participants’ subsequent memory of the objects on both conceptual and perceptual memory tests were included as covariates of no interest to remove any potential effect of subsequent memory, as this initial analysis was focused on identifying brain regions that represented visual or semantic information independently of subsequent memory success, and not focused on memory modulation of representations. As such, interpretations were based on the fitted intercepts which were memory-independent measures of neural representation of stimulus information. Memory-independent visual representations were found in ventromedial occipital cortex (vMOC; t = 3.80, p < 0.001), lateral occipital cortex (LOC; t = 4.79, p < 0.001), and left fusiform gyrus (FuG; t = 3.33, p = 0.003). Memory-independent semantic representations were found in LOC (t = 4.03, p < 0.001), bilateral FuG (left: t = 5.16, p < 0.001; right: t = 5.33, p < 0.001), bilateral AnG (left: t = 5.29, p < 0.001; right: t = 3.51, p = 0.002), left posterior superior temporal sulcus (left pSTS; t = 5.02, p < 0.001), left inferior frontal gyrus (left IFG; t = 4.30, p < 0.001), and right parahippocampal gyrus (right PhG; t = 4.05, p < 0.001). Meanwhile, the hippocampus did not represent either visual or semantic characteristics of the stimuli at encoding (visual: t = 0.40, p = 0.691; semantic: t = 1.49, p = 0.151) (Fig. 3 and Table 1).
Trial-level hippocampal–cortical interactions
The proposed four trial-level estimates of hippocampal task-engagement indeed captured unique aspects of hippocampal function during encoding, as evidenced by the low pairwise correlation coefficients (|r| ≤ 0.17, Fig. 2A), which suggest relatively independent effects of activation level, neural pattern similarity, and semantic or visual processing in our model. Using these hippocampal measures as well as interaction terms outlining hippocampal involvement coordinated with cortical representational strength for visual and semantic information, we examined how hippocampal–cortical interactions affected subsequent memory of each object. Of note, hippocampal interactions were only examined in the cortical regions that showed significant memory-independent representation of semantic or visual information during encoding, i.e., where neural representations of object properties are most robust. For a more complete series of exploratory analyses of subsequent-memory-modulation on neural representations of visual and semantic information across the cortex, readers are referred to Davis et al. (2021).
Transfer-appropriate analyses
As the semantic properties of the stimuli may be particularly informative of the concept while the visual details are more relevant to the exemplar image, two sets of transfer-appropriate models were first examined: one set of models that predict participants’ Conceptual Memory (e.g., “I remember seeing a panda”) using semantic representation in relevant cortical regions (e.g., AnG) and one set of models that predict participants’ Perceptual Memory (e.g., “I remember seeing this exact panda”) using visual representation in visual regions (e.g., vMOC).
Conceptual memory
All models revealed a main effect of Perceptual Memory: an object whose exemplar image was better remembered was more likely to have its concept remembered as well (b ≥ 0.22, SE ≤ 0.02, t ≤ 13.90, p < 0.001). In other words, remembrance of visual details in the stimulus image may help participants recognize the concept. There was no main effect of semantic representation in any tested cortical region. Yet, there was a series of significant interactions between cortical representation and different measures of hippocampal engagement (Fig. 4A). Specifically, semantic representational strength in left IFG showed significant interactions with the activation level in the hippocampus (b = 0.04, SE = 0.02, t = 2.37, 95% CI = [0.01, 0.07], p = 0.018, pperm = 0.027), such that conceptual memory was the best when the hippocampus was active and semantic information of the object was represented well in left IFG. We also found a significant interaction between the hippocampus and right AnG: conceptual memory success was more positively associated with the semantic representational strength in right AnG when hippocampal semantic representational strength was also higher (b = 0.03, SE = 0.01, t = 2.27, 95% CI = [0.00, 0.06], p = 0.023, pperm = 0.020).
Transfer-appropriate results. Glass brain and 3D plots illustrating transfer-appropriate hippocampal–cortical interactions that predicted subsequent memory. Brain regions are color coded to indicate the relevant measures. A, Subsequent Conceptual memory predicted by an interaction between hippocampal activation level (Act) and semantic representation (Sem) in the left inferior frontal gyrus (left IFG), as well as an interaction between semantic representations in the hippocampus (Hipp) and right angular gyrus (right AnG). B, Hippocampal activation level (Act) and neural pattern similarity (NPS) interacted with visual representations (Vis) in ventromedial occipital cortex (vMOC) to support subsequent Perceptual memory.
Perceptual memory
Like how Perceptual Memory predicted Conceptual Memory, Conceptual Memory success also predicted perceptual memory success in all models (b ≥ 0.24, SE ≤ 0.02, t ≥ 13.83, p < 0.001), suggesting some reciprocal facilitation between these two forms of memory. There was no main effect of visual representation in any tested cortical region, but several significant hippocampal–cortical interactions. Both hippocampal activation level and neural pattern similarity interacted with low-level visual features represented in vMOC to affect subsequent memory of image exemplars (Fig. 4B). Specifically, vMOC visual representational strength was more positively associated with better perceptual memory either when the hippocampus was more active (b = 0.04, SE = 0.02, t = 2.21, 95% CI = [0.00, 0.07], p = 0.027, pperm = 0.041) or when the hippocampus showed greater neural pattern similarity with all other objects (b = 0.04, SE = 0.02, t = 2.55, 95% CI = [0.01, 0.07], p = 0.011, pperm = 0.007). Hippocampal activation level also interacted with LOC visual representations, though the nonparametric permutation-based p-value was above the significance level (b = 0.04, SE = 0.02, t = 1.98, 95% CI = [0.00, 0.08], p = 0.048, pperm = 0.055).
Overall, these results are consistent with the theoretical predictions of the transfer-appropriate processing principle, as hippocampal interactions with representations in semantic and visual areas, respectively, were important to later success on the Conceptual and Perceptual Memory tests. To ensure that these significant hippocampal–cortical interactions from transfer-appropriate models were not due to global fMRI activity fluctuations, a set of control analyses were run for all models with significant hippocampal–cortical interactions using the paracentral lobule (sensorimotor region for the lower limbs) in lieu of the hippocampus and, as expected, yield no significant effects (p > 0.05).
Lateralized hippocampal–cortical interactions
Using a bilateral mask for the hippocampus is a common approach to understanding this region's contributions to memory encoding (LaRocque et al., 2013; Liang et al., 2013; Huffman and Stark, 2014; Poh et al., 2022). However, our particular exploration of semantic and visual information—which has noted laterality effects in correlational and stimulation studies of memory (Addante et al., 2015; Dahlgren et al., 2020; Dunne and Opitz, 2020; Mankin et al., 2021)—suggests value in explicitly examining the laterality of such effects in the hippocampus. To probe a potential functional lateralization regarding hippocampal interactions with cortical regions, we re-ran the transfer-appropriate analysis (e.g., in Fig. 2B) but replaced bilateral hippocampal measures with those from the left and right hippocampus, separately. For this follow-up analysis, we only included cortical regions that were found above to interact with the hippocampus in support of subsequent memory, as well as their contralateral counterparts (e.g., left AnG). For Conceptual Memory, there was a significant interaction between semantic representational strength in left IFG and activation level in both the left hippocampus (b = 0.04, SE = 0.02, t = 2.20, 95% CI = [0.00, 0.07], p = 0.028, pperm = 0.030) and right hippocampus (b = 0.04, SE = 0.02, t = 2.37, 95% CI = [0.01, 0.07], p = 0.018, pperm = 0.026), as well as an interaction between the semantic representational strength in the right AnG and in the right hippocampus (b = 0.04, SE = 0.01, t = 2.53, 95% CI = [0.01, 0.06], p = 0.012, pperm = 0.006). Interestingly, we additionally found that the semantic representational strength in left AnG and LOC interacted with activation level in the left hippocampus (b = 0.04, SE = 0.02, t = 2.10, 95% CI = [0.00, 0.07], p = 0.036, pperm = 0.033) and semantic representational strength in the right hippocampus (b = 0.03, SE = 0.01, t = 1.97, 95% CI = [0.00, 0.06], p = 0.049, pperm = 0.034), respectively.
In models predicting Perceptual Memory, visual representational strength in vMOC interacted with left hippocampal neural pattern similarity (b = 0.03, SE = 0.02, t = 2.04, 95% CI = [0.00, 0.06], p = 0.042, pperm = 0.047), right hippocampal activation level (b = 0.04, SE = 0.02, t = 2.13, 95% CI = [0.00, 0.07], p = 0.034, pperm = 0.032), and right hippocampal neural pattern similarity (b = 0.04, SE = 0.02, t = 2.55, 95% CI = [0.01, 0.08], p = 0.011, pperm = 0.007). Overall, these results largely confirm the role of bilateral hippocampus in supporting mnemonic cortical representations, while additionally suggesting some degree of functional lateralization of the hippocampus with a tendency for ipsilateral over contralateral hippocampal–cortical interactions.
Transfer-incongruent analyses
Two sets of exploratory analyses were conducted to probe the “transfer-incongruent effects” of hippocampal–cortical interaction on subsequent memory, namely, how hippocampal interaction with visual regions affects subsequent Conceptual Memory and how hippocampal interaction with semantic regions affect subsequent Perceptual Memory. As with the previously shown transfer-appropriate models, all fitted transfer-incongruent models revealed significant and positive associations between Conceptual and Perceptual Memory of each object (b ≥ 0.24, SE ≤ 0.02, t ≥ 13.81, p < 0.001). Interestingly, there were facilitatory main effects of semantic representation in right FuG (b = 0.04, SE = 0.02, t = 2.24, 95% CI = [0.01, 0.07], p = 0.025, pperm = 0.022) and right perirhinal cortex (b = 0.03, SE = 0.02, t = 2.09, 95% CI = [0.00, 0.07], p = 0.037, pperm = 0.043; Fig. 5) even after accounting for performance on the Conceptual Memory test; stronger representations of semantic information during encoding in all these regions improved subsequent perceptual memory of specific image exemplars. The converse was not true; in our data, there were no main effects or interactions of visual representations on subsequent memory of concepts.
Transfer-incongruent results. Stronger semantic representational strength (Sem) in both right perirhinal cortex (R. Perirhinal) and right fusiform gyrus (R. FuG) predicted better perceptual memory of the item. Main effects were plotted in 3D (with all other non-significant slopes set to zero) for visual consistency with other results visualizations.
No hippocampal subsequent memory effects
Interestingly, in all transfer-appropriate and transfer-incongruent models above we did not observe any significant main effect of any hippocampal measures on subsequent memory. To understand whether this could have resulted from a lack of power in these models with multiple interaction terms, we tested if the hippocampal measures predicted subsequent memory when we removed cortical representational strength and associated interaction terms. Still, the trial-level estimates of hippocampal function (i.e., activation level, neural pattern similarity, and representational strength) did not predict subsequent Conceptual or Perceptual Memory, whether we tested the entire bilateral hippocampus or separately for the left and right hippocampus (p > 0.08; Table 2).
Hippocampal measures did not predict subsequent memory
Discussion
In the current fMRI study, we investigated how the hippocampus engages with stimulus-specific information represented in the cortex during encoding to support subsequent memory. We identified cortical regions that represented low-level visual and high-level semantic information about object stimuli. We computed three distinct types of measures of hippocampal engagement—activation level, neural pattern similarity, and visual and semantic representational strengths—to examine possible hippocampal functions that modulate the mnemonic effect of cortical representations. We found that the specific ways in which the hippocampus interacts with cortical representations of stimulus information depends on their relevance to the memory test. These findings are discussed in depth below.
Our initial representational similarity analysis identified multiple cortical regions that represented visual or semantic information about a large set of real-world objects. Low-level visual information was found to be neurally represented in primary and extended visual regions, consistent with previous findings (Cichy et al., 2014, 2016; Khaligh-Razavi and Kriegeskorte, 2014; cf., Xu and Vaziri-Pashkam, 2021). Meanwhile, semantic knowledge showed correspondence with neural patterns in many cortical regions implicated in semantic processing, such as left IFG (Carota et al., 2017, 2021), AnG (Devereux et al., 2013; Neyens et al., 2017), FuG (Devereux et al., 2018; Carota et al., 2023), and perirhinal cortex (Clarke and Tyler, 2014; Liuzzi et al., 2015; Martin et al., 2018). Meanwhile, the hippocampus did not represent visual or semantic information about the stimuli, as commonly reported by studies examining hippocampal multivariate patterns during the presentation of real-world objects or scenes (Diana et al., 2008; LaRocque et al., 2013; Huffman and Stark, 2014).
Our principal finding was that the hippocampus interacts with cortical representations in a manner that is commensurate with memory demands, such that its interactions with cortical regions coding for semantic information predicted conceptual memory, while its interactions with cortical regions coding for visual information predicted perceptual memory. Specifically, hippocampal–cortical interactions supporting conceptual memory were evidenced by significant interactions between the hippocampus and both AnG and left IFG. The product of semantic representational strengths in both the hippocampus and right AnG served as a unique significant predictor of conceptual memory, suggests a selective role for “coordinated co-representation” of semantic information in right AnG and the hippocampus, independent of other forms of hippocampal involvement. A follow-up analysis examining the contribution of left and right hippocampus not only corroborated this result with right AnG, but also revealed an interaction between information coding in left AnG and the left hippocampus in predicting memory success, suggesting a tendency for ipsilateral hippocampal–cortical interactions. AnG is believed to participate in diverse cognitive processes including semantic processing (Bonner et al., 2013; Price et al., 2015; Liuzzi et al., 2020) and episodic construction and recollection (Hutchinson et al., 2009; Ramanan et al., 2018), and its neural patterns are found to reflect the semantic similarity of stimuli (Devereux et al., 2013; Carota et al., 2017; Neyens et al., 2017). Notably, our findings of memory-related angular–hippocampal interactions are consistent with abundant evidence for close angular–hippocampal communications from studies examining functional connectivity (Rushworth et al., 2006; Uddin et al., 2010) and transcranial magnetic stimulation (Tambini et al., 2018; Hermiller et al., 2019).
A puzzling aspect of this finding, however, is that subsequent conceptual memory of an object was good even when AnG represented semantic information poorly—so long as the hippocampus did the same. One possible explanation for this effect is that our broad semantic knowledge of and past experiences with everyday objects are not stored in right AnG alone, and neural measures from other brain regions might more accurately reflect the extent to which the semantic properties of some objects support successful memory. This view is also consistent with the evidence that the hippocampus is functionally coupled with prefrontal and parietal cortices in a frequency-specific manner during successful encoding (Watrous et al., 2013). Nevertheless, our finding establishes that when the hippocampus and AnG are communicating in the same, semantically informed “code”, stronger conceptual memories ensue.
In contrast to AnG, semantic coding in left IFG interacted with hippocampal activation, suggesting the hippocampus plays a role in boosting the gain on left IFG processes to support memory success. While univariate activation studies have found left IFG involvement in semantic cognition (Hagoort, 2005; Jefferies, 2013), representational similarity studies have shown that left IFG represents distributional semantics as opposed to experience-based information like AnG does (Carota et al., 2021, 2023; Fernandino et al., 2022). In the current study, left IFG showed strong feature-based semantic representational strength independently of memory performance. Moreover, its interaction effect with hippocampal activation suggests a more multifaceted view of the region, in that its relevance for memory success is driven by the degree to which it is coupled with hippocampal processes. More broadly, our finding observing the sensitivity of this hippocampal–cortical interaction to conceptual memory performance may then reflect the role of the hippocampus in indexing cortical regions responsible for semantic coding. This result is consistent with findings of increased hippocampal–cortical functional connectivity for subsequently remembered versus subsequently forgotten objects (Manelis et al., 2013; Becker et al., 2017) and other types of stimuli (Ranganath et al., 2005; Schott et al., 2013), providing support for the mnemonic relevance of frontotemporal communications. However, such functional connectivity analyses do not assess the informational content communicated in such interactions, which is central to the transfer-appropriateness of hippocampal–cortical interactions revealed by our approach. With a growing set of techniques to quantify informational and representational connectivity (Anzellotti and Coutanche, 2018), future research may reveal the content of neural conversations and help clarify the above-mentioned inferences.
A parallel set of transfer-appropriate hippocampal–cortical interactions was found for perceptual memory, such that hippocampal interactions with cortical regions coding for visual information supported perceptual memory success independently of conceptual memory performance. Specifically, we found that perceptual memory was predicted by two significant positive interactions: between visual information represented in vMOC and either hippocampal activation level or hippocampal neural pattern similarity. This suggests that the hippocampus may have an active role in upregulating the information processing operations, such that the information represented in upstream sensory regions is clearer and better stored for later retrieval. In particular, the neural pattern similarity measure, computed as the average similarity of one object to all other objects, may capture the “convergence” of hippocampal patterns; such a convergence state has been interpreted as an anticipatory state that prepares the mnemonic system for better memory formation (Poh et al., 2022).
Overall, our findings demonstrate the flexibility of a diverse array of hippocampal functions—its capacity to activate, code, or even represent real-world objects—in mediating the mnemonic contribution of transfer-appropriate cortical representations for subsequent conceptual and perceptual memory success. In order for the hippocampus to flexibly modulate distinct information coded across the cortex, the region must be able to multiplex—simultaneously encode multiple signals (Akam and Kullmann, 2014). The hippocampus interacts with distinct cortical networks to refine representations of the perceptual and semantic details of objects and their relation to one another (Ranganath and Ritchey, 2012). The current results expand our understanding of these interactions by revealing how they relate to different kinds of object representations, with ipsilateral hippocampal–cortical interactions appearing to be more influential. Of note, interactions between the hippocampus and AnG were largely attributed to the exchange of semantic information benefiting conceptual memory, which might be the basis for event models supported by the posterior medial network (Ritchey and Cooper, 2020). This is also consistent with recent work suggesting that the angular gyrus plays a general role in representing conceptual associations that underpin both semantic and episodic retrieval (see special issue: De Brigard et al., 2022). In addition, the hippocampus has been found to support the maintenance of multiple items in working memory through cross-frequency coupling of oscillatory activity (Axmacher et al., 2010; Leszczyński et al., 2015), rendering it possible to multiplex its modulation of different information in the temporal domain.
We also conducted a series of exploratory, transfer-incongruent analyses, assessing the extent to which semantic/visual representation predicted perceptual/conceptual memory. These transfer-incongruent models revealed only significant main effects of semantic coding in the right FuG and right perirhinal cortex for perceptual memory success. The main effects are not surprising given that, while conceptual and perceptual memory intuitively relies on encoded semantic and visual features, respectively, transfer-incongruent semantic cues (e.g., “Hawaiian pizza” vs “Pepperoni pizza”) that effectively summarize complex visual characteristics [e.g., yellow chunks (pineapple) vs red circles (sausage slices)] can be used to better distinguish old and novel exemplars. Moreover, the fact that these memory-complementary effects occurred independently of any hippocampal functions further speaks to the specificity of hippocampal modulation on cortical representations of stimulus information that is appropriate for certain types of subsequent memory.
In conclusion, we identified a specific set of hippocampal–cortical interactions that supports subsequent memory for real-world objects. The hippocampus uniquely modulates the mnemonic effect of cortical representations that are transfer-appropriate. Our finding demonstrates the importance of examining multiple measures of hippocampal functions and hippocampal–cortical interactions in investigating informationally rich episodic memory.
Footnotes
Author contributions: S.H., R.C., and S.W.D. designed research; M.H. and S.W.D. performed research; S.H., C.M.H., M.H., and S.W.D. analyzed data; S.H. and S.W.D. wrote the first draft of the paper; S.H., M.R., R.C., and S.W.D. edited the paper; S.H., M.R., and S.W.D. wrote the paper.
This work was supported by the National Institute of Health, R01-AG066901 and R21-AG058161.
The authors declare no competing financial interests.
- Correspondence should be addressed to Simon W. Davis at simon.davis{at}duke.edu.
