Abstract
How does the brain process continuous experiences so they can be remembered? Evidence suggests that people perceive their experience as a series of distinct and meaningful events. Information encountered within the same event shows greater temporal integration into memory as well as enhanced neural representational similarity. Although these data support the theory that the brain builds and maintains a mental model of the current event that represents recently encountered stimulus information, this hypothesis has not been directly tested. We used fMRI in humans (N = 21, 13 female) to test whether within-event neural similarity indicates the persistence of stimulus representations in a mental model. Participants viewed trial-unique visual images that were grouped into events. We calculated neural pattern similarity across time in the category-selective visual cortex to measure stimulus persistence. Pattern similarity was enhanced within, compared with between, events in the object-sensitive left lateral occipital (LO) cortex. This was specific to situations when objects could persist within a mental model, suggesting modulation of neural activity based on the features and structure of the event. Left LO object persistence was correlated with activity in a medial prefrontal cortex (mPFC) region linked to representing mental models within events. mFPC activity also correlated with pattern similarity in the hippocampus but more generally across stimulus categories. Critically, left LO similarity was related to estimates of temporal proximity in memory. The data suggest that temporal neural stability reflects stimulus persistence in mental models and highlight the importance of within-event representational stability in the transformation of experience to memory.
SIGNIFICANCE STATEMENT How does the brain process continuous experiences so they can be remembered? One idea is that information persists in mental models during stable events, facilitating the organization of events in memory. Using fMRI pattern similarity analysis, we found enhanced similarity within, compared with between, events in the object-sensitive LO but only when objects could persist within a mental model. mPFC activity correlated with left LO similarity when objects persisted within an event; in contrast, mPFC activity correlated with hippocampal similarity across stimulus categories. Left LO persistence was also related to the remembered temporal proximity of stimuli. The data suggest the brain dynamically maintains stimulus information in mental models during events, supporting the transformation of experience into memory.
Introduction
Our experience of the world is continuous, yet our memories are organized around distinct and meaningful events. This organization supports learning, prediction, and behavior, but many aspects of the transformation of continuous experience into episodic memory remain poorly understood. How does the brain integrate elements of an experience that are separated in time but are part of the same event? One possibility is that cognitive and neural mechanisms detect event boundaries and segment continuous experience into meaningful events (Reynolds et al., 2007; Zacks et al., 2007; Radvansky and Zacks, 2017). Such a process could organize the perceptual representations that are linked in an episodic memory trace for the current event (Davachi and DuBrow, 2015; Deuker et al., 2016; Brunec et al., 2018; Clewett et al., 2019).
During an experience, event boundaries affect how sequential representations become linked in memory (Polyn et al., 2009; DuBrow and Davachi, 2014; Heusser et al., 2016, 2018; Horner et al., 2016). Event boundaries also widely modulate brain activity, including in the cortex (Speer et al., 2007; Ezzyat and Davachi, 2011; DuBrow and Davachi, 2016; Reagh et al., 2020) and the hippocampus (Ben-Yakov and Dudai, 2011; Baldassano et al., 2017; Ben-Yakov and Henson, 2018; Silva et al., 2019; Reagh et al., 2020). Experiences that occur within the same event show more similar neural representations than experiences that are separated by event boundaries (Lerner et al., 2011; Schapiro et al., 2013; DuBrow and Davachi, 2014; Ezzyat and Davachi, 2014; Lositsky et al., 2016; Manning et al., 2016; Baldassano et al., 2017; Sols et al., 2017). Within-event representational similarity may therefore guide the transformation of continuous experience into distinct memory representations (Davachi and DuBrow, 2015; Griffiths and Fuentemilla, 2020).
However, it remains unclear what mechanism neural similarity reflects. Psychological theories suggest that representations of recent stimuli persist within an active internal model of the current event (Morrow et al., 1987; Bower and Morrow, 1990; Speer et al., 2007; Radvansky and Zacks, 2017). fMRI studies using naturalistic stimuli have showed greater neural similarity within, compared with between events, consistent with this idea (Baldassano et al., 2017; Chen et al., 2017; Baldassano et al., 2018; Bird, 2020). However, similarity is often measured across participants (Baldassano et al., 2017; Chen et al., 2017; Baldassano et al., 2018) or between average patterns of neural activity for each event (Chen et al., 2017; Baldassano et al., 2018), which may obscure the relative contributions of schematic and episodic information. In addition, the use of naturalistic stimuli leaves open the possibility that similarity in perceptual inputs contributes to the event-level similarity differences.
If people maintain active mental models during an event, then there should be neural evidence for within-event maintenance of information, even when such information is no longer directly perceptually available. Such maintenance could happen in areas of the sensory cortex that code for the encountered stimulus (Serences et al., 2009; Eriksson et al., 2015). If so, then neural activity for items encountered early within an event should persist over time within stable events but not across event boundaries. Furthermore, if persistence is related to a broader mental model of the current event, then it should be stimulus specific and correlated with activity in brain areas that contribute to the representation of internal event models.
We used fMRI to measure pattern similarity (PS) across time in the object-sensitive lateral occipital (LO) cortex as people encoded events with object stimuli. Left LO persistence was enhanced within, compared with across, events but only when objects occurred in early event positions, suggesting stimulus-specific maintenance. Persistence was also related to temporal memory, suggesting that it may support memory integration. Left LO persistence was correlated with event-specific activity in the medial prefrontal cortex (mPFC), a region that integrates the schematic knowledge and episodic information (Tse et al., 2007; van Kesteren et al., 2010; Bein et al., 2014; Baldassano et al., 2018) needed to support event model construction during an experience (Ezzyat and Davachi, 2011; Ranganath and Ritchey, 2012; Bertossi et al., 2016). mPFC activity also correlated with hippocampal pattern similarity, regardless of the category of the encountered stimuli. The data suggest that interactions between the mPFC, the sensory cortex, and the hippocampus are dynamic during encoding and related to the ongoing structure of our experiences.
Materials and Methods
Participants
Twenty-one right-handed native English speakers (13 female; age 18–31 years, mean = 24) were recruited from the New York University and New York City communities and participated for monetary compensation ($25/h). Informed consent was obtained from each participant in a manner approved by the University Committee on Activities Involving Human Subjects. Two participants were excluded from analysis, one because of sleepiness during during the encoding task and another because of excessive head motion during scanning. This resulted in N = 19 participants, whose data were analyzed. For analyses of activity in functionally defined visual areas [LO, parahippocampal place area (PPA), fusiform face area (FFA)], Ns were determined by the total number of participants in whom we could localize the visual areas and are noted in the Region of interest definition subsection.
Stimuli
Stimuli for the encoding task consisted of objects, famous faces, and outdoor scenes drawn from an on-line database (http://cvcl.mit.edu/database.htm; Oliva and Torralba, 2001 and from an Internet image search. Each participant viewed 128 object, 128 famous face, and 96 scene stimuli during encoding. Stimuli were counterbalanced so that no object or face was paired with the same scene more than once across participants. All stimuli were presented equally, often in the Boundary and NonBoundary conditions across participants. Object and scene stimuli for the localizer task were drawn from the same databases as the encoding stimuli; however, the localizer stimulus set did not overlap with the encoding stimulus set. Scrambled object stimuli were created by dividing images of objects into a 20 × 20 pixel grid and randomly scrambling the location of each 20 × 20 block in the grid. Face stimuli for the localizer were drawn from a separate set of nonfamous faces.
General procedure
The general procedure was previously reported in detail (Ezzyat and Davachi, 2014); we describe it again here, focusing on portions of the procedure most relevant to the current report (Fig. 1A). Each participant performed four fMRI scanning runs, which were composed of an encoding task, a recognition memory task, and a temporal memory task. Each encoding run consisted of 64 trials; on each encoding trial (4 s), an image of an outdoor scene was presented, paired with an image of either an object or a famous face. Participants were instructed to imagine a scenario in which the object/person was in the scene; after the 4 s stimulus period, participants were cued to respond yes or no to indicate their imagery success (1.5 s). Encoding trials were organized into quartets, consisting of four consecutive trials; for half of the quartets the same scene was presented across all four trials (NonBoundary condition) and for the other half, the scene switched on the third trial (Boundary condition; Fig. 1A). The order of NonBoundary and Boundary quartets was randomized within each run for every participant. The intertrial interval (2–20 s) varied pseudorandomly (Dale, 1999), subject to the constraint that the overall intertrial interval between pairs of trials was matched within-participant for NonBoundary and Boundary quartets. All object and face stimuli were trial unique. Within each quartet, two object and two face stimuli were presented consecutively (Fig. 1A). This meant that the object/face category switched at the same time as the scene switched for Boundary quartets. In contrast, for NonBoundary quartets, the object/face category switched without a concurrent scene switch. There were an equal number of object→face and face→object switches in the NonBoundary and Boundary conditions. We also randomized the order of the object/face category switches within the NonBoundary and Boundary conditions. Following each encoding run, participants performed a recognition memory test followed by a temporal memory test.
Experimental design and predictions. A, Participants performed an associative encoding task in which trial-unique object and famous face images were presented with outdoor scene images. B, We predicted that LO activity would show evidence of persistence in NonBoundary quartets in which objects were presented in early event positions but not in matched Boundary quartets. C, When objects were presented in late event positions, we predicted no LO persistence in either the NonBoundary or Boundary conditions.
Temporal memory test
The recognition memory test is not the focus of this report and is not further discussed. Behavioral and neural results from the temporal memory test have been previously reported (Ezzyat and Davachi, 2014) and are also not the primary focus of this article. However, we do include one unreported analysis examining within-event stimulus persistence as a function of temporal memory, so we describe the temporal memory task here. Participants were presented with two stimuli from the preceding encoding phase and asked to indicate how far apart in time the two items were at encoding, choosing from the following response options: very close, close, far, and very far (maximum 8 s per trial). Participants could also respond don't know and were instructed to use this option if they did not remember seeing one or both of the stimuli during the encoding phase. Participants performed 32 temporal distance judgment trials per run for a total of 128 trials across all runs. Half the distance judgments were on pairs of trials that were consecutive at encoding; the remaining half were on pairs of trials that were separated by two intervening trials at encoding. Behavioral and fMRI analyses of only these nonconsecutive pairs are reported here (the P1 and P4 trials). Thus, the actual temporal distance between items in a pair was matched. For comparing pattern similarity between temporal memory conditions we collapsed very close and close responses into a single bin, Close, and very far and far responses into a single bin, Far.
fMRI acquisition and preprocessing
A Siemens Allegra 3T head-only scanner (Center for Brain Imaging at New York University) with a custom head coil (NM-011; Nova Medical) was used for MR imaging. Functional data were acquired using the following parameters: echoplanar imaging (EPI) pulse sequence, 34 contiguous slices, voxel size 3 × 3 × 3 mm, repetition time = 2000 ms, echo time = 15 ms, flip angle = 82. Slices were oriented parallel to the anterior commissure–posterior commissure axis and positioned ventrally to provide coverage of the anterior temporal lobes and prefrontal cortex; this occasionally resulted in omission of areas of the superior parietal and motor cortices. A high-resolution T1-weighted anatomical scan (magnetization-prepared rapid-acquisition gradient echo sequence, voxel size = 1 × 1 × 1 mm) was also obtained for each subject following the final block of the localizer task. SPM8 software (Wellcome Trust Center for Neuroimaging) was used for data preprocessing. Functional images were realigned to the within-run mean to correct for head motion (one run from one participant was discarded because of head motion > 1 voxel). Realigned images were corrected for slice acquisition time and were then coregistered to the anatomic image to correct for between-run motion. For definition and analysis of subject-space anatomical (e.g., hippocampus) and functional (e.g., LO, PPA, FFA) regions of interest (ROIs), the coregistered images were smoothed using a 6 mm FWHM isotropic Gaussian kernel. For group-level analyses, the coregistered images were first spatially normalized to an EPI template in Montreal Neurologic Institute (MNI) space, resliced to 2 × 2 × 2 mm voxels, and finally smoothed with a 6 mm FWHM isotropic Gaussian kernel. Low frequencies (<2 cycles/run) were removed from the functional data in both the subject-specific and group analyses.
Region of interest definition
Anatomical ROIs for the hippocampus were defined for each participant using the Functional MRI of the Brain Software Library FIRST automatic segmentation tool (Patenaude et al., 2011). The resultant ROIs were then visually inspected to ensure that the left and right hippocampi were correctly identified. In the cases of three participants for whom automatic hippocampal segmentation failed, left and right hippocampal ROIs were hand drawn on the participants' T1-weighted anatomic images using an in-house drawing tool written in MATLAB (MathWorks). In addition, the anterior, middle, and posterior thirds of the hippocampus (determined by coronal slice position) were also separately defined as ROIs. Because we were interested in the role of the hippocampus in representational persistence within events, we focused on the anterior hippocampus (aHC) consistent with models proposing an integrative function for this area (Poppenk et al., 2013; Preston and Eichenbaum, 2013; Schlichting and Preston, 2016). We then defined our hippocampal ROI as the union of all voxels in both hemispheres in the anterior hippocampus. Finally, we also defined a spherical ROI in the bilateral mPFC, previously reported to show fMRI responses that correlate with event structure during encoding [Ezzyat and Davachi, 2011; center (−2, 29, 3), radius (anatomic resolution) = 6].
Category-specific ventral visual functional ROIs (the posterior portion of the lateral occipital complex, LO; PPA; FFA) were defined using data from the localizer scans. Blocks of each stimulus (faces, scenes, objects, and scrambled objects) were modeled as boxcars convolved with a canonical hemodynamic response function (HRF). Localizer thresholds were selected separately for each region based on the ROI-specific thresholds reported in the prior literature and in an effort to match the resulting ROI size. The LO ROI (Malach et al., 1995; Grill-Spector et al., 2001) was defined as a region in the posterior occipital cortex that responded more strongly to object blocks than to scrambled object blocks (p < 10−4) the PPA (Epstein et al., 1999) was defined as a region in the parahippocampal gyrus that responded more to scene blocks than to face and object blocks (p < 10−4); the FFA (Kanwisher et al., 1997) was defined as a region in the fusiform gyrus that responded significantly more to face blocks compared with object blocks (p < 0.005). Localizer data were not collected for one participant; in addition, some ROIs could not be defined in all participants at the given thresholds. We found that the left hemisphere LO was reliably larger (mean = 255 vs 180 voxels, p = 0.007) and was present in more participants (N = 15 vs 14) than the right hemisphere LO. We therefore focused category-selective ROI analyses on left hemisphere regions. The number of participants who contributed to each analysis is as follows: object stimulus, persistence N = 15; face stimulus persistence, N = 11; mPFC–LO connectivity within events, N = 15; mPFC–hippocampal connectivity, N = 19; LO persistence as a function of temporal memory, N = 15; and controlling for univariate activity in LO N = 15.
Pattern similarity analysis
PS analyses were conducted on functional data from the encoding runs. To extract single-trial parameters, we estimated a separate GLM for each trial of the experiment (Mumford et al., 2012); each model includes one regressor for the trial itself, modeled as an impulse at trial onset convolved with a canonical HRF, and one regressor that models all remaining N−1 trials as HRF-convolved impulses at the trial onsets. This procedure is then iterated for all trials to produce one GLM for each trial of the experiment.
Patterns of single-trial activation estimates were then extracted for pairs of encoding trials from the first, second, third, and fourth (last) position of each quartet (P1, P2, P3, and P4). The Pearson correlation between patterns on each P1 trial and other trials from the same quartet was used to measure PS (Fig. 1B,C). These values were grouped according to (1) whether the quartet contained a scene switch (Boundary/NonBoundary) and (2) whether the order of stimulus presentation from the first to the fourth trial in the quartet was object→face or face→object (Fig. 1B,C). The correlation values for each trial pair were transformed using Fisher's r to z transformation and were then averaged within participant before being used as input for across-participant statistical tests of differences between conditions.
To determine whether event boundaries modulated face stimulus persistence, we also conducted an exploratory whole-brain searchlight analysis (Kriegeskorte et al., 2006; Haynes, 2015). We used a 5 × 5 × 5 voxel cubic searchlight to compute local pattern similarity across P1, P2, P3, and P4 trials. The subject-space maps of similarity were then normalized to the MNI standard space and transformed using Fisher's r to z transformation. The normalized maps of PS values were then used as inputs to across-participant t tests at each voxel for differences between conditions. We used a permutation procedure to set a mapwise-corrected threshold of p < 0.05 to identify significant clusters.
Cross-region correlation
To determine how mechanisms in the PFC shape within- and across-event PSs in visual areas and the hippocampus, we also analyzed the event-by-event correlation between univariate activity in the mPFC and PS in the LO, PPA, and hippocampus. We extracted event-by-event activity in PFC regions of interest (see above, Region of interest definition) for the third trial of all quartets, averaged across all voxels in the region. These third positions are the critical items at which the scene stimulus either remains stable (NonBoundary) or switches to a novel image (Boundary). We then computed event-by-event PSs in the LO, PPA, and hippocampus for trials before and after these critical trials (see Fig. 5A), and then correlated univariate activity in the mPFC with PS across trials. We used this analysis to determine whether fluctuations in mPFC activity within and across events are related to representational persistence in the LO, PPA, and hippocampus.
Univariate analyses
Analyses controlling for univariate activity used the parameter estimates derived from the single- trial models (see above, Pattern similarity analysis). For the first analysis, we calculated the difference in activity from the first to the last item in each quartet and tested whether this univariate change in activity also differed as a function of object→face/face→object and NonBoundary/Boundary (see Fig. 7A). We also correlated the difference in univariate activity and the level of pattern similarity across P1→P4 pairs to determine whether univariate changes explained pattern similarity on an individual event level (see Fig. 7C).
Statistics
All statistical tests (t tests and ANOVA) were conducted as two-tailed tests. When necessary, non-normally distributed dependent measures were transformed (e.g., using Fisher's r to z transformation; see above, Pattern similarity analysis). All error bars denote SEM. Effect sizes are reported as either Cohen's d for t tests or partial η squared (ηp2) for ANOVA.
Results
Object stimulus persistence
Our primary aim was to determine whether neural representations of recently encountered stimuli are maintained within an event. To address this question, we extracted single-trial patterns of BOLD activity from the left LO and left PPA (see, above, Materials and Methods), two regions of the visual cortex that show differing selectivity for the object stimuli used in our experiment. We took advantage of an aspect of our experimental design, which was that the category of the trial-unique stimuli switched on the third trial of each quartet. If stimulus representations persist within events, then we should expect to observe greater similarity between patterns of neural activity in the object-sensitive LO when objects are presented early in the event, relative to a control region such as the PPA (Fig. 2A).
Analysis of object persistence. A, Pattern similarity across increasing lags showed evidence of NonBoundary object persistence in the left LO but not in the left PPA. B, LO P1→P4 persistence was greater for NonBoundary quartets compared with Boundary quartets (p = 0.005) but did not differ in the PPA (p = 0.25, interaction p = 0.003). C, Across-lag pattern similarity in the LO and PPA did not show face-related persistence. D, P1→P4 persistence in the LO and PPA did not demonstrate face-related persistence. LO Face–Object persistence was significantly lower than LO Object–Face persistence shown in B (p = 0.01). Not significant is denoted by n.s.
To test this prediction, we first calculated pattern similarity in the left LO between position pairs P1→P2, P1→P3, and P1→P4 (Fig. 1B). Left LO similarity remained stable when objects were presented in early event positions in the NonBoundary condition but not in the Boundary condition (Fig. 2A), consistent with maintenance of object representations in the NonBoundary condition (lag × boundary type interaction, F(2,28) = 5.70, p = 0.008, ηp2 = 0.29). In contrast, PPA similarity did not differ between the Boundary and NonBoundary conditions as a function of lag (interaction, p = 0.55); instead, PPA similarity was more consistent with contextual drift in both conditions (lag main effect, F(2,28) = 12.02, p = 2 × 10−4, ηp2 = 0.57).
We then directly tested whether stimulus representations persisted from the start to the end of an event (P1→P4) and found that object-related persistence in the left LO was higher for NonBoundary quartets compared with Boundary quartets (t(14) = 3.32, p = 0.005, d = 0.86, Fig. 2B). In contrast, there was no evidence for greater within-event object persistence in the PPA (p = 0.25). The boundary type × region interaction was significant (F(1,14) = 12.31, p = 0.003, ηp2 = 0.47).
The preceding analysis shows that when objects are encountered early within an event and have the opportunity to persist in an active internal event model, pattern similarity in the object-sensitive left LO is enhanced. By contrast, object-related similarity in the PPA is not affected by boundaries. Critically, the left LO similarity between repeated scenes was not different from similarity between distinct scenes (t(14) = 0.78, p = 0.45), suggesting that repeated perceptual input in the NonBoundary condition did not drive LO similarity. We also analyzed similarity in the right LO, which did not show significant object persistence nor persistence modulated by boundaries (as did the left LO), although there were also no significant differences between hemispheres (all main effects and interactions with hemisphere, p > 0.15).
The preceding left LO data suggest that representations of stimuli that are relevant to the current event are encoded across time in the patterns of neural activity in stimulus-responsive cortical areas. However, if enhanced LO NonBoundary pattern similarity truly reflects persistence of previously encountered objects, then we should be able to eliminate this enhancement by eliminating object stimuli from early event positions. Doing so would remove the possibility for object representations to persist from the beginning to the end of the event, which would manifest as equal LO pattern similarity for NonBoundary and Boundary quartets (Fig. 1C). To test this, we analyzed left LO similarity for quartets in which faces were presented in trial positions P1 and P2, followed by objects in positions P3 and P4 (Fig. 2C). Left LO similarity did not differ for NonBoundary and Boundary quartets (main effect and interaction, ps > 0.55). The NonBoundary/Boundary difference was greater for object→face compared with face→object quartets at the P1→P4 lag (t(14) = 2.77, p = 0.02). Similarity in the left LO for face→object quartets did not differ relative to the PPA as a function of NonBoundary/Boundary status (interaction, F(1,14) = 0.01, p = 0.92; Fig. 2D). The three-way interaction of stimulus order × boundary type × lag in the left LO (F(2,28) = 0.66, p = 0.53) and the PPA (F(2,28) = 1.43, p = 0.26) were not significant. Comparing the left LO and PPA directly, there was a significant three-way interaction of stimulus order × boundary type × region at the longest lag (F(1,14) = 7.8, p = 0.01, ηp2 = 0.36).
Face stimulus persistence
We also tested evidence for persistence of face stimuli in the FFA. We did not find evidence of NonBoundary persistence across lags in the face→object condition (lag × boundary type interaction, p = 0.50, Fig. 3A). Left FFA face→object pattern similarity at the P1→P4 lag did not differ between the NonBoundary and Boundary conditions (p = 0.45); there was also no difference in similarity when comparing face→object and object→face quartets (Fig. 3B) within the NonBoundary condition (p = 0.83). A post hoc test comparing NonBoundary and Boundary similarity for P1→P3 face→object pairs (Fig. 3A) was not significant (p = 0.10).
Analysis of face persistence. A, Left, FFA pattern similarity across lags did not show face-related persistence in the NonBoundary condition. B, The FFA also did not show evidence of object persistence. C, A searchlight analysis comparing NonBoundary face→object > object→face pattern similarity revealed two clusters in the left ventrolateral PFC and right cuneus (voxelwise p < 0.005, uncorrected). Data rendered on an MNI template brain.
Given the findings in the FFA, we conducted a whole-brain searchlight analysis to determine whether there were other cortical areas that showed evidence for P1→P4 persistence of face stimuli within events. We found no regions showing greater persistence for NonBoundary face→object compared with Boundary face→object quartets (using both a familywise-error-rate-corrected threshold of p < 0.05 and a more relaxed uncorrected voxelwise threshold of p < 0.005). We also found no regions showing enhanced persistence when comparing face→object and object→face quartets within the NonBoundary condition when correcting for multiple comparisons. However, using a relaxed voxelwise threshold of p < 0.005, we did identify two clusters showing greater similarity for face→object compared with object→face quartets within the NonBoundary condition—the left ventrolateral PFC and right cuneus (Fig. 3C).
Stimulus persistence in anterior hippocampus
We also assessed evidence for stimulus persistence in the aHC, given its proposed role in integrative processing related to memory formation (Poppenk et al., 2013; Schlichting and Preston, 2016). Pattern similarity did not differ between the NonBoundary and Boundary conditions, nor between the object→face and face→object conditions (Fig. 4; all p > 0.12). These data suggest that the hippocampus may serve a specific role in representing mental models in the service of episodic memory formation rather than a more general role in representation of one's current experience.
Analysis of persistence in the anterior hippocampus. A, Object→face pattern similarity across lags did not differ between the NonBoundary and Boundary conditions. B, Object→face pattern similarity at the longest lag. C, Face→object pattern similarity also did not differ across lags between the NonBoundary and Boundary conditions. D, Face→object pattern similarity at the longest lag. All main effects and interactions p > 0.12.
mPFC–LO connectivity within events
To determine whether persistence is related to the maintenance of internal event models, we identified a region of the mPFC from an independent group of participants performing a narrative encoding task (Ezzyat and Davachi, 2011), and extracted single-trial estimates of mPFC univariate activity in the current experiment. We correlated these event-level estimates with event-level estimates of pattern similarity in the left LO (Fig. 5A) and asked whether within-event temporal persistence during object→face quartets was related to within-event activity in the mPFC. Event-by-event fluctuations in univariate activity in mPFC were correlated with pattern similarity in LO for NonBoundary object→face quartets (t(14) = 5.34, p = 1.0 × 10−4, d = 1.38) but not Boundary object→face quartets (p = 0.22; NonBoundary > Boundary, t(14) = 3.65, p = 0.003, d = 0.94). In contrast, mPFC activity did not correlate with LO pattern similarity for face→object quartets (ps > 0.33), leading to a significant stimulus order × boundary type interaction (F(1,14) = 9.98, p = 0.007, ηp2 = 0.42). These data show a relation between univariate mPFC activity and LO pattern similarity during object→face quartets but not during face→object quartets and suggests that stimulus persistence in the object-selective visual cortex may be related to top-down inputs from the frontal lobe that selectively maintain stimulus representations within event models.
mPFC activity is correlated with LO and hippocampal pattern similarity within events. A, We correlated trial-by-trial univariate activity in prefrontal cortex with trial-by-trial LO pattern similarity. B, Univariate activity in the mPFC correlated with LO pattern similarity in the NonBoundary condition specifically when objects were presented in early event positions but not when faces were. C, Univariate activity in the mPFC correlated with pattern similarity in the anterior hippocampus in the NonBoundary condition but not in the Boundary condition.
mPFC–hippocampal connectivity within events
To determine how top-down signals from the PFC influence hippocampal mechanisms for encoding associations within an event, we conducted the same cross-region correlation between the mPFC and aHC. This analysis showed that activity in the mPFC was more correlated with pattern similarity in the anterior hippocampus in the NonBoundary condition compared with the Boundary condition (F(1,18) = 16.1, p = 0.0008, ηp2 = 0.47) but was not affected by the order of stimulus presentation (main effect, p = 0.16; interaction p = 0.78). In the NonBoundary object→face condition, the mPFC–hippocampal correlation was greater than zero (t(18) = 2.87, p = 0.010, d = 0.66) and greater than the Boundary condition (t(18) = 2.49, p = 0.023, d = 0.57]. These data show a relation between univariate activity in the mPFC and pattern similarity in the anterior hippocampus, and suggests that the mPFC influences stability in hippocampal representations within events.
We used a region × stimulus order × boundary type ANOVA to directly compare the mPFC– LO and mPFC–aHC across-trial correlations. This analysis showed a significant main effect of boundary type (F(1,14) = 5.1, p = 0.04, ηp2 = 0.27) and a marginal stimulus order × boundary type interaction (F(1,14) = 4.0, p = 0.066, ηp2 = 0.22). The three-way interaction was not significant (F(1,14) = 3.40, p = 0.09).
LO persistence as a function of temporal memory
Having identified LO object persistence we then asked whether persistence predicted later temporal memory. We separated pairs of object→face encoding trials into those that were later identified as having occurred Close compared with Far apart in time. Consistent with the analysis of all trials (see below, Results), LO pattern similarity for object→face was higher in the NonBoundary than the Boundary condition (F(1,14) = 9.09, p = 0.009, ηp2 = 0.39; Fig. 6). The NonBoundary versus Boundary difference was significant within the Close condition (t(14) = 2.81, p = 0.014, d = 0.73] but not in the Far condition (p = 0.11). However, neither the main effect of memory response (p = 0.27) nor the boundary type × memory interaction (p = 0.54) was significant.
Neural persistence is greater for within-event pairs remembered as close together. Stimulus persistence in the LO was higher within Close temporal memory trials, p = 0.014, but not within Far temporal memory trials, p = 0.11.
Controlling for LO univariate activity
We next analyzed LO univariate activity to rule out the possibility that our pattern similarity effects were related to changes in fMRI response driven by factors such as repetition suppression or adaptation (Grill-Spector et al., 2006; Ward et al., 2013). Unlike the pattern similarity results, we found no difference in univariate LO activity for NonBoundary versus Boundary quartets in the object→face condition (Fig. 7A,B; p = 0.97). We also found equivalent univariate activity effects when comparing object→face and face→object within the NonBoundary condition (Fig. 7B; p = 0.25). Together, these results suggest that our pattern similarity effects are not driven by univariate differences in LO activity.
LO Univariate activity is not consistent with object persistence. A, To determine whether changes in LO univariate activity were consistent with object persistence, we calculated the difference in univariate response from the first item to the last item in a quartet (P1–P4). B, The P1–P4 univariate difference did not differ between the NonBoundary and Boundary conditions for object-face events (p = 0.97) and did not differ within the NonBoundary condition between object-face and face-object events (p = 0.25). C, Within each subject, we correlated event-by-event estimates of the univariate difference (P1–P4) with event-by-event estimates of pattern similarity. D, The univariate difference in activity did not relate to pattern similarity within subject in a manner consistent with object persistence.
The preceding analysis supports the conclusion that aggregate changes in LO univariate activity within and across events do not relate to representational persistence. However, it leaves open the possibility that pattern similarity could nonetheless be related to univariate activity fluctuations at the individual event level. To address this hypothesis, we correlated the pattern similarity estimates for each event (i.e., pair of P1→P4 trials) with the corresponding univariate change in activity for the same events (Fig. 7C). We found no difference in the pattern similarity–univariate correlation for NonBoundary versus Boundary quartets in either the object→face condition (Fig. 7D; p = 0.96) or the face→object condition (p = 0.45). There was also no difference between NonBoundary quartets as a function of object→face/face→object (Fig. 7D; p = 0.60). These analyses further support the interpretation that the pattern similarity results reflect persistence of activity patterns in the left LO in a way that is consistent with maintenance of stimulus information within stable events.
Discussion
We sought to understand how the event structure of an experience can modulate the ongoing representation of information in the brain. To test this, we asked whether event boundaries modulate the neural representations of recently encountered stimuli in high-level visual cortex. We found that neural representations in the left LO were more similar within events than across events compared with the PPA, consistent with maintenance of stimulus information within events. Enhanced left LO similarity was specific to object stimuli and correlated with univariate activity in the mPFC, suggesting top-down modulation based on the event structure of an experience. The data demonstrate neural evidence for stimulus persistence within events and provide support for theories of event cognition based on maintenance of stimulus information within mental models.
By designing our experiment to use stimuli that evoke distinct patterns of activity in the human visual cortex, we aimed to examine the specificity of representational persistence in the brain. By comparing object→face and face→object events, our study used a novel approach to show that stimulus persistence within events is related to the identity of the stimuli that are encountered but is not driven by perceptual similarity in the stimulus across time. The data are consistent with the notion that people use mental models to process incoming information within a stable event and suggest that that this process occurs via interactions between the prefrontal and sensory cortex. The current findings complement prior work that showed neural similarity within events using naturalistic stimuli (Baldassano et al., 2017; Chen et al., 2017) and integrate models of event cognition (Zacks et al., 2007; Radvansky and Zacks, 2017) with mechanistic models of working memory (Curtis and D'Esposito, 2003; Eriksson et al., 2015) to provide support for event-based maintenance of information in patterns of neural activity within events.
We experimentally manipulated item and event-level information in the stimuli that our participants encoded and observed that activity in the left LO was modulated by the interaction of the recent history of items and events. These data suggest that activity in the left LO is not solely a function of the current environment (Turk-Browne et al., 2012), consistent with work showing that activity in visual areas reflects simultaneous representation of perceptual and mnemonic information (Rademaker et al., 2019). Mechanistically, the LO BOLD pattern similarity that we observed may reflect simultaneous maintenance of perceptual representations that can be targeted via attentional selection (Oztekin et al., 2010; Lewis-Peacock and Postle, 2012).
Although our data are consistent with mental-model-based theories of event cognition, there are also some limitations. We did not observe a significant three-way interaction in the left LO, which one would predict under a stimulus maintenance account. We additionally did not observe persistence in the right LO; both findings could be related in part to overlapping LO responses to objects and faces (Grill-Spector et al., 2001). Future work using more sensitive methods for decoding stimulus representations from neural data (Long and Kuhl, 2021), causal manipulations of neural activity (Mullin and Steeves, 2011; Tambini and D'Esposito, 2020), and more differentiable stimulus classes will help establish the generalizability within and beyond LO (as well as across hemispheres) of the event-processing mechanisms that we report. We also did not find evidence of persistence of face stimuli in the FFA, which one would predict under a model in which people maintain stimulus representations within events. Differences between our localizer task, which used unfamiliar nonfamous faces, and the experimental task, which used familiar famous faces, may have resulted in an FFA region that was not optimized to identify activity related to the face stimuli presented during the experiment (Sergent et al., 1992; Natu and OToole, 2011). The FFA results may also be related to differences in temporal receptive fields along the ventral visual pathway (Gilaie-Dotan et al., 2008; Lerner et al., 2011; Gerber et al., 2017). Face representations may undergo more rapid updating than objects, at least in the context of the current design. In addition, although the FFA did not show evidence of face stimulus persistence within events, a searchlight analysis did show within-event face persistence in the left ventrolateral PFC and right cuneus. In the present experiment, enhanced ventrolateral PFC pattern similarity may reflect general selection and maintenance of semantic information related to the famous faces used as memoranda (Thompson-Schill et al., 2005; Jonides and Nee, 2006; Blumenfeld and Ranganath, 2007; Murty et al., 2010; Kim, 2011). In contrast, the pattern similarity effect in the right cuneus may reflect stimulus-specific (i.e., face) perceptual processing (Kim et al., 1999).
Overall, left PPA pattern similarity was greater than left LO similarity but did not differ based on boundary type or stimulus order. The fact that similarity was generally higher in the PPA could suggest that the PPA integrates information over longer time scales, consistent with models of hierarchical process memory (Lerner et al., 2011; Hasson et al., 2015) that propose increasing temporal receptive windows along the cortical hierarchy from sensory to association cortex (Gilaie-Dotan et al., 2008; Baldassano et al., 2017; Chien and Honey, 2020). However, overall similarity may reflect a combination of two other factors related to our experimental design. First, scene stimuli repeated across consecutive trials, which was important for our manipulation of event boundaries but may have led to repetition suppression of PPA responses. Second, PPA responses are different for object and face stimuli (Epstein and Kanwisher, 1998; Harel et al., 2013), which were presented simultaneously with scene stimuli on each trial. The PPA can show simultaneous repetition suppression and pattern similarity effects within the same task (Ward et al., 2013); future studies will be needed to understand how the PPA temporal receptive window interacts with stimulus repetition during an experience.
Our study provides a link between a cognitive theory of events and neural mechanisms underlying the construction and retrieval of information in episodic memory (Bird, 2020). During a stable event, new information that is encountered is thought to be incrementally incorporated into the current event model until a more global update at an event boundary (Kurby and Zacks, 2012; Bailey and Zacks, 2015). Event boundaries reduce the behavioral accessibility of recently encountered information (Zwaan, 1996; Rinck and Bower, 2000; Speer and Zacks, 2005; Radvansky and Copeland, 2006; Swallow et al., 2011), suggesting that information is maintained in an active or easily accessible state as long as the current event remains stable. Similarly, neural activity is more stable within events than across events (Baldassano et al., 2017; Sols et al., 2017; Silva et al., 2019), and our data suggest that this stability during encoding reflects the maintenance of stimulus-specific information in the high-level visual cortex. Previous work has shown evidence for a broadly distributed and lingering representation of the recent context during encoding (Chan et al., 2017), and our data build on this prior work by showing that event boundaries reduce stimulus-specific neural persistence in the visual cortex.
The current findings contribute to a growing understanding of the role of the mPFC in the neural representation of ongoing experience. Previous animal and human studies show that the mPFC integrates new episodic learning into established knowledge networks (Tse et al., 2007, 2011; van Kesteren et al., 2010; Zeithamova et al., 2012; Bein et al., 2014; Hsieh and Ranganath, 2015; Baldassano et al., 2018). When features are shared across memories, mPFC integration can lead to an overlap in neural representations (Tompary and Davachi, 2017); however, when memory distinctiveness is maintained following consolidation, mPFC BOLD activity (Takashima et al., 2006; Gais et al., 2007; Sterpenich et al., 2009; Sweegers et al., 2014) and multivariate discriminability (Bonnici et al., 2012; Ezzyat et al., 2018) are higher. These studies suggest that the mPFC flexibly activates schematic knowledge to shape encoding of the current episodic event. Our data suggest a mechanism whereby such activation leads to maintenance of information that is consistent with (and likely to be related to) the current moment (Deshpande et al., 2008). An important open question concerns how the mPFC might serve in this role via direct and indirect interactions with posterior brain areas (Saleem et al., 2008; Greicius et al., 2009; Ritchey et al., 2015). Such interactions are likely to reflect the position of the mPFC within a broader network of high-level cortical regions involved in constructing mental models of our experiences, which are used in processing incoming perceptual information, making predictions about the future, and influencing long-term memory representations (Bertossi et al., 2016; Ritchey and Cooper, 2020; Stawarczyk et al., 2021).
We found that overall pattern similarity in the anterior hippocampus did not depend on whether objects or faces occurred in early serial positions within or across events. However, we did find that within-event (NonBoundary) pattern similarity in the anterior hippocampus was correlated with univariate activity in the mPFC. The anterior (relative to posterior) hippocampus is thought to be specialized for integrating information across memories that share a context (Poppenk et al., 2013; Preston and Eichenbaum, 2013; Schlichting and Preston, 2016). In rodents, hippocampal place fields are larger in the ventral (anterior) hippocampus (Kjelstrup et al., 2008), and activity of anterior hippocampal neurons generalizes more readily across events (Komorowski et al., 2013). In humans, intracranially recorded theta oscillations in the anterior hippocampus are slower and less related to spatial navigation than in the posterior hippocampus (Goyal et al., 2020), whereas BOLD fMRI activity in the anterior hippocampus has been shown to reflect generalization across memory representations (Schapiro et al., 2012; Liang et al., 2013). Together with data demonstrating connectivity between the anterior hippocampus and the mPFC (Swanson et al., 1978), our findings suggest that the mPFC may influence hippocampal representational persistence across time to associate temporally separated elements of an experience.
The current findings show how event boundaries modulate the persistence of information in the visual cortex and how this persistence is related to activity in the mPFC. The data suggest that event boundaries shape the neural representation of our current environment, leading to maintenance and integration of information within mental models that then influence later memory for prior experience.
Footnotes
This work was supported by National Institute of Mental Health Grant R01-MH074692 to L.D.
The authors declare no competing financial interests.
- Correspondence should be addressed to Youssef Ezzyat at yezzyat{at}wesleyan.edu
