Abstract
How temporal and contextual information interactively impact on behavior and brain activity during the retrieval of temporal order about naturalistic episodes remains incompletely understood. Here, we used fMRI to examine the effects of contextual signals derived from the content of the movie on the neural correlates underlying memory retrieval of temporal-order in human subjects of both sexes. By contrasting SAME versus DIFF storyline conditions during the retrieval of the temporal order of cinematic events, we found that the activation in the precuneus, as well as behavior, are significantly modulated according to storyline condition, supporting our prediction of contextual information contributing to temporal retrieval. We suggest that the precuneus engages in memory retrieval via reconstructive mechanisms, entailing search within a movie-specific, situational knowledge-structure. Furthermore, information-based analyses of multivoxel activity revealed that the precuneus also contains a context-independent linear representation of temporal distances, consistent with a chronological organization of memory traces. We thus put forward that the retrieval of the temporal-order of naturalistic events encoded in rich and dynamic contexts relies on the joint contribution of chronological and reconstructive mechanisms, both of which rely on the medioposterior parietal cortex in humans.
SIGNIFICANCE STATEMENT Successful retrieval of episodic memory is dependent on both temporal and contextual signals. However, when contextual signals derived from multiple storylines or narratives are complex and intertwined, the behavioral and neural correlates underpinning the interplay between time and context is not completely understood. Here we characterized the activation level and multivoxel pattern of BOLD signals underlying the modulation of such contextual information during temporal order judgment in the precuneus. Our findings provide us with an elucidation of subprocesses implicating the medial parietal cortex in realizing temporal organization of episodic details.
Introduction
Episodic memory allows us to consciously relive past experiences, including contextual details related to the encoded episodes. While classical memory research has focused on the medial temporal and the prefrontal cortex, neuroimaging studies have also highlighted retrieval-related activation of the parietal cortex (Wagner et al., 2005) including the precuneus, which may store context-related (Donaldson et al., 2010; Ranganath and Ritchey, 2012) and episodically linked information (Jonker et al., 2018) in memory.
In the processing of real-life activities, it is known that there exists a hierarchical temporal topography in the processing of sensory information in the human cortex. Both visual and auditory information can be accumulated over relatively long time scales (on the order of minutes) in high-order frontal and parietal areas (Hasson et al., 2008; Lerner et al., 2011). The notion that the parietal cortex (including the precuneus) is at the apex of the chronoarchitecture of the temporal receptive windows (Hasson et al., 2015; Baldassano et al., 2017) implies that this region is “receptive” to and sensitive to the length of temporal structures in the stimulus. This response sensitivity aligns with our previous findings that the memory signals modulated by temporal distance (TD) between two to-be-judged frames during encoding would affect subsequent retrieval activity in the precuneus (Kwok et al., 2014). We previously suggested that the activation of the precuneus reflects reconstructive search-processes entailing some structured representation of the movie content (Zacks et al., 2001; Kurby and Zacks, 2008; and Anderson and Conway, 1993; Conway and Jobson, 2012, for related arguments concerning autobiographical memory). However, how the more complex elements related to internal knowledge and high-level content information such as plots and storylines affect the putative memory reconstructive processes during temporal-order judgments remains unclear (see Conway and Jobson, 2012; Gilboa and Marlatte, 2017; Xue, 2018, for reviews).
The aim of the current study was to examine the influence of high-level content information (here operationalized in terms of “SAME vs DIFF” movie's storylines) on these memory reconstructive processes during temporal-order retrieval. As in Kwok et al. (2012), two frames of the movie were presented side-by-side during the retrieval stage and participants were asked to report which one of the two events happened first in the movie that they had seen the day before. Across trials, the temporal distance between the two events was manipulated parametrically, so as to obtain TDs ranging between 30 s and 30 min. In different conditions, the two frames were extracted either from the same storyline (SAME condition) or from different storylines (DIFF condition). Critically, the TDs were exactly matched in the two conditions so that we could investigate temporal-order retrieval of events separated by identical temporal distances during encoding (thus having them matched in terms of the time scales mediated by “temporal receptive windows”, see Hasson et al., 2008; Lerner et al., 2011), whereas they belong to either the SAME versus DIFF storyline contexts.
We postulated that retrieval processes strictly linked to the chronological organization of memory should lead to analogous effects of TD regardless of SAME versus DIFF storyline context, whereas any modulation by the storyline would imply that contextual information about the movie content contributes to the temporal-order retrieval. Together with behavior and brain activation, we also used information-mapping multivariate analyses (representational similarity: RSA; Kriegeskorte and Kievit, 2013) to identify possible multivariate representation of TDs. Several previous imaging studies used multivariate approaches to track information represented in long-term memory (Chadwick et al., 2010; Hsieh and Ranganath, 2014; Nielson et al., 2015; Robin, 2018). Most of these information-mapping studies emphasized the role of the hippocampus (Barnett et al., 2014; Thavabalasingam et al., 2018, 2019). Here, we endeavored to apply similar multivariate approaches to reveal whether the multivoxel patterns in the posterior medial parietal cortex—including the precuneus—contain information about temporal distances and, if so, whether these reflect context-independent chronological representations or context-dependent representations affected by storyline (Jonker et al., 2018).
Materials and Methods
Participants
A total of 19 right-handed native Italian speakers (age: 21–34, 7 male) participated in the experiment. All of them had normal or corrected-to-normal vision and no neurological impairments or cognitive dysfunction was reported. The study was approved by the Fondazione Santa Lucia (Scientific Institute of Research Hospitalization and Health Care) Independent Ethics Committee, in accordance with the Declaration of Helsinki. All participants gave written consent.
Experimental design
The current study used a modified version of the paradigm used in Kwok et al. (2012). Briefly, this comprises an encoding phase, when the participants watch an episode of the TV series “24,” followed by fMRI scanning during a temporal-order retrieval task 24 h later. During retrieval, the participants were asked to make temporal choices on two static images extracted from the encoded TV episode (“which image occurred first in the movie?”), Figure 1B. Across trials, the TD between the two events/pictures was varied parametrically with distances ranging between approx. 30 s and 32 min. This manipulation provided us with a measure of the influence of the temporal organization of events at encoding on subsequent memory retrieval.
In the original paradigm (Kwok et al., 2012), the pairs of images presented on each retrieval trial always belonged to the SAME storyline context. Specifically, the TV-episode included five concurrent storylines portraying different characters at various locations (see also “Stimuli and Task,” below). Here we addressed the question about the interplay between chronology and content-structure during memory retrieval by including trials comprising either pairs of pictures from the same storyline/context (condition: “SAME”) or from different-storyline/context (condition: “DIFF”; Fig. 1A). This allowed us to test whether any TD-dependent activity in the precuneus relates only to the chronological distance between the two events to be judged or, instead, combines this temporal information with knowledge about the content of the encoded material. The latter would support the hypothesis that the precuneus is involved not only in the temporal judgment of naturalistic events, but more specifically in reconstructive processes (see also Kwok et al., 2014). Moreover, we extended our investigation of TD-dependent effects using a multivariate information-mapping approach (RSA: representational similarity analysis). This allowed us to ask the additional question of whether the precuneus contains any representation of the temporal distances—that is, TDs mapped into corresponding distances in terms of multivariate patterns of brain activity—that would point to a possible role of chronological information during retrieval.
Stimuli and task
During the encoding phase, the subjects were instructed to watch and remember one episode of the American TV series “24” (Season 6, Episode 6, 11:00–12:00; duration = 42 min). None of the participants had watched any episode of Season 6 before. The episode contains five main storylines that are narrated in distinct contexts, with each including different characters and events occurring at different locations. The storylines are as follows: the president and his team in the white house; exchanges between inmates in a detention center; events in the office of the counter terrorism unit; the main character, agent Jack, on the move; and a middleman working for the terrorists and his girlfriend. During the retrieval phase, each trial consisted of the presentation of two images side-by-side and the participants were asked to report which image occurred earlier in the TV episode (temporal order judgment), Figure 1.
The extraction and the pairing of the images for the retrieval task was based on the detailed segmentation of the TV episode into 161 epochs, defined as periods comprising the same scenario/setting (see also Kwok et al., 2014; Kwok and Macaluso, 2015, for details). Each pair of images belonged to different epochs, either from the SAME or DIFF storylines. In total, we created 128 unique image pairs/trials for each storyline condition. The TDs were identical for the SAME and DIFF trials. The 128 trials were ranked according to TD and grouped in 32 bins, with average TDs ranging between 0.5 (bin1) and 32 min (bin32, see also Fig. 1). Each participant underwent four fMRI runs. Each run included 32 SAME trials (one from each bin) and the corresponding 32 DIFF trials, with identical TDs. Here we present the data of only two fMRI runs when the participants performed the temporal-order judgment task, whereas during the other two runs they performed a different task (temporal distance estimation) that will be reported elsewhere.
Each retrieval trial started with the presentation of a central fixation cross (duration = 500 ms). This was followed by the two test images that were presented on the screen for 5 s and it was followed by a 2 s fixation screen. The participants responded by pressing one of two buttons (left/right) with their right hand. The left/right position of the target image was counterbalanced. The 32 TDs and trials belonging to the SAME/DIFF conditions were presented in randomized order. The intertrial interval ranged between 4 and 12.5 s, with an exponential distribution (Dale, 1999).
Analyses of the behavioral data
Following the results of Kwok et al. (2012), we expected to find a parametric relationship between TDs and the reaction times (RTs) and we investigated any change of this as a function of context/storyline. First, separately for the SAME and DIFF conditions, the relationship between TD and RTs was analyzed using a hierarchical regression approach. We started testing for the linear relationship and progressively moved to higher-order relations (quadratic, cubic). At each step, we assessed the significance of the new predictor, having accounted for all the variance fitted by the lower-level predictors. We stopped this procedure when the F-value associated with the new predictor (e.g., cubic TD) was not significant (p > 0.05, Table 1).
Behavior results obtained from hierarchical multiple regression on RT or accuracy as a function of TD
Next, we formally compared the contextual effect of storyline on the effect of TD (i.e., the “TD × storyline” interaction) by applying the same hierarchical regression procedure, but now using the RT differences (RTsame − RTdifferent) as the dependent measure. The same procedure was also applied to the accuracy data. All the analyses of the behavioral data were performed using SPSS version 21.
fMRI
Image acquisition and preprocessing.
A Siemens Allegra 3 T scanner equipped for echoplanar imaging (EPI) was used to acquire functional magnetic resonance (MR) images. The data were acquired using gradient-echo EPI (32 contiguous transverse slices, covering the entire cerebral cortex; TR = 2.08 s, echo time = 30 ms, flip angle = 70°, 64 × 64 matrix, voxel size = 3 × 3 mm in-plane, slice thickness = 2.5 mm; 50% distance factor). For each participant we acquired 4 fMRI runs, each including 311 volumes. Here, we report the data of the two fMRI runs when the participants performed the temporal-order task. During the other two fMRI runs the participants performed a different retrieval task (temporal distance estimation) that will be reported elsewhere. The order of the two tasks was counterbalanced across participants and thus across participants the two relevant runs (i.e., those reported here) could occur at any point in the 4 runs sequence. By labeling the two types of runs/tasks as “A” and “B,” all possible sequences were used: AABB, ABAB, ABBA, BAAB, BABA, and BBAA. This should minimize any order effect in our analyses that also averaged across both runs of the same task.
In addition, a high-resolution T1-weighted MPRAGE structural scan of each participant was acquired as well (resolution isotropic 1 mm; TR = 2 s, TE = 4.38 ms, TI = 910 ms, flip angle = 8°, 256 mm × 256 mm in-plane resolution, 176 slices of 1 mm thick without gaps). Data preprocessing was performed in SPM12 (Wellcome Department of Cognitive Neurology) as implemented on MATLAB 2017a. The first four scans of each fMRI run were discarded to allow the signal to stabilize. The remaining scans were corrected for motion between slices (spatial realignment) and for slice-timing acquisition differences using the middle slice as a reference. Images were normalized to the Montreal Neurological Institute (MNI) standard space and resampled to 3 × 3 × 3.75 mm voxels. Normalization parameters were estimated using subject-specific structural T1 images and then applied to all functional images. For the univariate analyses, the images were spatially smoothed with an isotropic Gaussian kernel of 8 mm FWHM (full-width half-maximum); while un-smoothed images were used for the multivariate analyses.
fMRI univariate analysis.
The analyses of the imaging data followed a standard two step approach: first-level analyses estimating the effects of interest for each subject, followed by second-level analyses for statistical inference at the group level (Friston et al., 2002; Penny and Holmes, 2007). All the first-level models described below (both univariate and multivariate analyses) included six nuisance regressors modeling motion estimates (3 translations and 3 rotations), and the time-series were high-pass filtered by means of a discrete cosine basis-set removing low frequencies drifts up to 1/128 Hz and were prewhitened by means of an autoregressive model AR(1).
Based on the results of our behavioral analyses showing that TD modulated performance in a nonlinear manner (Fig. 2, Table 1), for the imaging analyses we also considered both linear and quadratic effects of TD. The first-level analysis modeled each trial with an individual predictor, resulting in 64 predictors for each fMRI run (i.e., 32 TD × 2 SAME/DIFF storyline conditions). Each trial was modeled as a miniblock with a duration of 5 s (equal to the duration of the presentation of the two scenes on the screen), time-locked to the onset of the scenes and convolved with the SPM12 hemodynamic response function. Linear contrasts were used to average the β images of corresponding TDs between the two fMRI runs, resulting in 64 contrast images for each participant (32 TDs × “SAME/DIFF” storyline). For statistical inference at the group level, the contrast images were entered in a repeated-measures ANOVA with 64 conditions. Sphericity correction was applied to account for possible differences in error variance across conditions and any nonindependent error term for the repeated measures (Friston et al., 2002).
An alternative approach to study the modulatory effect of TD in SAME versus DIFF conditions would be to construct first-level models comprising two predictors modeling the “categorical” effect of SAME and DIFF events, plus two predictors accounting for the modulatory effect of TD on each of the two conditions. The two approaches, albeit statistically not identical, are conceptually analogous and indeed our main results could be fully replicated using this second approach. Here, we choose to report the results modeling each event with a single predictor for two reasons: First, the RSA analyses (see below) requires estimating the responses for each single event/TD, thus by doing this also for the univariate analyses the same modeling approach was applied in a consistent manner in the different analyses. Second, by estimating TD-specific parameters in the first-level analyses and performing TD-related contrasts at the group level, we can visualize the changes of activity as a function of TD (and storyline), see signal plots in Figure 3B,C. The latter appears relevant given the complexity of the tested contrasts (cf. Fig. 3A).
In the group-level ANOVA, we addressed our main question related to the TDs and the interaction between TDs and storyline/context. Based on the results of the behavioral data (Fig. 2), we tested for both the linear and the quadratic effect of TDs on fMRI activity. To test for the quadratic relationship having accounted for any linear effect (cf. hierarchical regression used for the behavioral analyses), we regressed out the linear effect of TD from the quadratic contrast. Linear and quadratic contrasts were first assessed separately for the SAME and DIFF conditions. Then we formally tested for the impact of storyline on TDs (TD × context interaction) by multiplying by “−1” the contrast weights of the DIFF condition (Fig. 3A, illustrating the relevant contrasts). With the same ANOVA model, we also tested for the main effect of storyline comparing the 32 TD conditions of the SAME storyline minus 32 TD conditions of the DIFF storyline. We report activations corrected for multiple comparisons at the cluster level, considering the whole brain as the volume of interest (FWE p-corr. < 0.05, cluster-size estimated at a voxel-level threshold of p-unc. = 0.005).
fMRI multivariate analysis.
We made use of a RSA (Kriegeskorte and Kievit, 2013) to test the hypothesis that TDs would be mapped into corresponding differences in terms of multivariate pattern of fMRI activity. Specifically, RSA allowed us to target multivariate representations related to a specific stimulus-dimension (i.e., the TDs that were identical for trials involving SAME and DIFF storylines) that may differ from univariate activations, which instead may be linked with context/task-dependent readout of these representations (cf. interaction between TD and storylines). Moreover, similar multivariate approaches have been successfully used to investigate temporal representations in memory, but typically focusing on the medial temporal lobe (Thavabalasingam et al., 2018, 2019). Relying on the same analytical approach for interpretation, we can thus more readily relate our current results that the parietal cortex is sensitive to both temporal and contextual information to the hippocampal-related literature. The RSA analyses were performed with CoSMo toolbox (http://www.cosmomvpa.org/; Oosterhof et al., 2016). Based on our behavioral and univariate fMRI results we constructed two representational dissimilarity matrices (RDMs). These coded for the pairwise differences between TDs (RDM linear) and the differences between the squared TDs having first regressed out the linear effect (RDM quadratic; cf. behavioral and univariate fMRI analyses; Fig. 4A).
For each participant, we re-ran the same first-level model as described above (64 conditions: 32 TDs × SAME/DIFF context, plus 6 movement parameters), but now using unsmoothed fMRI time series. Contrast images were used to average the parameters estimates of each TD between the two fMRI runs. The 32 contrast images of the SAME condition were first used to estimate the individual RSA using a whole-brain, searchlight approach (sphere with a radius of 3 voxels, comprising 100 voxels). The resulting correlation coefficients (Spearman's rank coefficients) were Fisher transformed for the group analyses. Two separate one-sample t tests (RDM-linear and RDM-quadratic) were used to identify any representation of TDs at the group-level using SPM12. The statistical threshold was set to cluster-level FWE p-corr. < 0.05, corrected for multiple comparisons considering at the whole brain as the volume of interest (cluster-size estimated at a voxel-level threshold of p-unc. = 0.005).
We considered two possible types of representations: one directly (linearly) related to the TDs and one related to higher-order (quadratic) relationships, in accordance to the behavior and univariate results. Whole-brain searchlight analyses (100-voxel sphere) compared patterns of local activity for the 32 TDs of the two conditions (SAME and DIFF) with two representational models based on pairwise distances between the TDs (RDM-linear) or the TD2 (RDM-quadratic, Fig. 3A). The whole brain analysis of the SAME condition revealed a significant effect in the precuneus for the RDM-linear (Fig. 4B). Accordingly, we asked whether this region contained an analogous (i.e., linear) representation of TDs also for the DIFF condition. For each subject, we assessed the multivariate patterns for the DIFF condition using the same set of 100 voxels that yield the highest significance in the SAME condition at the group level (ROI analysis, sphere centered at x-y-z = [9 −61 54]; Table 2, last row). The linear RDM was identical to the one used for the SAME condition, because the TDs were the same in the two context-conditions. A one-tailed t test on the Fisher-transformed correlation coefficients was used to assess the significance of the RSA for the DIFF condition at the group level.
Functional imaging: univariate and multivariate analyses
Results
Behavioral results
Behaviorally we examined the change of the RTs as a function of TD and investigated any modulation of this relationship as a function of storyline. Seeking to replicate the results of Kwok et al. (2012), we initially ran simple linear regression between the mean RTs and TDs for the SAME condition. This reveled the expected decrease of the RTs with increasing TDs (p < 0.005), but the regression plot also indicated that the linear model did not account well for the data. Accordingly, we performed a hierarchical multiple regression analysis fitting progressively higher-order functions of the TD (quadratic TD2, cubic TD3; Fig. 2A). This reveled that adding TD2 to the regression model significantly increased the fitting of the RTs, whereas there was no significant additional gain by adding the cubic term (Table 1). We performed the same analysis for DIFF condition and found that now only the linear term approached significance (p < 0.057, Fig. 2A, plot in the center, Table 1). We formally compared the differential effect of TDs on RTs in the two storyline conditions by analyzing the RT differences with the hierarchical regression approach (RTsame − RTdifferent; Fig. 1, rightmost plot). This confirmed that context/storyline modulated the relationship between TD and RTs, highlighting a significant effect on the nonlinear, quadratic term (Table 1, last column).
Storylines and task. A, Schematic illustration of the segmentation of the TV episode in five different storylines. Pairs of frames belonging either to the same or different storylines (SAME vs DIFF conditions) were extracted for the subsequent temporal-order judgment task. Across trials, the TD between the two frames was varied parametrically, but identical TDs were selected for the two storyline conditions (see example of 713 s in the illustration). B, Two example trials of the temporal order judgment task. During fMRI, the participants were presented with pairs of frames extracted from the movie they had seen the day before and they were asked to report which scene happened earlier in the movie. The example at the top corresponds to a SAME storyline trial (both frames from “storyline 2”), whereas the bottom shows a trial of the DIFF storyline (one frame belonging to “storyline 4” and the other to “storyline 3”).
Behavior. Nonlinear effects of temporal distance on: RTs (A) and retrieval accuracy (B), modulated by context–storyline. Data fitting using linear (black dotted line), quadratic (black continuous line), and cubic (gray line) models. Hierarchical multiple regressions highlighted a significant contribution of the quadratic-term for the SAME condition only and confirmed this context-specificity when fitting the pairwise differences “SAME minus DIFF”, see rightmost panels.
The same hierarchical regression analysis was repeated using the accuracy data. This replicated the results of the RTs analysis, again showing a nonlinear effect of TD (quadratic term, TD2) on the retrieval accuracy in the SAME condition and corresponding differential effects when directly comparing SAME versus DIFF conditions (Fig. 2B, Table 1).
The analyses of the behavioral data revealed the expected effect TDs on performance and, most importantly, that this relationship changed according to the storyline/context. Notably, the hierarchical regression model highlighted that in the SAME condition both the RTs and the retrieval accuracy did not simply change linearly with TDs. Instead, a significant nonlinear component (here captured by TD2) was found to modulate performance: the participants were slower and less accurate at intermediate temporal distances (∼10–15 min) rather than at the shortest TDs (i.e., <5 min). Moreover, we found that storyline/context modulated specifically this nonlinear term (see SAME vs DIFF in Table 1 and the rightmost plots in Fig. 2).
The subsequent analyses of the fMRI data took into account the presence of both linear and nonlinear effects of TDs and sought to identify their correlates using univariate and multivariate approaches.
fMRI results
Univariate analyses
Based on our behavioral findings, we investigated the effect of TD on brain activity in the SAME condition, using two different contrasts. These corresponded to the linear effect of TD and the effect of TD2 having removed the linear component. The contrast weights are illustrated in the first two plots of Figure 3A. The linear TD contrast did not reveal any significant activation, whereas the quadratic-TD contrast highlighted two significant clusters of activation. These were located in the precuneus and in the right inferior parietal lobule (IPL) including the posterior superior temporal sulcus (pSTS) and the angular gyrus (AG) (Table 2). Figure 3B shows the anatomical location of the two clusters and the corresponding parameter estimates for the 32 TDs in the SAME condition. In both regions, the BOLD signal displayed an inverted U-shape profile, with the largest activations at intermediate TDs (corresponding to ∼10–15 min), whereas activity was lower both at long TDs (>20 min) and at short TD (< 5 min). The plots also show the best-fitting line for the TD2 nonlinear term (see black dotted-line). This highlighted that in the precuneus the BOLD data substantially deviated from the quadratic-fit in the very last bin (i.e., the longest TD). This was confirmed by examining the residuals of the quadratic-fit revealing that in the last bin the error was larger than 2 times the SD of the residuals. Based on this observation, the last TD was excluded from the subsequent analysis when testing for the interaction between TD and storylines.
TD: Univariate fMRI. A, Illustration of the contrasts used to investigate the effect of TD: linear component considering the SAME condition only; quadratic component (SAME only), having removed the linear component; and the differential contrast used to compare the quadratic effect of TD in the SAME vs DIFF conditions. It should be noticed that the last contrast (cf. also C) considered only 31 TDs for each storyline-condition (i.e., the weights of the longest TD were set equal to zero; see also main text). B, Significant effects of quadratic-TD for the SAME condition, in the precuneus and the right pSTS/inferior parietal lobule (IPL), and the corresponding signal plots as a function of TD (quadratic fits shown in black dotted lines; C) Interaction between quadratic TD and storyline in the precuneus, considering 31 TDs for each storyline. Clusters are displayed at p-FWE-corr. = 0.05 (cluster-level, with cluster-size defined at voxel-level p-unc. = 0.005). All sections are taken through the peak of activation (cf. Table 2).
We then compared the effect of TD between the two context/storylines conditions by multiplying by “−1” the weights of the linear-TD and quadratic-TD contrasts (see rightmost plot in Fig. 3A, showing the relevant interaction contrast for the quadratic effect). The comparison using linear-TD did not reveal any significant effect, whereas the contrast considering the quadratic term showed a significant effect in the precuneus (Table 2). These effects are illustrated in Figure 3C, which shows the significant cluster in the precuneus and the signal plot with the effect of TD in the two storyline conditions. The parameter estimates highlight that the quadratic effect found in the SAME condition (compare Fig. 3B) was not present in the DIFF condition. At an uncorrected threshold, this interaction between TD and storyline was found also in the right pSTS/IPL (x,y,z = 42 −55 20, Z-value = 3.45, p-unc. < 0.001).
To further address the relevance of context/storyline during temporal-order retrieval, we tested for the main effect of storyline regardless of TD. We first compared SAME > DIFF conditions and revealed two significant clusters located in the AG/posterior part of the right superior temporal sulcus (pSTS) and in the posterior part of the right inferior frontal gyrus (IFG), Table 2. The cluster in the pSTS/IPL overlapped with the region showing a (quadratic) modulation according to TD and the significant TD-by-storyline interaction (see above). The opposite contrast (DIFF > SAME) did not reveal any significant activation.
These results show that temporal distances impacted on both activity in the precuneus and behavior (RTs and accuracy), including analogous nonlinear effects selectively in the SAME storyline condition (cf. Figs. 2, 3C, Table 1). It is thus possible that our findings in the precuneus simply reflect different retrieval success performances, driven by TDs and/or storyline. Accordingly, as a control analysis, we generated new contrasts based on retrieval accuracy for each TD and both storylines and tested for any (positive or negative) linear relation between BOLD and accuracy. These control contrasts did not show any significant activation, indicating that the differential behavioral performance is an unlikely explanation for the effects that we report in the precuneus. Moreover, we tested an analogous contrast, but now considering TD-related changes of RTs rather than accuracy. This revealed a statistical trend in the medial supplementary motor area (x, y, z = 0, 17, 50; Z-score = 4.54; p-corr = 0.096), with activation increasing with slower RTs. This may indeed relate to changes of task difficulty across TDs, yet it could not explain our main results in the precuneus.
In sum, the univariate fMRI analyses revealed that activity in the precuneus, and also the inferior parietal lobule (including the posterior part of the superior temporal sulcus extending into the AG), closely matched the effects observed at the behavioral level, with a nonlinear effect reflecting a TD × storyline interaction, with greatest activation at intermediate TDs and specifically in the SAME condition.
Multivariate analyses
Together with any impact of TD on BOLD activation, we used a multivariate approach (RSA) to assess whether TDs are represented in terms of local patterns of activity. First, we considered multivoxel patterns during the SAME condition. One-sample t tests on Fisher-transformed correlation coefficients mapped at each voxel in the brain (voxel corresponding to the center of the searchlight-sphere) were used for statistical inference at the group level (Fig. 4). Unlike the univariate analyses presented above, the significant effects were found for the model based on TDs distances (RDM linear), whereas the model based on distances between quadratic TDs did not show any significant effect after correction for multiple comparisons. The RDM linear whole-brain analysis revealed a significant cluster located in the precuneus, plus a related effect in the right AG/pSTS (Fig. 4B, Table 2). Given the significant finding in the precuneus for the SAME condition, we tested whether this same region contains a related (i.e., linear) representation of TDs also for the DIFF condition. We considered the patterns of activity for the 32 TD of the DIFF condition, targeting specifically the 100 voxels where we detected the most significant effect in the SAME condition (ROI analysis, using the RDM-linear). At the group level, the one-tailed t test on the Fisher-transformed coefficients revealed a statistical trend (t(18) = 1.35; p = 0.097), supportive of the notion of context-independent representation of TDs in the precuneus.
TD: Multivariate fMRI. A, Illustration of the two RDMs used to assess multivariate activity related either to linear TD or to quadratic TD, having removed the linear component (see also behavior and univariate fMRI). B, The whole-brain searchlight RSA analysis using the linear RDM model revealed a significant cluster in the precuneus for the SAME condition. A targeted ROI analysis of the precuneus now considering the DIFF condition suggested that this linear representation of the TDs is independent of context–storyline (statistical trend only, see main text). Significant cluster displayed at p-FWE-corr. = 0.05 (cluster-level, voxel-level p-unc. = 0.005) and section through the peak voxel.
Discussion
We investigated the role of contextual, high-level information during temporal order retrieval using complex and dynamic naturalistic movie-stimuli. Our main findings concern the context-dependent effects in relation to temporal distances and storyline contexts. Our results showed that, on the one hand, the retrieval performance and the activation of the precuneus were significantly modulated by the content-structure of the encoded material (storylines), consistent with mechanisms of temporal-order retrieval relying on high-level contextual information (e.g., reconstructive view, Friedman, 1993, 2007). On the other hand, information-mapping using multivoxel analyses revealed that the precuneus contains a context-independent linear representation of the temporal distances, consistent with the view of chronological organization of memory (Friedman, 1996, 2001).
Since we obtained different effects of TD for the two storyline conditions, the effects observed here should be related to retrieval processes taking into account high-level contextual information/knowledge structures (Hassabis and Maguire, 2007; Gilboa and Marlatte, 2017). Operationally, here we considered the storylines as the relevant dimension of the movie knowledge structure and postulated that, if the effect of TD in the precuneus reflects knowledge-based processes, then the relationship between the probes (i.e., the two events to be discriminated) and the storyline structure should affect memory retrieval. Previous work with movies highlighted the importance of segmentation/boundaries both for perception (Zacks et al., 2001) and memory (Swallow et al., 2009; see also Kwok and Macaluso, 2015). A more recent study has identified a distributed pattern of brain activity in higher-order cortical networks, including the precuneus and AG, that appears to underlie ability to relate relevant memories with ongoing events in a complex movie story (Kauttonen et al., 2018). Nonetheless, this study did not manipulate experimentally memory retrieval from SAME versus DIFF storylines/context.
Our results showed context/storyline modulated TD-dependent retrieval performance (Fig. 2) and corresponding fMRI activity in the precuneus (Fig. 3C). The relationship between TD and brain activity in the SAME condition was nonlinear, possibly reflecting the specific content-structure of the movie. Specifically, at intermediate TDs the similarity between the two to-be-judged events in “content structure space” may be smaller than for pairs of events occurring at shorter or longer distances. Search processes making use of the content-structure would thus require more resources (cf. behavioral results) to determine the order of the two events. This content structure space account aligns with the original idea of “chronoarchitecture,” which stipulates that there is a functional subdivision that exploits the hierarchical temporal receptive windows during natural viewing conditions (Bartels and Zeki, 2004, 2005). Successively higher areas (such as the precuneus; see Baldassano et al., 2017) have longer temporal windows to process larger units of information, such as narratives and storylines (Hasson et al., 2015). Our current results indicate that the principles of “temporal receptive windows” may go beyond mere temporal information, with context/content also playing a role during temporal order retrieval (Hasson et al., 2015).
Situational models, as proposed in the AT/PM model (Ranganath and Ritchey, 2012), can be considered to be the apex of this hierarchical memory system. The AT/PM model distinguishes between high-level signals in terms of general semantic knowledge (associated with the anterior temporal system) versus the multidimensional, structured information associated with specific episodes (i.e., “situational models” in the PM network, including the precuneus). Thus, whereas the AT network may relate to the assessment of entities such as people and things (Gilboa and Marlatte, 2017), the PM network would perform reconstructive operations to integrate spatial, temporal, as well as higher-level (e.g., situational or social) information associated with the stored episode (Ranganath and Ritchey, 2012). Here, we suggest that the precuneus makes use of the specific content-structure of the movie to construct a situational model, which in turn would support the retrieval of the associated temporal information about the encoded events (see also Kauttonen et al., 2018). This account fits with various studies showing that personal experiences throughout life (Conway and Jobson, 2012), statistical properties of the environment (van Buuren et al., 2014), semantic knowledge (Packard et al., 2017), or some combination of these, can constitute the scaffolding of such knowledge-structures (Xue, 2018).
Together with the precuneus, we found TD-dependent activation in the right inferior parietal lobule covering the AG and part of the superior temporal sulcus. These regions, together with the right IFG, also showed a main effect of storyline, with larger activation during the judgment of SAME compared with DIFF storyline events (Table 2). The AG is often coactivated with the precuneus (Kwok et al., 2012) and is known to be an important hub in memory recollection (Vilberg and Rugg, 2012; Tibon et al., 2019). This region is also part of the dorsal medial subsystem of the default mode network, which has been linked with story comprehension (Spreng and Andrews-Hanna, 2015), schema-mediated encoding (Gilboa and Marlatte, 2017) and with high-level situation model representations (Baldassano et al., 2017). With respect to the AT/PM dual system framework, several imaging studies linked the activation of the precuneus with the engagement of the default mode network (Spreng et al., 2009; Gilmore et al., 2015; Sestieri et al., 2017) during retrieval of episodes encoded in highly structured contexts (e.g., autobiographical memory; Addis et al., 2012; Chow and Rissman, 2017; Monge et al., 2018). Putting these together, the precuneus and the AG might work in concert to help deploying conceptual knowledge and supporting temporal judgments about naturalistic episodes. While watching the action TV show, the participants may have engaged some prior knowledge structures that they had acquired before the experiment. It is likely that some schema instantiation would have occurred during viewing and might even subsequently affect the reconstructive processes during memory retrieval. However, in the context of the current temporal order task, the retrieval demands during scanning may not engage such systems in a systematic fashion across temporal delays. This may explain the lack of any delta-related modulation in the ventromedial prefrontal cortex and the hippocampus, which have been consistently associated with schema instantiation, memory assimilation and schema adaptation processes (Gilboa and Marlatte, 2017).
Together with our main finding highlighting the role of contextual information in modulating TD-related activation of the precuneus, we also found context-independent effects in the precuneus, using a multivariate approach. These results can be related to previous findings which identified representations of temporal information in long-term memories. For example, Thavabalasingam and colleagues associated multivoxel pattern in the human hippocampus with the integration of duration information of short event sequences (Thavabalasingam et al., 2018, 2019). In Barnett et al. (2014)'s match-mismatch task, when a duration-between-events mismatch was detected (compared with match condition), the hippocampus showed increased connectivity to the posterior cingulate cortex. The authors interpreted this finding as evidence that the hippocampus functions in concert with the posterior cingulate cortex for the processing of contextual information and events-sequence binding in memory. This latter finding aligns with our current results that the precuneus/posterior cingulate cortex is sensitive to both temporal and contextual information, as manifested in the significant temporal-distance by context–storyline interaction (Fig. 3C). Moreover, Zeithamova and Preston (2017) showed that, during retrieval, multivoxel activity in frontal regions discriminate whether two objects were encoded on the “same” versus “different” days (see also Nielson et al. (2015) for related results concerning the representation of temporal information about autobiographical events in the hippocampus). It is noteworthy that since most of these previous works rely on multivariate methods in assessing the neural correlates of memories, it would be of importance of applying the same analytical methods such as RSA for comparative interpretation.
All in all, in addition to the hippocampal representations, we add to the extant literature that temporal memories are also represented in the precuneus when multivoxel information is considered. However, it remains an open question how the precuneus works in concert with the hippocampus during memory formation (Brodt et al., 2016), consolidation (Brodt et al., 2018) and memory retrieval monitoring (Ye et al., 2019). As suggested previously, the intimate coupling between the hippocampus, AG and posterior medial cortex may mediate the storage of event memories at multiple temporal resolutions (Baldassano et al., 2017).
In summary, we showed the neural correlates of how contextual signals about the content of complex and naturalistic stimuli contribute to memory retrieval. We demonstrated that the activation in the precuneus, as well as behavior, was significantly modulated according to storyline condition, supporting the idea that contextual information contributing to temporal retrieval. We suggest that the precuneus engages in memory retrieval via reconstructive mechanisms, which may rely on long temporal receptive windows to process larger units of information such as storylines (Hasson et al., 2015; Baldassano et al., 2017), within an episodically unique knowledge structure (“situational models”; Ranganath and Ritchey, 2012). Furthermore, information-based analyses of multivoxel activity revealed that the precuneus contains a context-independent representation of temporal distances, consistent with other studies on duration information and their integration into representations of events (Barnett et al., 2014; Thavabalasingam et al., 2018, 2019). We put forward that the retrieval of the temporal-order of naturalistic events encoded in rich contexts relies on the joint contribution of chronological and reconstructive mechanisms and that these may be jointly integrated in the medioposterior parietal cortex.
Footnotes
This work was supported by the European Research Council under the European Union's Seventh Framework Program (FP7/2007-2013/ERC Grant 242809). E.M. is supported by the program “Investissements d'Avenir” (ANR-11-IDEX-0007) and by a chair INSERM-UCBL1. S.C.K. and E.M. are jointly sponsored by Joint Research Institute for Science and Society (JORISS) Incubating Project Grant 2017/4. S.C.K. acknowledges internal funding from NYU Shanghai and NYU-ECNU Institute of Brain and Cognitive Science at NYU Shanghai.
The authors declare no competing financial interests.
- Correspondence should be addressed to Sze Chai Kwok at sze-chai.kwok{at}st-hughs.oxon.org
