Abstract
Structures in the medial portions of the human temporal lobes (MTL) play a vital role in the ability to learn new facts and events, whether such learning is intentional or incidental. We examined neural activity in the MTL both while participants studied pictures of novel scenes and while they attempted to recognize which scenes had been previously presented. In a second surprise test we assessed participants' memory for items that were presented only during the previous recognition memory test. We present a novel approach to cross-participant alignment of neuroimaging data that provides more precise localization and enhanced statistical power within regions such as the MTL. Using this technique, we observed that the amount of MTL activity predicted participants' ability to subsequently remember scenes not only during the intentional study task, but also during the first memory retrieval test when only incidental encoding occurred. This encoding-related activity during memory retrieval was in the same subregions of the MTL as encoding-related activity during intentional study and is hypothesized to be one of the primary reasons why retrieval-related activity is often difficult to observe with neuroimaging techniques.
Introduction
Damage to structures in the medial portions of the temporal lobe (MTL) in humans results in a profound deficit in the ability to learn new facts and events and in a temporally graded impairment in the ability to retrieve memory for facts and events that were learned before the damage (Milner et al., 1998). This suggests that, at least for some period of time, the MTL is vital for declarative memory during both encoding and retrieval tasks. Neuroimaging studies have strongly supported the role of the MTL in encoding, demonstrating greater activity during intentional or incidental study tasks for novel items than for familiar items that is lateralized based on the type of information (e.g., words vs pictures) to be learned (Stern et al., 1996; Brewer et al., 1998; Kelley et al., 1998; Wagner et al., 1998; Martin, 1999; Zeineh et al., 2000; Stark and Squire, 2001b; Reber et al., 2002; Strange et al., 2002). Furthermore, the amount of activity at study for novel items has been shown to predict subsequent memory performance (Brewer et al., 1998; Wagner et al., 1998; Fernandez et al., 1999; Otten et al., 2001; Davachi and Wagner, 2002; Fernandez et al., 2002; Strange et al., 2002; Davachi et al., 2003). Neuroimaging studies of retrieval have been less clear in their support. Although several studies have successfully observed MTL activity related to retrieval success using functional magnetic resonance imaging (fMRI) (Gabrieli et al., 1997; Eldridge et al., 2000; Stark and Squire, 2000a,b; Cabeza et al., 2001), depth electrode recordings (Paller and McCarthy, 2002), and magnetic source imaging (Papanicolaou et al., 2002), these effects have been notoriously difficult to observe, with numerous studies of retrieval failing to observe any activity related to retrieval success within the MTL.
Here, we present data suggesting that the failures to observe MTL activity related to retrieval success are not the result of a lack of reliance on the MTL for memory retrieval. Rather, we demonstrate that a failure to observe MTL activity associated with retrieval success can be a result of activity associated with incidental memory encoding while participants are performing the memory-retrieval task. We show that not only does MTL activity during an intentional memory task predict subsequent memory performance, but also that MTL activity during a memory-retrieval task predicts subsequent memory for novel foils presented only during the test. This encoding-related activity can be strong enough to obscure retrieval-related activity in traditional measures of activity related to retrieval success (e.g., contrasts between studied targets and unstudied foils). Such incidental encoding-related activity has often assumed to be the source of many null results at retrieval (Martin, 1999; Schacter and Wagner, 1999; Stark and Squire, 2000a,b), but has never been observed this directly (see Discussion). Furthermore, this observation also serves to demonstrate that the assumption that participants are engaged only in the task at hand (an assumption present in the vast majority of neuroimaging studies) can be unfounded.
Materials and Methods
Overview. The experiment was divided into three phases (Fig. 1), with the first two phases conducted with participants inside the fMRI scanner (see Buckner et al., 2001 for another use of this design and for results outside of the MTL). In the first phase (study), participants were shown a series of color pictures of outdoor scenes (list A) and asked to study each picture for a later test while fMRI data were collected. After a 10 min delay, the first recognition memory test (test 1) was administered while fMRI data were collected. During this test, participants were shown both the studied pictures (list A, targets) and a set of novel pictures that were not presented during the study phase (list B, foils). They were asked to press a key to indicate whether they recognized the picture as having been presented earlier in the study phase. After test 1, participants were removed from the scanner and a surprise recognition memory test was administered (test 2). During this test, participants were shown the pictures they had originally studied (list A, targets), the pictures that had been presented as unstudied foil items during test 1 (list B, targets), and a novel set of pictures that had not been presented at any time during the experiment (list C, foils). In test 2, participants were asked to indicate whether they recognized the picture as having been presented at any time during the experiment. Thus, they would ideally respond “yes” to items from both list A and list B, and “no” to items from list C.
As shown in Figure 1, the behavioral data from test 1 was then used to code trials during the study phase into those that were subsequently remembered (study-R) and those that were subsequently forgotten (study-F). The fMRI data from the study phase were then analyzed to determine whether the amount of activity during this intentional study task would predict participants' ability to recognize the items in test 1. This contrast between the activity at study for items that are subsequently remembered (study-R) and those that are subsequently forgotten (study-F) was initially observed in the frontal lobes and in the posterior portions of the parahippocampal gyrus that blended into the fusiform gyrus (Brewer et al., 1998; Wagner et al., 1998) and has subsequently been observed more extensively in the MTL, including the hippocampal region (Kirchhoff et al., 2000; Otten et al., 2001; Davachi and Wagner, 2002; Reber et al., 2002; Strange et al., 2002; Davachi et al., 2003).
The behavioral data from tests 1 and 2 were used to code trials during test 1 that were associated with successful recognition of studied items (hits), failure to recognize study items (misses), false recognition of unstudied items (false alarms), and correct rejections of unstudied items. These correct rejections were divided into those that were later recognized in test 2 (CR-R) and those that were later forgotten in test 2 (CR-F) so that the same style of subsequent memory effect could be examined during the recognition test itself.
Participants. Thirteen men and seven women participated in the experiment (18 to 41 years old; mean age, 23). All were right-handed native English speakers and all provided written, informed consent before participation. Test phase data from one volunteer was discarded because of excessive motion and all data from a second volunteer was discarded because of exceptionally poor coverage of the MTL.
Behavioral tasks. Before the study session began, participants practiced a simple odd/even judgment task on a series of 40 digits. This task has been previously demonstrated to provide a more suitable baseline for activity in the MTL than even short periods of a blank screen (Stark and Squire, 2001b). During the study phase 100 pictures were presented [2.5 sec duration, 0.5 sec intertrial interval (ITI)] along with 50 pairs of odd/even digit trials (1.25 sec duration, 0.25 sec ITI each), the latter serving as “null events” that induce jitter into the interstimulus interval, allowing for a more accurate estimate of the hemodynamic response to individual trial types. Assignment of list (A-C) to condition (study items, foil items in test 1, foil items in test 2) was counterbalanced across participants.
During test 1, all 100 targets and 100 foils were presented in a random order (2.5 sec duration, 0.5 sec ITI), with participants indicating their response by pressing one of two keys on a fiber optic button box. In addition, 100 pairs of odd/even digit trials were randomly interspersed throughout the task. Test 1 was split into two 7 min scans. Test 2 was self-paced and conducted outside the scanner. A total of 300 pictures were presented (100 study items, 100 items that served as foils in test 1, and 100 novel foils) and participants responded using a four-choice scale (sure old, probably old, probably new, and sure new) to assess confidence. To increase our statistical power, the confidence scale was collapsed into “old” and “new” judgments during data analysis.
fMRI scanning. Imaging was performed on a Philips Gyroscan 1.5 T MRI scanner equipped with a head coil. 16 T2*-weighted triple-oblique axial functional images were collected per three-dimensional volume (316 volumes at study and 304 volumes in each half of test 1) using a single-shot echo-planar pulse sequence (64 × 64 matrix, echo time, 43 msec; flip-angle, 90°; in-plane resolution, 3.59 × 3.59 mm; thickness, 4 mm; repetition time, 1.5 sec; 4 initial volumes for T1 stabilization). Slices were aligned with the principal axis of the left and right hippocampus to optimize the signal from the medial temporal lobes and to minimize partial-voluming effects (so that voxels could be clearly constrained to lie within subregions of the MTL). This acquisition was chosen to optimize the quality of our data from the MTL, but did come at the cost of coverage outside of the MTL. During the delay between the study phase and test 1, a high-resolution structural MRI was acquired (magnetization-prepared rapid gradient echo pulse sequence, 1 mm3 resolution, 180 triple-oblique axial slices in the same orientation as the functional images) for anatomical localization.
fMRI analyses. Image analysis was done using a combination of AFNI (http://afni.nimh.nih.gov), the MINC toolset from the Montreal Neurological Institute (ftp://ftp.bic.mni.mcgill.ca/pub/minc/). Functional images were coregistered in time and space and the data from the two halves of test 1 were concatenated. A general linear model was constructed for the fMRI data from the study phase using a deconvolution and multiple regression approach coding for two trial types (hits in test 1 and misses in test 1) at 10 time points each (0-15 sec poststimulus onset) along with eight nuisance vectors (six for three-dimensional head motion detected two for drift). A similar general linear model was constructed to analyze the fMRI data from test 1, but used behavioral vectors coding for hits, misses, correct rejections recognized in test 2, correct rejections forgotten in test 2, and false alarms.
The sum of the β-coefficients for the time points corresponding to the expected peak in the hemodynamic response (∼3-9 sec poststimulus onset) was taken as the estimate of the response of the model to each trial type. These maps were spatially normalized to align participants' MTLs (see below), and resampled to 2.5 mm3 resolution in the processes. After a slight burring of the maps (4 mm Gaussian), repeated-measures ANOVAs were used to test for differences in activity as a function of trial type (treating participant as a random effect). In the study phase, two trial types were entered (study-R and study-F) into the ANOVA, and at testing, the three trial types of interest (hits, CR-R, and CR-F) were entered into the ANOVA to identify regions whose activity varied in any way by trial type of interest. All tests were bidirectional and used a voxel threshold and a spatial threshold to correct for multiple comparisons within the voxels of the MTL for a corrected α-threshold of p < 0.05 (individual voxel, p < 0.02; minimum cluster size, 188 mm3).
Unfortunately, signal drop-off associated with the distortions from the sinus cavities significantly reduces what one can observe in the entorhinal cortex. The lack of significant effects in the entorhinal cortex cannot be interpreted as meaningful, because there was little to no fMRI signal of any kind from the entorhinal cortex in a number of participants.
MTL alignment with ROI-AL. Our ability to resolve the effects of incidental encoding during retrieval and to localize them to specific regions within the MTL was greatly enhanced by an improved method for cross-participant alignment of MRI scans presented here. In previous work, limitations of the techniques used for the alignment and coregistration of participants' brains have restricted the ability to resolve effects and have confidence in their specific location within the MTL. Even when such subsequent-memory effects have been observed within the hippocampal region or within what appears to be adjacent perirhinal cortex (Otten et al., 2001; Strange et al., 2002), the notoriously poor cross-participant alignment of structures within the MTL not only reduces confidence in the precise localization of such findings (e.g., hippocampus vs entorhinal cortex vs perirhinal cortex), but also reduces the statistical power available (e.g., forcing researchers to use tests that are not corrected for multiple comparisons).
The technique demonstrated here (ROI-AL) aligns participants' brains based on anatomically defined regions of interest. Unlike the ubiquitous alignment of brains to the atlas of Talairach and Tournoux (1988) or to the collection of 305 normalized brains assembled by the Montreal Neurological Institute (Collins et al., 1994), ROI-AL optimizes regional alignment at the expense of whole-brain alignment to increase the statistical power and precision of localization within anatomically defined regions of the MTL (see below for additional details and Zeineh et al., 2000, for an alternative technique).
Figure 2 shows a comparison between the average structural image across 20 participants aligned using traditional Talairach techniques (Fig. 2a) and aligned using ROI-AL (Fig. 2b). ROI-AL produced a clearly superior alignment within or near the chosen ROIs (note the definition in the collateral sulcus, indicated by the arrows). The improved alignment resulted in an improved ability to localize activity both by a reduction in blurring and by the ability to determine where a particular active voxel lies within the modal model. Furthermore, the improved alignment resulted in increased statistical power over traditional Talairach alignment. This can be demonstrated by comparing the distribution of various cross-participant statistical maps within voxels of the MTL. A particularly clean comparison, shown in Figure 2c, is between the distribution of the absolute value of the t statistics for the subsequent memory effect in the MTL during the study phase (study-R vs study-F). The difference in the histograms demonstrate a shift toward higher magnitude t values using the ROI-AL technique (a Kolmogorov-Smirnov test on the 1525 voxels within the MTL indicated a significant difference in the distributions; Z = 1.575, p < 0.02).
To align the regions of the MTL, each participant's perirhinal cortex, temporopolar cortex, and entorhinal cortex were first defined bilaterally according to the techniques described by Insausti et al. (1998). The parahippocampal cortex was further defined bilaterally as the portion of the parahippocampal gyrus caudal to the perirhinal cortex and rostral to the splenium of the corpus callosum. The hippocampal region (the CA fields of the hippocampus, the dentate gyrus, and the subiculum) was also defined bilaterally. All anatomical ROI definitions were conducted after alignment to the Talairach atlas (Talairach and Tournoux, 1988) to provide the ROI-AL alignment technique with a reasonable rough alignment to use as a starting point.
These 10 anatomically defined ROIs were then used as the basis for constructing a 12-parameter transformation matrix (3 rotations, 3 translations, 3 scales, and 3 shears) to align the ROIs optimally across participants. A single participant was first chosen to serve as the model to which all other participants would be initially aligned. An objective function that calculates the percentage of voxels labeled in the model that are identically labeled in the individual participant was coded into the minctracc program provided in the MINC toolset (ftp://ftp.bic.mni.mcgill.ca). The 12-parameter transformation matrix was then applied to each participant's map of ROIs, so that they would be optimally aligned to the representative participant.
The set of all 20 participants' ROI-AL aligned ROIs was then merged to create a single modal map of the ROIs within the MTL. A new 12-parameter transformation matrix was then calculated for each participant to align each participant's ROIs optimally to this modal model. Each participant's transformation matrix was then applied to the structural and statistical data.
Note that one of us has presented a version of this technique previously (Stark and Squire, 2001a), but here, regions across the entire MTL are used and the objective function is significantly improved. Source code for the objective function will be freely available (cstark{at}jhu.edu).
Results
In test 1, participants were 73% correct overall [discriminability (d′), 1.3; hit rate, 0.64; false alarm rate, 0.18]. The reaction times for hit responses averaged 1222 msec. Correct rejections remembered in test 2 (CR-R) averaged 1332 msec, and those forgotten in test 2 (CR-F) averaged 1320 msec. False alarms averaged 1390 msec, and misses averaged 1340 msec. Reaction times for hits were reliably faster than all of the other trial types (all p values < 0.05), but no other pairwise differences were observed. In test 2, participants were 77% correct overall (d′, 1.6; hit rate, 0.77; false alarm rate, 0.22). Their hit rate for the items originally studied (list A) was 0.82 (d′, 1.69) and their hit rate for the items presented as foils during test 1 (list B) was 0.72 (d′, 1.35). Across the two types of hits in test 2, participants responded “sure old” on 54% of the trials, “probably old” on 23% of the trials, “probably new” on 17% of the trials, and “sure new” on 5% of the trials (39, 20, 23, and 18%, respectively, across all trials).
An ANOVA on the fMRI data from the study phase (Fig. 3, study) revealed six regions within the MTL whose activity varied by trial type (study-R and study-F). The modal model of all participants' ROIs from the ROI-AL technique was used to identify the location of each region of activity. Greater activity was observed for trials associated with successful encoding (study-R, red lines) than with poor encoding (study-F, blue line) bilaterally in the hippocampal region (CA fields, dentate gyrus, and subiculum), perirhinal cortex, and parahippocampal cortex. Therefore, these results replicate and extend the previous findings, demonstrating activity associated with intentional encoding success that can be localized with a high degree of precision to the hippocampal region, the perirhinal cortex, and the parahippocampal cortex.
An ANOVA on the fMRI from test 1 (Fig. 3, test 1) revealed six regions within the MTL whose activity varied by trial type of interest (hit, CR-R, and CR-F). Using the model from the ROI-AL technique, these six regions were identified as being within the hippocampal region bilaterally, bilateral perirhinal cortex, and bilateral parahippocampal cortex. All voxels within these regions were collapsed for a functional ROI analysis. The critical analysis was whether activity during test 1 was predictive of subsequent recognition memory performance in test 2. In all six regions whose activity varied by trial type, activity for the novel items in test 1 that were subsequently remembered (CR-R, red line) was significantly greater (pairwise t tests with 18 degrees of freedom, all t values >3.0 and all p values < 0.01) than activity for the novel items in test 1 that were subsequently forgotten (CR-F, blue line). Thus, there we observed clear evidence that activity in the MTL during what is nominally a memory-retrieval task was also associated with memory encoding.
A significant consequence of this encoding-related activity during retrieval is that it can obscure activity associated with memory retrieval when traditional retrieval success contrasts are used. To image activity associated with retrieval success, researchers typically contrast activity associated with hits (“yes” responses to studied items) with activity associated with correct rejections (“no” responses to novel foil items). Using this contrast, very little MTL activity was observed during test 1, with only left hippocampal activity observed (not shown). However, if the effect of incidental encoding during test 1 is ameliorated by contrasting activity associated with hits (Fig. 3, green line) with activity associated with correct rejections that were being poorly encoded at the time (CR-F), significant retrieval-related activity was observed in all six regions active in test 1 (pairwise t tests with 18 degrees of freedom, all t values >2.7 and all p values < 0.02). Therefore, significant retrieval-related activity was observed bilaterally in the hippocampal region, bilaterally in the perirhinal cortex, and bilaterally in the parahippocampal cortex. It is worth noting that activity associated with misses (“no” responses to study items), which would likely be associated with weak retrieval-related activity as well as potential incidental encoding-related activity was consistently similar to activity associated with hits and with CR-R trials (data not shown). Notably, the subregions of the MTL associated with intentional encoding during the study phase were also associated with incidental encoding during memory retrieval (bilateral hippocampal region, perirhinal cortex, and parahippocampal cortex).
Discussion
Using a technique designed to enhance the ability to localize fMRI results to specific, anatomically defined structures (ROIAL), we observed that activity that could be clearly localized to the perirhinal and parahippocampal cortices and to the hippocampal region was predictive of subsequent memory ability. Critically, this encoding-related activity was observed not only when participants were engaged in an intentional study task, but also when participants were actively performing a retrieval task (recognition memory). Activity for the novel correct rejections during the recognition memory task of test 1 was predictive of subsequent memory for these items in test 2. Thus, we observed incidental encoding-related activity during a recognition memory task (test 1). Notably, this activity was within the same subregions of the MTL as activity associated with encoding success during the intentional study task.
The idea that automatic encoding might exist is not a new one. Numerous theories of memory and of MTL function have predicted such an effect (Fletcher et al., 1997; Morris and Frey, 1997; Martin, 1999; O'Reilly and Rudy, 2001), and it has often been hypothesized to be a potential source for failures to observe activity related to retrieval success. The incidental encoding of items presented during a test has even been observed behaviorally on numerous occasions (Glover, 1989, for review; see Buckner et al., 2001). Despite this, to date there have been no direct observations of the effect in the MTL. At least two fMRI studies have observed the effect indirectly (Stark and Squire, 2000a,b). In these studies, a recognition memory test using objects as stimuli was administered twice in succession. Both reported activity in the hippocampal region related to retrieval success only during the second recognition test (when the foil items were relatively familiar). During the first test, no activity was observed using a target versus foil (or hit vs correct rejection) contrast. Furthermore, in both, greater activity was present for foils in the first presentation of the recognition test (when they were novel) than in the second presentation of the test (when they were more familiar). Thus, the results presented here are consistent with these two previous studies.
However, the results presented here are not entirely consistent with the one other study that has looked for such an effect directly. In this study, Buckner et al. (2001) examined test 1 activity based on test 2 performance and observed predictive effects in numerous regions. However, no such predictive effects were observed in the MTL. In the present study, more powerful techniques were used that both increased the quality of the data overall (e.g., more subjects, more data points per subject) and increased the ability to resolve effects within the MTL (e.g., ROI-AL alignment, oblique-axial imaging, choice of null task), which has presumably allowed us to measure the effects in greater detail than was possible previously.
Here, we have presented the most direct evidence that incidental encoding not only exists during retrieval, but that it can have a profound impact on the activity attributed to retrieval processes. We suggest, as have others, that memory encoding is a relatively ongoing process that may even be best described as “automatic” (Martin, 1999) and that it will occur even during memory-retrieval tasks. We have demonstrated that when left unaccounted for, this encoding-related activity can obscure activity related to retrieval success. Very little MTL activity was observed in the contrast between hits and correct rejections in test 1, consistent with numerous previous studies that have reported little or no activity related to retrieval success. However, when the encoding-related activity is at least partially accounted for by contrasting hits with correct rejections that are not being well encoded at that time (those that are forgotten by test 2), clear retrieval-related activity was observed throughout the MTL.
Notably, the subregions of the MTL associated with intentional encoding during the study phase were also associated with incidental encoding during memory retrieval (bilateral hippocampal region, perirhinal cortex, and parahippocampal cortex). Although these same subregions of the MTL were associated with encoding-related activity during both the study phase and test 1, the overlap of individual above-threshold voxels while present, was by no means absolute. Therefore, it is possible that the encoding-related processes in the two tasks were associated with activity in different portions of each subregion. To assess the degree of functional overlap, the functional ROIs defined from the study phase were used to assess encoding-related activity in test 1. Significant encoding-related activity in test 1 (CR-R vs CR-F) was observed in the left perirhinal cortex, left parahippocampal cortex, and right parahippocampal cortex ROIs. Masking in the other direction, significant encoding-related activity at study (study-R vs study-F) was observed in the right perirhinal cortex and right parahippocampal cortex ROIs in test 1. Therefore, although overlap could be found between encoding-related activity during intentional study and encoding-related activity during recognition, these data do not suggest that the processes are absolutely identical.
However, from these data we would argue that intentional encoding, incidental encoding, and memory retrieval (even in the form of simple recognition memory) all occur in numerous subregions of the MTL. Activity correlated with each of these processes was observed bilaterally in the hippocampal region, the perirhinal cortex, and the parahippocampal cortex, suggesting similarly heterogeneous roles for each of these structures in declarative memory.
Footnotes
This work was supported by Johns Hopkins University and National Science Foundation Grant BCS-0236431. We thank Lisa Caitlin Perri, Marcel Poisot, and the staff of the F.M. Kirby Center for Functional Brain Imaging for their assistance in data collection.
Correspondence should be addressed to Dr. Craig Stark, Department of Psychological and Brain Sciences, Johns Hopkins University, Ames Hall, 3400 North Charles Street, Baltimore, MD 21218. E-mail: cstark{at}jhu.edu.
Copyright © 2003 Society for Neuroscience 0270-6474/03/236748-06$15.00/0