Abstract
Incoming events that match or mismatch stored representations are thought to influence the ability of the hippocampus to switch between memory encoding and retrieval modes. Electrophysiological work has dissociated match and mismatch signals in the monkey perirhinal cortex, where match signals were selective for matches to goal states, whereas mismatch signals were not modulated by intention (Miller and Desimone, 1994). To investigate whether the theoretically important relational match and mismatch signals in the hippocampus are modulated by goal states, we fully crossed whether a probe stimulus relationally matched or mismatched a previously perceived image or goal state. Subjects performed two working memory tasks in which they either responded “yes” to probes that were identical to the previous sample scene or, after performing a relational manipulation of the scene, responded “yes” only to a probe that was identical to this perceptually novel image. Using functional magnetic resonance imaging, we found evidence for relational match enhancements bilaterally in the hippocampus that were selective for matches between the probe stimulus and goal state, but were not modulated by whether that goal was perceptually novel. Moreover, we found evidence for a complementary hippocampal mismatch enhancement that was triggered by stimuli containing salient perceptual manipulations. Our results provided evidence for parallel memory signatures in the hippocampus: a controlled match signal that can detect matches to internally generated goal states and an automatic mismatch signal that can identify unpredicted perceptual novelty.
Introduction
The formation and retrieval of relational memories critically depends on the hippocampus, as demonstrated by numerous lesion, electrophysiological, and human imaging studies (for review, see Squire et al., 2004; Davachi, 2006; Eichenbaum et al., 2007; Mayes et al., 2007). Nonetheless, how it performs these operations remains poorly understood. Strictly in terms of neuronal firing rate, previously encoded events can be distinguished from novel ones by increased activity for events that match stored representations (i.e., match enhancement), or by greater activity for novel events (i.e., mismatch enhancement). Electrophysiological studies provide evidence for both of these signals within the hippocampus (Freid et al., 1997; Wood et al., 1999; Rutishauser et al., 2006), although neither the distinction between these seemingly redundant signals nor the nature of the representations against which these comparisons are being made are known.
An intriguing clue comes from previous dissociations of memory signatures for item information in the monkey perirhinal cortex (Miller and Desimone, 1994). In a standard sequential delayed match-to-sample task, which can be performed by detecting any repetition, mismatch enhancement predominated in the perirhinal cortex (see Fig. 1B legend for a more detailed explanation of how we use the terms “match enhancements” and “mismatch enhancements”). However, in an altered paradigm in which behaviorally irrelevant repetitions intervened between sample and target, requiring active maintenance of a goal, distinct neuronal populations expressing match and mismatch enhancements were found. Interestingly, match neurons signaled the repetition of the goal image, whereas mismatch neurons were not modulated by behavioral relevance.
Recent functional magnetic resonance imaging (fMRI) studies have reported relational mismatch enhancement in the human hippocampus (Kumaran and Maguire, 2006, 2007a). Specifically, the hippocampus displayed the greatest response during sequences of objects that relationally differed from the expectations developed by the previous sequence. A complementary match enhancement may not have been found, however, because participants in these studies performed one-back tasks that were unrelated to the experimental manipulation. If match enhancements convey behaviorally relevant matches, then these signals would only be uncovered in experiments designed to investigate goal relevant matches.
Consistent with this hypothesis, Hannula and Ranganath (2008) found enhanced hippocampal activation for images that matched a maintained goal. In this task, participants mentally rotated arrangements of objects and identified probes that matched this perceptually novel, generated goal. Although finding match enhancements in response to perceptually novel stimuli suggests that internally generated representations may be used in goal-match calculations, it is still possible that a combination of perceptual-mismatch and goal-match signals are driving the enhancement seen. In other words, goal-matches were present in conditions that always contained perceptual-mismatches.
We developed a paradigm that fully crosses whether stimuli match or mismatch a goal state or a previous perception to help elucidate the mechanisms underlying match and mismatch enhancements in the human hippocampus. We predict that hippocampal match enhancements will reflect goal-matches rather than perceptual-matches, whereas mismatch enhancements may be influenced by perceptual novelty.
Materials and Methods
Subjects.
Eighteen (six females) right-handed native English speakers with normal or corrected-to-normal vision participated in the study. Their mean age was 21 with a range of 18–27. Consent was obtained in a manner approved by the Institutional Review Board at New York University.
Design.
Participants performed two working memory tasks (object-match and object-switch). Both began with participants viewing a sample image (a room containing a pair of objects), maintaining information over a variable delay, and then making a memory decision on a probe image. In the object-match task, participants were instructed to respond “yes” only to a probe that was identical to the sample stimulus, whereas in the object-switch task, participants were instructed to swap the locations of the objects and respond “yes” only to this perceptually novel goal.
Across the two tasks, six experimental conditions were developed (Fig. 1). In the object-match task, if the probe contained the same objects as the sample image in their original locations, the probe would match both the goal and previous perceptual experience (BOTH), and if it contained the same objects in the reversed locations, it would match neither the goal state nor the perceptual experience (NEITHER). In the object-switch task, if the probe was identical to the sample, then it matched the previous perception while mismatching the goal (PERCEPT), and if it contained the same objects in the reverse locations, then the probe matched the goal but mismatched the percept (GOAL). In both tasks, a probe stimulus could contain a new object (OBJ) or an old object in a new location (LOC) with respect to the sample image. Subjects should respond negatively to these probe conditions. All subjects were presented with an equal number of trials of each condition.
Experimental conditions and predicted patterns of activation. A, A schematic illustration of the two working memory tasks. The first three columns illustrate the events of a trial and the four possible probe images. Both tasks begin with the presentation of a sample followed by a delay period, and finally participants make their memory decision during the probe presentation. In the object-match task, subjects are instructed to respond “yes” to probes that contain the same objects in their original locations, whereas in the object-switch task, probes that contain the same objects in the reversed locations receive a “yes” response. The last two columns label the trial types contained in the object-match and object-switch tasks along with their required responses. See Materials and Methods, Behavioral procedures, for a more detailed explanation of the trial types. B, Predictions for distinctive patterns of activation that would be consistent with four types of memory signals. Goal-match enhancement refers to the pattern of activity that would be consistent with a region that is sensitive to goal-matches. Greater activity would be found in response to trials that match the participants' goal state (BOTH and GOAL) compared with trials that mismatch the goal (NEITHER and PERCEPT). The perceptual-match enhancement pattern would be consistent with a region sensitive to perceptual-matches. Greater activity would be elicited by probes that are identical to the sample stimulus (BOTH and PERCEPT) compared with perceptually novel probes (NEITHER and GOAL). The goal-mismatch enhancement would be consistent with a region sensitive to goal-mismatches with stimuli that mismatch the goal state (NEITHER and PERCEPT) eliciting greater activity than stimuli that match it (BOTH and GOAL). The perceptual-mismatch pattern would be consistent with a region that is responsive to perceptual-mismatches with perceptually novel stimuli (NEITHER and GOAL) eliciting greater activity than stimuli that are identical to the sample (BOTH and PERCEPT). It should be noted that these signals are defined by the difference in response across conditions. Although we adopt a terminology that is consistent with an underlying enhancement mechanism, each pattern could also be explained with a suppression mechanism. For example, the goal-match enhancement pattern could be also called a goal-mismatch suppression.
Stimuli.
All stimuli were generated from a set of 138 two-color rooms (12 reserved for training) and 299 color drawings of objects (26 reserved for training). The room stimuli were rendered in Adobe Illustrator with a common spatial layout, but unique in the combination of colors and patterns of stripes or checkerboards used on the floor and walls. Two hundred sixty of the object images were from a colorized Snodgrass image database (Rossion and Pourtois, 2004); the remainder (26 training and 13 test) were selected from web-based clip art collections with the criteria that they did not have the same names as any objects in the original set and had a similar style of rendering. Each scene stimulus was composed of one room with two overlaid objects, each positioned in one of nine locations defined by an invisible 3 × 3 grid on the floor of the room. All scenes were generated during the experiment or training sessions using the psychophysics toolbox (version PTB-3) (Brainard, 1997) for Matlab (MathWorks).
Twenty-one arrangements of objects were used in the study. They were selected to ensure that no object was obscured by the other and that each location was filled a similar number of times. For every subject, each of the 21 arrangements was used in each of the six conditions. In the new location condition, one of the objects was moved to an adjacent grid location in either the x or y directions with approximately equal frequency. The objects were broken into 13 groups of 21 objects (plus the 26 reserved for the training trials). Over the subjects, each group of objects had approximately an equal likelihood of occurring in each of the six conditions. Each object was used as the new object in the new object condition and as old objects in all conditions across subjects.
Behavioral procedures.
Subjects performed the object-match and object-switch working memory tasks while in the scanner. Both tasks began with the presentation of a sample stimulus (3 s duration) with an auditory task cue played during the last 0.5 s of the presentation. A variable delay period (7, 9, 11 s) followed, and each trial ended with a probe stimulus presentation for 2 s. Subjects were able to make their response on a button box only during this 2 s period. A variable intertrial interval (8, 10, 12 s), during which subjects viewed a blank screen, separated trials.
In the object-match task (cued with the sound of a chime), subjects were instructed to make a “yes” response only when the probe stimulus contained the same objects in the same locations as the sample and a “no” response in all other cases. In the object-switch task (cued with the sound of a buzzer), subjects were instructed to make a “yes” response when the probe contained the same objects as the sample stimulus but in the reversed locations (i.e., object 1 was in the location formally occupied by object 2 and vice versa). There were four possible probe scenes: the same objects in the original locations (BOTH in the object-match task and PERCEPT in the object-switch task); the same objects in the reversed locations (NEITHER in the object-match task and GOAL in the object-switch task); the same objects with one moved to a new location (LOC); and the same locations with one new object (OBJ).
Subjects were trained on the two tasks before entering the scanner. They first received written instructions outlining how to perform the object-match and object-switch tasks. They were instructed to respond “yes” by pressing a key with their index finger and to respond “no” by pressing their middle finger key. They were warned that some probes would contain new objects or old objects in new locations and that they would need to attend to both the object identity and location to perform the task. Next, they performed a training session to ensure they understood the requirements of each condition and were able to make their response in the allotted time.
The experimental task was presented over three separate runs. Each run had 42 trials, seven of each condition. The order of trial types was determined by the “Optseq2” algorithm (http://surfer.nmr.mgh.harvard.edu/optseq/) to provide the best sequence for subsequent estimation of condition-specific effects.
fMRI methods.
All scanning was performed using a 3T Siemens Allegra MRI system with a whole-head coil. Functional data were collected using a gradient-echo planar pulse (EPI) sequence (repetition time, 2000 ms; echo time, 30 ms; field of view, 192; 36 slices aligned perpendicular to the hippocampal long axis; 3 × 3 × 3 mm voxel size; three runs; 505 volumes per run). A T1-weighted, high-resolution sequence (magnetization-prepared rapid-acquisition gradient echo; 176 sagittal slices, 1 × 1 × 1 mm voxels) was run to collect anatomical data after the completion of the final experimental run. Visual stimuli were projected onto a screen that was viewed through a mirror attached to the subject's head coil. Auditory stimuli were presented through a Siemens headphone system. Before the first experimental run, the subjects adjusted the volume of the auditory cues over the noise of the same EPI sequence used for the collection of functional data. During the same EPI sequence, they then performed a brief auditory discrimination task to insure that they could easily tell the cues apart over the noise of the scanner.
The imaging data were analyzed using SPM2 (Wellcome Department of Cognitive Neuroscience, University College London, London, UK). During preprocessing, the functional data were corrected for differences in slice acquisition timing and then motion was corrected across runs. Subjects who moved >3 mm in any direction would have been removed from analysis, but all subjects' movement estimates were below this criterion. The functional data were then spatially normalized to an EPI template in Montreal Neurological Institute (MNI) space. The transformation obtained from this normalization was used to normalize the anatomical data. Finally, the voxels were spatially smoothed with a 6 mm full-width at half-maximum isotropic Gaussian kernel.
fMRI statistical analysis.
Functional data were analyzed using a general linear model implemented in SPM2. An impulse response at the onset of the sample stimulus and probe presentation was used to model the activity during these events. A box car of appropriate length (7, 9, or 11 s) was used to model the intervening delay period activity. The two tasks, object- match and object-switch, were modeled separately during the delay period, the six possible probe conditions were modeled separately, and all sample presentations were modeled together. Additionally, the sample, delay, and probe periods of error trials were modeled separately from those of successful trials. Because performance was very high in both tasks, error trials were not included in any subsequent analysis. The three runs were concatenated and modeled as one to increase the number of trials used to estimate each parameter. Each regressor was convolved with both a canonical hemodynamic response function and its temporal derivative, resulting in two regressors per event. The mean signal and linear drift were separately modeled for each run as confounds. Subject-specific fixed-effects models were used to estimate effects of task and probe conditions. These subject-specific contrasts were entered into a random-effects analysis (one-sample t test) for group analysis. To investigate effects in the medial temporal lobe (MTL), an adjusted reliability threshold of p < 0.005 (five contiguous voxels) was adopted to accommodate the relatively low signal-to-noise ratio within the MTL (Ojemann et al., 1997; Schacter and Wagner, 1999; Davachi and Wagner, 2002; Strange et al., 2002; Dobbins et al., 2003; Weis et al., 2004; Mitchell et al., 2004; O'Kane et al., 2005). Regions outside of the MTL were considered reliable at a p < 0.001 level (five contiguous voxels). All subsequent comparisons were preformed on subject-specific β estimates for each condition across all voxels contained in clusters identified by the random-effects analysis.
Conjunction analyses.
The experiment was designed so that different types of memory signatures can be distinguished by the patterns of activation triggered by all of the probe conditions (see Fig. 1B for predicted patterns of activation). A region with activity that is consistent with a match enhancement should exhibit greater activation during BOTH compared with NEITHER probe stimuli. Importantly, however, goal-match and percept-match enhancements can be distinguished by the response to probe types in the object-switch task. Specifically, if there is more activation in the GOAL condition, the match enhancement is likely driven by a match between the probe and the goal; if there is more activation in the PERCEPT, the signal is more likely driven by a match to the previous percept. This same logic holds for mismatch enhancements defined by greater activation during the NEITHER than the BOTH condition. Greater activation during the PERCEPT than GOAL trials would be consistent with a goal-mismatch enhancement, whereas a percept-mismatch enhancement would be inferred when activity is greater in the GOAL than the PERCEPT condition. Hence, consideration of activation across all four experimental conditions will yield a greater understanding of what representations are driving either response.
It should be noted that we are not able to conclusively determine whether an observed pattern is being driven by an enhancement or suppression of neural firing. For example, the pattern that we refer to as a goal-match enhancement (Fig. 1B) could be driven by an increased response to probes that match a goal state or, rather, a suppressed response to probes that mismatch a goal. Thus, we could have referred to this pattern as a goal-mismatch suppression. We have chosen to adopt terms here that are consistent with enhancement, and they will be used throughout this report to refer to the four patterns in Figure 1B.
Regions showing the predicted pattern of activation across the four relational conditions (BOTH, NEITHER, GOAL, and PERCEPT) were identified by looking at the conjunctions of within-task contrasts of probe regressors. We entered the results of targeted group random-effects contrasts into each conjunction analysis to identify voxels that show a combination of effects. Regions showing a goal-match enhancement were identified by the conjunction of the BOTH > NEITHER and GOAL > PERCEPT contrasts. Regions showing a goal-mismatch enhancement were identified by the conjunction of the contrasts of NEITHER > BOTH and PERCEPT > GOAL. Regions showing a perceptual-match enhancement with the conjunction of BOTH > NEITHER and PERCEPT > GOAL and perceptual-mismatch enhancements were identified with the conjunction of NEITHER > BOTH and GOAL > PERCEPT contrasts. Similarly, second-level analyses were used to investigate identified regions of interest using a 2 × 2 within-subjects ANOVA (goal: match vs mismatch; percept: match vs mismatch) so that a main effect for goal where match > mismatch would indicate that the selected region shows goal-match enhancement. Critically, paired t tests were used to verify that the expected conditions did indeed differ from each other.
A conjunction analysis was also performed to identify regions that were preferentially responsive to conditions with both types of salient perceptual novelty (OBJ and LOC). This novelty-conjunction analysis consisted of the conjunction of LOC > (BOTH and NEITHER and GOAL and PERCEPT) and OBJ > (BOTH and NEITHER and GOAL and PERCEPT).
Regions in the MTL were considered reliable in a conjunction analysis at a combined p < 0.005 level, and regions outside the MTL were considered reliable at a combined p < 0.001 level.
Results
Behavioral results
See Table 1 for the mean accuracy and reaction times for correct trials in each condition. Paired sample t tests revealed that responses in the object-match task were faster than those in the object-switch task (t(17) = 4.0; p < 0.001) and that responses to probes that matched the goal (“yes” responses) were faster than those that mismatched this representation (“no” responses; t(17) = 5.7; p < 0.001). Hence, greater activation for goal-matches, compared with mismatches, is not simply driven by time on task. The accuracy measure did not show these effects. Instead, trials in which the probe stimulus was a perceptual-match to the sample (BOTH and PERCEPT trials) had greater accuracy than conditions in which the probe contained reversed objects (NEITHER and GOAL trials), t(17) = 3.3; p < 0.005. Critically, we only used correct trials in all functional imaging analyses.
Behavioral performance
fMRI results
The global pattern of activation elicited by the task was first assessed with contrasts of the sample period > baseline (supplemental Fig. 1A, available at www.jneurosci.org as supplemental material) and the probe period (collapsed across conditions) > baseline (supplemental Fig. 1C, available at www.jneurosci.org as supplemental material). Both of these periods were associated with robust activity throughout the ventral visual pathway, MTL regions including bilateral hippocampus and perirhinal and parahippocampal cortices, as well as the frontal cortex, basil ganglia, thalamus, and parietal cortex.
The crucial comparisons in this design, however, are based on the patterns of activation seen across conditions during the probe period. To determine this, four conjunction analyses were completed to reveal regions that exhibited activation patterns consistent with match and mismatch enhancements selective for goal or perceptual information. Critically, we predicted that if regions in the hippocampus express match enhancements driven by matches to goal states and not perceptual novelty, then activity would be greater for BOTH compared with NEITHER trials in the object-match task and for GOAL compared with PERCEPT trials in the object-switch task. This should be the case even though the probe on GOAL trials has never been seen before.
Hippocampal match enhancements
Of the four match/mismatch conjunction analyses, only the goal-match pattern (BOTH > NEITHER and GOAL > PERCEPT) yielded voxels within the MTL. This was true even when the combined threshold was reduced to a very liberal threshold of p < 0.01. In particular, bilateral posterior hippocampal regions (extending into parahippocampal cortex on the right) were found to exhibit a goal-match pattern of response. Critically, when the condition-specific β estimates were examined in these regions, the benefit for trials that matched the goal state over those that mismatched was comparable across both encoding tasks (Fig. 2). Because the goals in each task differed in perceptual novelty, this comparable benefit implies that the match calculation is not affected by whether the goal was perceived or had to be internally generated. It should be noted, however, that BOTH and GOAL activations are not significantly greater than the activations for OBJ or LOC trials (all p > 0.1; with the exception of GOAL being greater than OBJ in the right hippocampal/parahippocampal region; t(17) = 2.4; p < 0.05). BOTH and GOAL conditions would be expected to have the greatest activation in voxels that are solely devoted to goal-match calculations. Instead, in the right hippocampal/parahippocampal region, LOC activations are significantly greater than NEITHER activations (t(34) = 2.34; p < 0.05). OBJ and LOC responses were not significantly different from any other conditions (all p > 0.1). The heightened response for conditions with salient perceptual novelty suggests that a weak mismatch enhancement may be occurring alongside the match calculations made in these regions, but, as noted above, the match calculation itself does not seem to be modulated by the perceptual novelty of the goal image.
Regions in bilateral posterior hippocampi that were revealed in the goal-match conjunction analysis (BOTH > NEITHER and GOAL > PERCEPT). Parameter maps (t-maps) are overlaid on coronal slices of a representative subject's T1-weighted image. Accompanying bar graphs display the mean β estimates across subjects for each of the probe types averaged across the voxels contained in the highlighted cluster. *p < 0.05. A, Left (L.) posterior hippocampal region (MNI coordinates: x = −30, y = −30, z = −11). B, Right (R.) posterior hippocampal region that extends into parahippocampal cortex (MNI coordinates: x = 29, y = −35, z = −8. t-stat, t-statistic.
Moreover, when voxels within the left posterior hippocampus were isolated with a sample period > baseline contrast, these voxels also displayed a significant goal-match pattern (F(1,34) = 4.4; p < 0.05). Voxels in this same area isolated with a probe period (collapsed across stimulus types) > baseline contrast showed a similar, but marginally significant, goal-match trend (F(1,34) = 3.0; p = 0.09). Finally, voxels in the right anterior hippocampus that were activated by the probe (compared with baseline) also displayed this same goal-match pattern of activation (F(1,34) = 5.2; p < 0.05). The finding of goal-match enhancements in hippocampal voxels that were selected only by their activation during the task reinforces the interpretation that goal-match enhancements were the predominant mnemonic signals elicited in the hippocampus by this paradigm.
A small region in the left posterior hippocampus (MNI coordinates: −27, −24, −9) was revealed by a GOAL > PERCEPT contrast, but it did not show a benefit for goal-matches (BOTH > NEITHER) in the object-match task (t < 1). If the benefit for GOAL trials reflected a general preference for perceptual-mismatches or goal-matches (as is seen in the two posterior hippocampal regions), then some selectivity for either NEITHER or BOTH trials, respectively, would be expected. The pattern in this region, however, is more consistent with an area that has a preference for a combination of goal-matches and perceptual-mismatches, consistent with associative match-mismatch detection (Kumaran and Maguire, 2007b). Interestingly, although it is found in the left instead of right hemisphere, this region is similarly located along the anteroposterior axis to a region (MNI coordinates: 27, −27, −6) found by Kumaran and Maguire (2007a).
Hippocampal mismatch enhancements
Although no regions within the MTL were found to show mismatch enhancements in response to goal-mismatches or subtle perceptual-mismatches, regions in the MTL were found to show preferential activity for the more salient perceptually novel OBJ and LOC conditions. The novelty-conjunction analysis revealed regions in the posterior right hippocampus (Fig. 3), left parahippocampal cortex, right perirhinal cortex, and right amygdala. Interestingly, the hippocampal region does display a pattern consistent with a goal-match enhancement, but this was not found to be significant (F(1,34) = 2.4; p > 0.1). Moreover, within each task, BOTH activation was not significantly greater than NEITHER, and GOAL activation was not significantly greater than PERCEPT (both p > 0.1). Additionally, a large region in the anterior hippocampus was also found to have preferential activity for the novel OBJ condition, as revealed by an OBJ > (BOTH and NEITHER and GOAL and PERCEPT) contrast (Fig. 3). In this region, the responses to BOTH, NEITHER, GOAL, and PERCEPT conditions were also not significantly different from each other (all p > 0.1).
Regions in the right (R.) hippocampus that displayed activation patterns consistent with a mismatch enhancement. Parameter maps (t-maps) are overlaid on coronal slices of a representative subject's T1-weighted image. Accompanying bar graphs display the mean β estimates across subjects for each of the probe types averaged across the voxels contained in the highlighted cluster. *p < 0.05. A, A right posterior hippocampal region (MNI coordinates: x = 23, y = −35, z = −2) that was revealed in the novelty conjunction analysis ([LOC > BOTH and NEITHER and GOAL and PERCEPT] and [OBJ > BOTH and NEITHER and GOAL and PERCEPT]). This pattern of activity is consistent with a region that responds to salient perceptual novelty, whether it be new objects or new locations. B, A right anterior hippocampal region (extending into entorhinal cortex; MNI coordinates: x = 20, y = −8, z = −24) that was revealed in a contrast of OBJ > BOTH and NEITHER and GOAL and PERCEPT. t-stat, t-statistic.
Conjunction analysis results
Delay period activity
A region in the posterior left hippocampus (MNI coordinates: −32, −31, −8) showed sustained delay period activity regardless of trial type as determined by an object-match delay and object-switch delay > baseline contrast. t tests confirmed that both the object-match and object-switch delay periods were significantly greater than baseline (both p < 0.05) and not different from each other (t < 1). It is worth noting that this region overlaps with the left hippocampal goal-match region. When the β estimates for the probe were extracted from this delay region, a goal-match pattern was also observed (F(1,34) = 10.2; p < 0.005) (Fig. 4). See supplemental Figure 1B (available at www.jneurosci.org as supplemental material) for cortical regions revealed by this contrast.
A region in the left (L.) posterior hippocampus revealed by a delay period (collapsed across object-match and object-switch trials) > baseline contrast. This region (MNI coordinates: x = −32, y = −31, z = −8) overlaps with the left hippocampal region revealed by the goal-match conjunction analysis and also displays a goal-match pattern of activity during the probe. A, Parameter map (t-map) of this contrast overlaid on coronal slices of a representative subject's T1-weighted image. t-stat, t-statistic. B, β Estimates from the cluster highlighted in A for each stimulus type during the probe period. The voxels in this region also show a significant goal-match effect (F(1,34) = 10.2; p < 0.005).
Additional regions in MTL and prefrontal cortex were revealed in a contrast of object-switch delay > object-match delay. A region in the right posterior hippocampus (MNI coordinates: 29, −20, −11) and left amygdala (−29, −10, −21) both displayed this pattern, along with medial prefrontal (−3, 14, 49) and anterior cingulate cortex (3, 24, 33). Bilateral dorsolateral prefrontal cortex (MNI coordinates: 36, 39, 39; −42, 48, 15) was also revealed in this contrast, consistent with its proposed role in the manipulation of information during working memory tasks (Owen et al., 1996; D'esposito et al., 1999; Postle et al., 1999; Ranganath, 2006; Blumenfeld and Ranganath, 2006).
Perirhinal signals
Although there is some evidence for match enhancement in the perirhinal cortex (Miller and Desimone, 1994), activation patterns in the human perirhinal cortex are more commonly consistent with mismatch enhancement (Henson et al., 2003; Brozinsky et al., 2005; Gonsalves et al., 2005; Köhler et al., 2005; Danckert et al., 2007). For example, a recent study that compared hippocampal and perirhinal responses to stimuli over multiple repetitions demonstrated that although the hippocampus can express enhanced responses to repeated stimuli, the perirhinal and parahippocampal cortices show their greatest response to the first presentation of a stimulus (Preston and Gabrieli, 2008).
In the current study, however, we were able to find evidence for both match and mismatch enhancement in the perirhinal cortex. A region in the right perirhinal cortex, extending laterally into the fusiform gyrus (MNI coordinates: 38, −10, −35), was revealed in the goal-match conjunction analysis. Another region in the right perirhinal cortex (MNI coordinates: 36, −21, −27) was identified in the novelty conjunction that reveals regions that are sensitive to salient perceptual manipulations. When the different probe responses were examined in perirhinal voxels that were significantly activated by the cue (greater than baseline at a p < 0.005 level), the activation to the OBJ condition, which elicited the strongest response, was significantly greater than the activation related to the PERCEPT condition, which elicited the weakest response (t(17) = 2.9, p < 0.05 in the left hemisphere and t(17) = 4.0, p < 0.001 in the right hemisphere). No other condition comparisons were significant in these voxels. Left perirhinal voxels activated by the probe (all probe conditions greater than baseline at a p < 0.005 level) had a similar pattern with the OBJ response being significantly greater than the PERCEPT response (t(17) = 2.2; p < 0.05). Together, these results provide evidence for a goal-match enhancement in the human perirhinal cortex but also suggest that, unlike the hippocampus, the global pattern of activation in the perirhinal cortex is biased toward responses to novel objects and is suppressed for repeated presentations of images, consistent with the repetition suppression literature.
Discussion
The calculation of matches/mismatches between the environment and expectation has been proposed to mediate the switching between encoding and retrieval modes in the hippocampus (Hasselmo and Schnell, 1994; Hasselmo et al., 1995; Lorincz and Buzsaki, 2000; Lisman and Otmakhova, 2001; Vinogradova, 2001; Meeter et al., 2004; Lisman and Grace, 2005; Kumaran and Maguire, 2007b). According to these accounts, encoding and retrieval biases must be established in the hippocampus because uncontrolled co-occurrence of these processes would result in the overwriting of memories and the formation of excessively overlapping traces.
Although these theoretically important match and mismatch signals have been reported before, to our knowledge, this is the first experiment, using either functional imaging or electrophysiological techniques, to modulate intentional and perceptual states to systematically examine relational match and mismatch enhancements in the hippocampus. To accomplish this, the present paradigm decoupled whether a probe stimulus matched the actively maintained goal or the previous perceptual experience. Thus, regions could be identified that exhibit match or mismatch enhancements selective for each type of representation. Based on previous electrophysiological research (Miller and Desimone, 1994; Suzuki et al., 1997), we predicted that goal states would drive match enhancements, whereas perceptual experience would influence mismatch enhancements.
Goal-match enhancements were found in bilateral posterior hippocampi, extending into right parahippocampal cortex. Critically, the benefit seen for stimuli matching the goal state was similar across tasks even though these stimuli differed in their perceptual novelty, suggesting that these match calculations were not influenced by whether the goal image was perceived or had to be internally generated. The finding that match enhancements are goal dependent provides a framework for synthesizing apparently discordant findings of match versus mismatch enhancements in the hippocampus (Kumaran and Maguire, 2006, 2007a; Hannula and Ranganath, 2008); the analyses in the studies by Kumaran and Maguire (2006, 2007a) were not sensitive to goal-match enhancements because the participants' goal (i.e., one-back task) was orthogonal to the experimental manipulations. Moreover, our finding that match enhancements can operate on internally generated goals also supports the interpretation of Hannula and Ranganath (2008) that it was the match-to-goal states that drove the enhanced hippocampal response to perceptually novel target trials.
The reversal of memory signals seen in two recent high-resolution imaging studies (Kirwan and Stark, 2007; Bakker et al., 2008) further supports the selectivity of match enhancements for goal information. In both experiments, participants were presented with a series of images that contained exact repetitions, manipulated repetitions, and novel images, but the task performed by the subjects differed. When instructed to perform a continuous recognition task in which repetitions had to be distinguished from similar lures, match enhancements predominated in the hippocampus (Kirwan and Stark, 2007). In the second study, participants passively viewed the stimuli (Bakker et al., 2008). Under these instructions, a mismatch enhancement was seen throughout the subfields of the hippocampus. Although not discussed within our proposed framework, these previous results are consistent with our findings that, whereas both match and mismatch enhancements co-occur within the hippocampus, match enhancements are responses to goal-matches.
Evidence for the more commonly observed mismatch enhancements (Ranganath and Rainer, 2003; Köhler et al., 2005; Kumaran and Maguire, 2006, 2007a) was also obtained in the current study. Whereas a perceptually driven mismatch enhancement was not revealed in the conjunctive contrast of NEITHER > BOTH and GOAL > PERCEPT, a hippocampal mismatch enhancement was seen in conditions in which a novel, unexpected object occurred and/or when an old object occurred in a new location. It is likely that these activations reflect the detection and subsequent encoding of these unpredictable and salient events. It should be noted that a hippocampal region showing a mismatch response also displayed a pattern consistent with a goal-match enhancement, although this did not reach significance. This raises the important question of the spatial independence of these signals and suggests that neurons expressing these signals may be intermixed.
A region similar in location to the hippocampal novel object area described above was found in a recent study that modulated attention to novelty (Dudukovic and Wagner, 2007). In that experiment, participants were presented with two previously studied words and one novel word and were asked to either perform a recency judgment or select the novel word. A contrast of novelty task > recency task revealed an anterior hippocampal region. Because the stimuli were identical in both tasks, this contrast does not reveal regions that have preferential firing for novel items, but instead for attention to novelty. This signal may also have been related to the successful encoding of novel stimuli because a follow-up behavioral study indicated that the novel words in the novelty task were better remembered during a subsequent recognition task than novel items in the recency task.
There remain other possible causes of the match and mismatch enhancements seen here. The trial structure of this experiment resulted in fewer probes containing novel objects and locations than other probe types. On any given trial, participants had a one-in-three chance of seeing a probe that was identical to the sample (BOTH and PERCEPT trials), a one-in-three chance of seeing a probe with objects in reversed locations (NEITHER and GOAL trials), a one-in-six chance of seeing a novel object, and a one-in-six chance of seeing a novel location. It is possible, therefore, that the mismatch response was not driven solely by the saliency of the perceptual manipulation but also by the breaking of expectations, similar to the interpretation given by Kumaran and Maguire (2006). Furthermore, to maintain an equal number of trials in each condition and maximize the power in this design, it was necessary to have fewer “yes” responses than “no” responses. It is possible that part of the observed match enhancement was driven by a mismatch in response expectation. However, there are several reasons why we think this is unlikely. In a preliminary behavioral pilot experiment, participants generally reported that they were not aware of any differences between the frequency of making “yes” and “no” responses, although processes occurring outside of conscious awareness could drive a response mismatch enhancement. If this were the case, we might expect to see some effect on response times, which could result in the less frequent “yes” responses being slower than “no” responses, but, in fact, the opposite pattern was found. Of course, the possibility remains that the effect of response bias on response times was overshadowed by the relative ease of making “yes” responses. As a more direct test, we conducted a post hoc analysis by separately extracting time courses from the hippocampal match regions during the first and third runs. It was reasoned that, if the benefit for trials that match the goal state were driven by the novelty of the “yes” response, then this benefit would be larger during the final run when response probabilities were well established. We found that the difference between the response to trials that matched the goal state and those that mismatched it was actually stronger during the first run than the third (see supplemental Fig. 2, available at www.jneurosci.org as supplemental material), again providing evidence that it is the match between the active goal state and the probe that is driving the effect.
The possibility that active maintenance of goals contributes to match calculations is supported by sustained delay period activation seen in the hippocampus during both tasks. Interestingly, the region that shows a delay period response also displayed a goal-match activation pattern during the probe presentation. Additionally, this region overlapped with the left hippocampal area revealed in the goal-match analysis. This overlap suggests that the mechanism underlying goal-match enhancement may be a reactivation, through perceptual input, of the already excited delay active neurons. Together, these results suggest that active maintenance of a goal is supported, at least in part, by sustained hippocampal neuronal firing and that, at probe presentation, a comparison is made between the maintained goal and the perceptual probe. Our data show that this comparison is reflected in two distinct memory signatures. First, when the probe matches the internally maintained goal, there is an enhancement in hippocampal activation, perhaps resulting from an enhancement in the firing of the very same neurons used to maintain the goal throughout the delay. Second, when a probe contains a novel element, enhanced activation may reflect attention to or encoding of novel elements.
The dissociation between match and mismatch signals by their sensitivity to top-down intentional information and bottom-up perceptual information intuitively fits the role of a memory system. Although it would be inefficient to continually monitor matches between expectations and the environment, calculating matches to goals that are being actively maintained is a necessary step toward achieving those goals. Similarly, significant novel events need to be identified by an automatic memory decision based on perceptual information to allow for quick orienting toward unpredicted events.
Footnotes
-
This work was supported by National Institute of Mental Health Grant MH074692.
- Correspondence should be addressed to Dr. Lila Davachi, Department of Psychology, New York University, 6 Washington Place, Room 866B, New York, NY 10003. lila.davachi{at}nyu.edu
