Abstract
During everyday interactions, we continuously monitor and maintain information about different individuals and their changing emotions in memory. Yet to date, working memory (WM) studies have primarily focused on mechanisms for maintaining face identity, but not emotional expression, and studies investigating the neural basis of emotion have focused on transient activity, not delay related activity. The goal of this functional magnetic resonance imaging study was to investigate WM for two critical social cues: identity and emotion. Subjects performed a delayed match-to-sample task that required them to match either the emotional expression or the identity of a face after a 10 s delay. Neuroanatomically, our predictions focused on the orbitofrontal cortex (OFC) and the amygdala, as these regions have previously been implicated in emotional processing and long-term memory, and studies have demonstrated sustained OFC and medial temporal lobe activity during visual WM. Consistent with previous studies, transient activity during the sample period representing emotion and identity was found in the superior temporal sulcus and inferior occipital cortex, respectively. Sustained delay-period activity was evident in OFC, amygdala, and hippocampus, for both emotion and identity trials. These results suggest that, although initial processing of emotion and identity is accomplished in anatomically segregated temporal and occipital regions, sustained delay related memory for these two critical features is held by the OFC, amygdala and hippocampus. These regions share rich connections, and have been shown previously to be necessary for binding features together in long-term memory. Our results suggest a role for these regions in active maintenance as well.
Introduction
Everyday, we process and maintain information about the identity and changes in emotional expression of those with whom we interact. This study focuses on how this continuously changing social information is maintained in working memory (WM). Early WM studies explored how objects and locations, initially processed separately by dorsal and ventral visual streams, were maintained in WM. It was initially postulated that object “what” and location “where” data were maintained separately in prefrontal cortex (PFC) (Wilson et al., 1993), but a seminal study challenged this idea of anatomical segregation, suggesting instead that individual PFC neurons can hold both spatial and object information across a delay (Rao et al., 1997). In the present study, instead of examining two nonsocial cues (e.g., what and where), we used a delayed match to sample task to examine two salient social cues: Emotion and Identity. We contrasted two competing predictions. The first, based on evidence that emotion and identity are processed by different anatomical regions (Hasselmo et al., 1989), was that delay related activity would be anatomically segregated for emotion and identity. The second prediction, based on the demonstration that PFC neurons bind together information in WM, was that both emotion and identity are maintained by the same neuroanatomical system. A summary of neuroanatomical regions that we predicted would be critical for social WM follows.
Studies exploring face perception have demonstrated that attending to emotional expression activates superior temporal cortex, whereas attending to face identity activates inferior temporal and fusiform cortex (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Both areas have strong anatomical connections with the orbitofrontal cortex (OFC) (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992). A strong functional connection also exists between the amygdala and fusiform gyrus during face processing (Fairhall and Ishai, 2007).
The OFC is implicated in processing emotional information (Iidaka et al., 2001; Beer et al., 2006; Barbas, 2007; Eippert et al., 2007; Wright et al., 2008), and may maintain visual information in WM when task demands are high (Schon et al., 2008). Consistent with its role in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004), the OFC contains face-selective neurons (Thorpe et al., 1983; Rolls et al., 2006) and damage to the OFC can impair face identification (Hornak et al., 2003).
Facial expressions consistently activate the amygdala (Schneider et al., 1997; Morris et al., 1998), and the amygdala enhances long-term memory for emotional stimuli (Hamann et al., 1999; Kilpatrick and Cahill, 2003; Dolcos et al., 2004; Kensinger and Schacter, 2006). Face-selective neurons exist in the amygdala (Leonard et al., 1985), and patients with amygdala damage have impaired recognition of expressions (Adolphs et al., 2002; Adolphs and Tranel, 2003).
Strong anatomical connections exist between amygdala, OFC, and additional medial temporal lobe (MTL) structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007), suggesting that these regions play a critical role in WM for social cues.
Materials and Methods
Participants
Nineteen healthy individuals (six male; mean age, 23 ± 3) from the Boston community, with no history of neurological or psychiatric illness, were recruited for this study. Vision was either normal or corrected to normal. All subjects were screened to ensure magnetic resonance imaging (MRI) environment compatibility. Eligible individuals who agreed to participate gave signed informed consent in accordance with the Partners Human Research Committee and the Institutional Review Board of Boston University.
Procedure
Stimuli.
Stimuli for the task were selected from the University of Pennsylvania database of facial expressions (Gur et al., 2002). We selected three moderate positive, negative, and neutral valenced expressions from photographs of 24 individual actors of varying age, sex, and race, for a total set of 72 unique stimuli. All stimuli were cropped to 350 × 467 pixels at 28.35 pixels/cm resolution (12.35 × 16.47 cm), put on a gray background, and converted to gray-scale. An oval mask was used to isolate the central facial features (see Fig. 1). All peripheral features (e.g., hair, clothes, eye color) were removed. Control stimuli were Fourier phase scrambled versions of the original face stimuli.
DMS and CON task. In all tasks, a trial consisted of three time-locked components: a 2 s face presentation (sample), followed by a 10 s delay period, and by a 2 s face presentation (test). Trials were separated by a variable-length ITI. During EMO_DMS trials, subjects matched the emotional expression of the face, and facial identity was irrelevant. During ID_DMS trails, subjects matched the identity of the face and the emotional expression was irrelevant. In the case of an EMO-match trial (top), the test face showed the same emotional expression as the one shown by the sample face, but was from a different actor. In the case of an ID-match trial (middle), the test face was from the same actor as that seen during the sample phase, but showed a different emotional expression. During CON trials (bottom), Fourier phase scrambled versions of the face stimuli were presented and subjects were instructed to respond to the number presented in the center of the test stimulus.
Delayed match-to-sample task.
During MRI scanning, subjects performed a delayed match-to-sample task (DMS) consisting of eight runs. The DMS task required subjects to make match or nonmatch judgments by comparing a test stimulus to a sample stimulus separated by a short delay period (see Fig. 1). Subjects performed two types of the DMS task that differed only in instruction. The first type required the subject to match the emotional expression (EMO) of the face, and the second type required the subject to match the identity (ID) of the face. For the EMO_DMS task, the test face was always a different identity than the sample face, and for the ID_DMS task, the test face was always a different emotion than the sample face (Fig. 1). This prevented the subject from seeing an identical stimulus for the sample and test phases of a trial. Emotional expressions and identities were counterbalanced throughout the 8 functional MRI (fMRI) runs, and the two tasks were equated in terms of the emotional and identity information that was presented. The emotional expressions were characterized as positive, negative, and neutral for counterbalancing and analysis purposes only, however the subjects were not provided with these verbal labels. Behaviorally, subjects were simply told to match the emotion or the identity, and were not informed of the number of individuals or types of emotions presented. Labeling the emotion was not a factor in selecting a response. Each trial consisted of 2 s presentations of a sample face and a test face with an intervening 10 s delay period. The length of the delay period was kept fixed because previous work has shown that different delay lengths can result in different neural response patterns (Elliott and Dolan, 1999). A variable-length intertrial interval (ITI) (6–26 s, counterbalanced using Optseq2) (Dale, 1999) separated all trials. The variable ITI provided temporal “jitter” for selective averaging across trials (Postle et al., 2000a; Schon et al., 2004).
Visuomotor control task.
One-half of all trials consisted of a visuomotor control task (CON). During these trials, subjects were presented with scrambled versions of the face stimuli and responded to a “1” or “2” presented in the center of the test stimulus. The CON task acted as an effective baseline because it was designed to account for activity that was generalized to low-level visual and motor responses. These trials required no active maintenance, but maintained the subjects' attention to the stimuli by requiring an arbitrary response.
Functional runs either consisted of EMO trials or ID trials to minimize brain activation because of task switching. Before each functional scan, the subjects received verbal instruction that the upcoming run was an EMO or ID task. In addition, the color of the central fixation cross present during delays and ITIs reminded the subject of the task type. Subjects performed 12 trials in each of the eight functional runs (six DMS and six CON trials per run; four EMO and four ID runs, randomly ordered), for a total of 96 trials (24 EMO_DMS, 24 ID_DMS, 48 CON).
Before scanning, subjects viewed task instructions and performed a practice version of the task. Responses and reaction times (RTs) were recorded from an MRI compatible button box. We created four different randomized versions of the task so that no >5 subjects performed identical versions. Tasks were designed and presented and behavioral data were recorded using Psyscope 1.2.5 (Cohen et al., 1993).
fMRI data acquisition
Data were acquired on a 3.0 Tesla Siemens (Erlangen, Germany) Trio scanner at the Athinoula A. Martinos Center for Biomedical Imaging (Massachusetts General Hospital, Charlestown, MA). Two high-resolution T1-weighted anatomical images were acquired for each subject. Cognitive tasks were performed during 8 functional T2*-weighted gradient-echo, echo-planar blood oxygen level dependent (BOLD) scans [matrix, 64 × 64; field of view, 200 mm; repetition time (TR), 2 s; echo time, 30 ms; flip angle, 90°; 26 interleaved axial-oblique slices; voxel size, 3.125 × 3.125 × 5 mm; no interslice gap]. Slices were aligned parallel to the line connecting the anterior and posterior commisures and 144 images per slice were acquired during each scan.
fMRI data analysis
Preprocessing.
fMRI data were preprocessed using the SPM2 software package (Statistical Parametric Mapping, Wellcome Department of Cognitive Neurology, London, UK). BOLD images were corrected for differences in slice timing and realigned to the first image within a series. Realignment and unwarping was performed to correct for image distortions caused by susceptibility-by-movement interactions. The images were then normalized into MNI (Montreal Neurological Institute) stereotactic space, resampled to 2 × 2 × 2 mm isotropic voxels, and spatially smoothed using a 4 mm full-width half-maximum Gaussian filter.
fMRI statistical analysis.
Analysis of BOLD activity during the delayed match-to-sample task was assessed with multiple regression with near orthogonal regressors using the SPM2 software package. This approach has been successfully used to assess delay period activity in fMRI studies with delayed-match-to-sample tasks (Courtney et al., 1997, 1998; Postle and D'Esposito, 1999; Schon et al., 2004; Schon et al., 2005). As originally proposed by Courtney et al. (1997), and similar to Postle et al. (2000) and previous work in our laboratory (Schon et al., 2004; Schon et al., 2005), we created six regressors that reflected the comparisons of interest and convolved them with a Gamma hemodynamic response function (HRF) (Boynton et al., 1996) in Matlab 6.5 (MathWorks, Natick, MA). The regressors consisted of the following comparisons: regressor 1 assessed activation caused by nonspecific visual stimulation, regressor 2 assessed the difference between DMS stimuli and CON stimuli, regressor 3 assessed the difference between the sample stimulus and the test stimulus for CON trials, regressor 4 assessed the difference between the sample stimulus and the test stimulus for DMS trials, regressor 5 assessed differences in activity during task delays (across tasks) versus during the ITI, and regressor 6 assessed the difference between activity during the DMS delay versus activity during the CON delay (see Fig. 2).
Multiple regression analysis. A, Stick functions were used to create six contrasts for comparison of DMS and CON task components. S, Sample; D, delay; T, test. Regressors 2 and 6 were used to assess activity related to face processing and active maintenance, respectively. Bars indicate the positioning of the HRF. Stick functions for delay periods were divided into five bars spread across the five TRs of the delay to account for the sustained time course of the delay period activity (Schluppeck et al., 2006). B, Example of regressors created by convolving the six stick contrasts with a gamma-variate function (HRF) (Boynton et al., 1996). The bottom graph shows all six regressors together.
This procedure generated a statistical parametric T-map (SPM{T}) for each regressor and for each subject. Group analyses were performed on the regressors of interest (regressors 2 and 6) by entering the SPM{T} maps from each subject into second-level random-effects one-sample t tests, treating subjects as a random factor. For regressor 2, the significance level to identify suprathreshold voxels was p ≤ 0.001 with false discovery rate (FDR) correction (Genovese et al., 2002) across the whole brain. For regressor 6, the significance level was p ≤ 0.01, with FDR correction across the whole brain. The statistical threshold was more stringent for regressor 2 than for regressor 6 because more robust activity was expected during face presentations than during the intervening delay period. In both cases, the threshold extent was five contiguous voxels.
Our regressors were designed to isolate brain regions that contribute specifically to the processing of the sample and test stimuli (see Fig. 2, regressor 2) and the maintenance of the sample stimulus during the delay period (see Fig. 2, regressor 6) by contrasting DMS trials to CON trials. These regressors were constructed to account for the slow onset of the BOLD signal (HRF) relative to stimulus presentation. In addition, regressor 6 was separated into five ⅕ size stick functions spread across the five TRs of the delay period to account for the sustained time course and expected weaker signal during this phase of the task (Schluppeck et al., 2006) (see Fig. 2). We compensated for this expectation by using a more lenient significance threshold (described above) for analysis of delay period activity. This regression model allowed us to differentiate neural activity that was sustained across the delay component of the task from transient activity that occurred only during presentation of the face stimuli. However, because of residual collinearity between regressors 2 and 6 caused by overlap of the HRFs, the distinct contribution of a ROI to the stimulus and/or delay components of the task can be determined most accurately by analyzing the time course of activation within that region. Therefore, regions that were initially activated by the sample face presentation and sustained activity across the delay were revealed by both regressors. Further exploration of the contribution of brain regions to the DMS task was accomplished by analysis of the time course of activation within a given region.
In addition, a subsequent analysis was also performed using a 16-regressor model that allowed for all possible comparisons of task conditions and trial components. This analysis searched for regions showing differential EMO and ID task effects that may not have been captured by the six-regressor model. Data from this analysis are provided as supplemental text (available at www.jneurosci.org as supplemental material).
ROI analysis.
The regions of interest (ROIs) chosen for our time-series analysis were the orbitofrontal cortex, the amygdala, the hippocampus, parahippocampal cortex, inferior occipital/fusiform cortex, and superior temporal cortex. ROI peak activation coordinates were derived from second-level one-sample t tests on regressors 2 and 6. In all cases, time-series analysis was conducted on suprathreshold voxels within a 5 mm sphere surrounding the peak coordinate within each ROI (Table 1).
Significant areas of activation from the active maintenance analysis (regressor 6) and the face processing analysis (regressor 2; threshold at p ≤ 0.001FDR)
Time-series analysis.
The VOI (voxel-of-interest) tool in SPM2 was used to extract signal intensity time courses for each subject from suprathreshold voxels within each ROI. Signal intensities (Y) were adjusted for effects of interest and were extracted for each time point (144 per BOLD scan) and selectively averaged across subjects. Group level time course line charts were generated by plotting Y values across the 13 TRs (26 s) of each trial for the DMS and CON tasks (see Figs. 3⇓–5). Because the grand mean of the data were scaled to 100, the signal intensity values represent percentage signal change with respect to the global mean intensity of the scaled images. The tasks were designed such that the CON task serves as the baseline. Thus, significant positive activations are evident for events where DMS > CON. To contrast activity from DMS and CON trials, signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference (see Fig. 3⇓–5). Signal difference values were selectively averaged by task type (EMO and ID) and emotion condition (positive, negative, and neutral). To differentiate activity related to the individual components of the task, time points were averaged together to compute separate signal amplitudes that represent the sample (4th and 5th TRs), delay (6th–9th TRs), and test (10th and 11th TRs) phases of the DMS trial (see Figs. 3⇓–5). TR bins were selected to capture the peak of the HRF for the sample and test face responses and to account for an initial temporal lag of 6 s (Buckner, 1998). These values were statistically analyzed with t tests and a three-factor repeated-measures ANOVA with the within-subject factors task type (EMO vs ID), emotion condition (positive vs negative vs neutral), and task component (sample vs delay vs test). All statistics were corrected for unequal variance among factor levels whenever necessary.
Sustained delay period activity in orbitofrontal cortex. A, fMRI results from the active maintenance analysis (regressor 6) demonstrating sustained activity in left OFC, x = −46, y = 26, z = −6 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. B, Sustained activity is displayed in the line graph representing the signal intensity time course extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. C, Transient dissociation between EMO and ID tasks in OFC. Greater activity was found for EMO than ID during sample face presentations. In contrast, significant delay period activity was similar for EMO and ID tasks. The bar graph represents signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. D, Greater OFC activity at test for EMO_DMS trials with negative valence. During test phase face presentations within EMO trials, negatively valenced faces elicited a significantly greater signal than positively valenced faces. L, Left; R, right. Error bars indicate SEM. Asterisks indicate significant difference.
Sustained delay period activity in medial temporal lobe structures. A, Left amygdala, x = −18, y = −4, z = −20 (arrows). B, Left mid-hippocampal body, x = −32, y = −30, z = −8 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. Sustained activity is displayed in the line graphs representing signal intensity time courses extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. Bar graphs represent signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. Significant delay period activity was similar for EMO and ID tasks in amygdala and hippocampus ROIs. L, Left; R, right. Error bars indicate SEM.
fMRI results from face processing analysis (regressor 2) demonstrating transient activity that is modulated by task. A, Right inferior occipital cortex, x = 50, y = −74, z = −14 (arrows). B, Right superior temporal sulcus, x = 50, y = −36, z = 0 (arrows). C, Right posterior parahippocampal cortex, x = 16, y = −36, z = −6 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. Line graphs represent signal intensity time courses extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. Bar graphs represent signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. A, Greater activity for ID than EMO during sample face presentations in the right inferior occipital cortex. B, Greater activity for EMO than ID during sample face presentations in the right STS. C, Greater activity for EMO than ID during test face presentations in right parahippocampal cortex. L, Left; R, right. Error bars indicate SEM. Asterisks indicate significant difference.
Behavioral data analysis
Match/nonmatch responses and reaction times were recorded for each trial to ensure the subjects were performing the task as instructed. Accuracy and average RTs were analyzed with t tests.
Results
Behavioral Results
Subjects performed DMS tasks at close to 90% accuracy and showed no significant difference in accuracy between the EMO task (mean ± SE: 85.9 ± 1.7%) and ID task (89.6 ± 2.8%). Reaction times for the two tasks (EMO: 1210 ± 35 ms; ID: 1189 ± 34 ms) also did not differ significantly. Identity trials were excluded for two subjects because of poor task performance. The CON task was performed at close to 100% accuracy (mean ± SE: 99.4 ± 0.6%) and with significantly faster reaction times (mean ± SE: 822 ± 15 ms) than both DMS tasks (EMO vs CON, t(26) = 10.328, p < 0.001; ID vs CON, t(21) = 10.000, p < 0.001).
fMRI results
Sustained delay related activity in the orbitofrontal cortex, amygdala, and hippocampus
Active maintenance of information in working memory was assessed by contrasting DMS delay period activity with CON delay period activity (Fig. 2, regressor 6). This analysis revealed significant delay period activity in left OFC (peak at x = −46, y = 26, z = −6; T = 7.84, Z = 5.92, p < 0.01FDR) (Fig. 3). This cluster of activation extended dorsally and rostrally into inferior frontal gyrus (triangularis) and laterally into the insula. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI (Fig. 3C) revealed a significant main effect of task (EMO vs ID) (F(1,16) = 9.76, p = 0.007) and a significant main effect of component (sample vs delay vs test) (F(2,32) = 4.20, p = 0.032). t tests revealed significantly greater activation for EMO trials than ID trials (t(99) = 2.262, p = 0.026) during the sample phase in this ROI. T tests on the individual emotion conditions (positive vs negative vs neutral) in the OFC revealed significantly greater activation for negative faces than positive faces (t(36) = 2.613, p = 0.013) (Fig. 3D) during the test phase for EMO trials.
Significant delay period activity was also found in MTL structures including left amygdala (peak at x = −18, y = −4, z = −20; T = 3.63, Z = 3.32, p < 0.01FDR) (Fig. 4A) and left mid-hippocampal body (peak at x = −32, y = −30, z = −8; T = 5.23, Z = 4.46, p < 0.01FDR) (Fig. 4B). A repeated-measures ANOVA on the signal difference (DMS − CON) time-series in the left amygdala ROI revealed a significant main effect of component (sample vs delay vs test) (F(2,32) = 4.92, p = 0.016). A repeated-measures ANOVA and t tests did not reveal any significant differences between any of the conditions in the left hippocampal ROI. In both ROIs, activation appeared to be sustained early in the delay period (Fig. 4A,B, line graphs). Significant activation was also present in left insula, left inferior frontal gyrus (triangularis), the supplementary motor area (SMA), anterior parahippocampal gyrus, right posterior MTL (including the hippocampus and parahippocampal cortex), and left inferior parietal cortex.
In summary, robust activity in the OFC was sustained across the delay period. In OFC, during the sample phase, responses were larger for EMO trials than ID trials, and during the test phase of EMO trials, Negative faces elicited a greater response than Positive faces. Sustained activity was also found in the hippocampus and amygdala early in the delay period, but activity in these MTL areas was not modulated by task. See Table 1 for a complete list of areas of activation revealed by regressor 6.
Transient activity in temporal and occipital cortices and posterior parahippocampal cortex
Transient responses to the presentation of face stimuli were assessed by contrasting DMS sample and test phase activity with CON sample and test phase activity (Fig. 2, regressor 2). This analysis revealed significant transient responses to the sample and test face presentations in right inferior occipital cortex (peak at x = 50, y = −74, z = −14; T = 9.87, Z = 6.78, p < 0.001FDR) (Fig. 5A). The activation extended anteriorly into inferior temporal gyrus and medially into the fusiform gyrus. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI revealed a significant main effect of component (sample vs delay vs test) (F(2,32) = 47.72, p < 0.001) and a significant task (EMO vs ID) by component interaction (F(2,32) = 3.82, p = 0.042). t tests revealed significantly greater activation for ID trials than EMO trials (t(105) = 2.300, p = 0.023) (Fig. 5A) during the sample phase in this ROI.
This analysis also revealed significant transient responses during sample and test periods in right superior temporal sulcus (STS, peak at x = 50, y = −36, z = 0; T = 6.50, Z = 5.23, p < 0.001FDR) (Fig. 5B). This activation extended along the rostral/caudal axis of the STS. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI revealed a significant emotion (positive vs negative vs neutral) by component (sample vs delay vs test) interaction (F(4,64) = 4.59, p = 0.009). t tests revealed significantly greater activation for EMO trials than ID trials (t(98) = 1.987, p = 0.050) (Fig. 5B) during the sample phase in this ROI.
Transient responses were also found in right posterior parahippocampal cortex (peak at x = 16, y = −36, z = −6; T = 8.16, Z = 6.07, p < 0.001FDR) (Fig. 5C). A repeated measures ANOVA on the signal difference (DMS − CON) time-series did not reveal any significant effects; however, t tests did show significantly greater activation for EMO trials than ID trials (t(105) = 2.177, p = 0.032) (Fig. 5C) during the test phase in this ROI. Significant activation was also present in left inferior occipital cortex, left fusiform gyrus, left STS, bilaterally in the insula, bilaterally in orbitofrontal and inferior frontal cortex, left posterior parahippocampal cortex, bilaterally in the anterior MTL (including the amygdala), and in the SMA spanning ventrally along the medial wall into anterior cingulate cortex.
In summary, transient activity related to the sample and test phases of each trial that returned to baseline during the delay period was found in temporal and occipital cortices, including the right inferior occipital cortex and right STS, and in right posterior parahippocampal cortex. The most robust activity was found in the inferior occipital cortex. During the sample phase, responses in inferior occipital cortex were larger for ID trials than EMO trials, whereas responses in STS were larger for EMO trials than ID trials. During the test phase, responses in posterior parahippocampal cortex were larger for EMO trials than ID trials. See Table 1 for a complete list of areas of activation revealed by regressor 2.
Results from the 16-regressor model comparing EMO and ID tasks can be found in the supplemental text (available at www.jneurosci.org as supplemental material).
Discussion
Our goal was to explore WM for two critical social cues, emotion (EMO) and identity (ID). We examined two competing predictions. The first, based on the fact that emotion and identity are processed perceptually by separate systems, was that sustained delay related activity in EMO and ID trials would be anatomically segregated and actively maintained within ventral visual areas. The second, based on the demonstration that PFC neurons can actively maintain disparate stimulus types, such as what and where, was that emotion and identity would be maintained by a single neuroanatomical system. Our results support this second prediction. Neuroanatomically, our predictions and analysis focused on (1) a perceptual system within temporal and occipital cortices (TOC) that is known to separately process identity and emotion and (2) a neuroanatomically interconnected memory system comprised of the OFC, amygdala, and hippocampus, that has been implicated in long-term emotional memory and more recently in visual WM paradigms.
A discussion of these two neuroanatomical systems, one critical for active maintenance and the other critical for transient perceptual dissociation, follows.
Active maintenance of social cues
Our results demonstrate sustained delay related activity within the OFC, the amygdala, and hippocampus for both emotion and identity (Figs. 3, 4, Table 1). Neuroanatomically, strong connections exist between these three structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007). Additional connections from TOC to OFC provide a critical link between the regions we found that were transiently active, and those that demonstrated sustained activity (Barbas, 2007).
Our results extend previous studies of WM demonstrating contributions of prefrontal structures, including the OFC, to the active maintenance of faces (Courtney et al., 1996; Haxby et al., 2000; Sala et al., 2003), and differ from several studies demonstrating active maintenance in TOC during face working memory tasks (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006). However, it should be noted that fMRI studies examining WM maintenance in TOC do not consistently find delay related activity, and our results may help explain these inconsistencies. Critically, the present study differs from these earlier studies in that two features, identity and emotion, were varied across trials, rather than simply varying a single stimulus feature, identity. Therefore, it may be that the TOC is sufficient for the active maintenance of simple stimuli, including objects and faces (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006), but when multiple stimulus dimensions such as emotion and identity are held across a delay, these TOC regions may no longer be sufficient. Therefore, although one might have predicted sustained activity in the superior temporal sulcus for maintaining emotion and sustained activity in inferior occipital/fusiform cortex for maintaining identity, our data do not support this. Instead, our data suggest that these separate streams of information are relayed from the TOC to the OFC and to the amygdala (Barbas, 2007). These structures, together with the hippocampus, act to bind the two features together and hold this unified information in WM. A similar system for maintaining conjunctions of object and spatial information in WM has been proposed in the lateral PFC in the monkey (Rao et al., 1997; Rainer et al., 1998) and human (Owen et al., 1998; Postle et al., 2000b).
Our data are consistent with the role of the OFC in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004, 2007), which, in turn, is consistent with its prominent role in rapid stimulus-reinforcement learning (Rolls, 2000, 2004; O'Doherty, 2004; Rolls et al., 2006; Dolan, 2007). A recent review suggests the OFC may be critical for promoting flexible behaviors and guiding response selection, and in addition, is critical for reflecting changes in reinforcement values, and in modifying behavioral responses when predicted outcomes (such as reinforcement) change (for review, see Murray and Izquierdo, 2007). In our task, the subjects need to flexibly switch between selecting their responses based on emotion and identity. This might be accomplished by inhibiting one stimulus dimension and maintaining the other. Or, as we suggest, it might be accomplished by binding together and maintaining both stimulus dimensions, emotion and identity, across the delay, and then flexibly selecting the correct response.
We predicted that our task would engage the amygdala. Face responsive neurons have been identified in the amygdala (Leonard et al., 1985), and imaging studies have demonstrated amygdala activation in response to expressions (Schneider et al., 1997; Morris et al., 1998; Blair et al., 1999; Iidaka et al., 2001; Killgore and Yurgelun-Todd, 2004). Memory studies have implicated the amygdala in the enhancement of long-term encoding of emotionally charged stimuli (Cahill et al., 1996; Canli et al., 2000; Kensinger and Schacter, 2006), and the amygdala exerts an influence on activity in prefrontal cortex and MTL structures during long-term memory encoding (Hamann et al., 1999; Kilpatrick and Cahill, 2003). In addition, amygdala activity has been demonstrated during implicit processing of emotional stimuli (Critchley et al., 2000; Phelps and LeDoux, 2005) or when these stimuli are task-irrelevant distracters (Siegle et al., 2002; Dolcos and McCarthy, 2006; Siegle et al., 2007), which is consistent with the fact that this limbic region showed sustained activity in our task regardless of trial type. In addition to its known role in long-term emotional memory and consolidation (McGaugh, 2004), the sustained activity noted in the present study suggests an additional role for the amygdala in the active maintenance of emotional and socially relevant material in WM.
In addition to sustained activity in the OFC and amygdala, we also found sustained activity within the hippocampus. Previous fMRI data has demonstrated a role for MTL structures, including the hippocampus and surrounding parahippocampal cortex, in the short term maintenance of novel information (Ranganath and D'Esposito, 2001; Stern et al., 2001; Ranganath et al., 2003, 2005; Schon et al., 2004; Schon et al., 2005). Based on the rich connections between the hippocampus, amygdala, and OFC (Barbas, 2000; Insausti and Munoz, 2001; Munoz and Insausti, 2005), we suggest that together these areas form a circuit that binds together and actively maintains emotion and identity information in WM.
Transient responses to face presentations in temporal and occipital regions
Our results demonstrated transient activity during the sample period in the inferior occipital cortex, extending into the fusiform gyrus, that was significantly greater during ID trials than during EMO trials, and transient activity in the superior temporal sulcus that was significantly greater during EMO trials than ID trials (Fig. 5A,B). This result is consistent with data showing that inferior occipitotemporal regions are involved in perception of facial identity (Kanwisher et al., 1997; Haxby et al., 2001), whereas superior temporal regions are involved in perception of emotional cues and may be important for social perception in general (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Neuroanatomically, the superior temporal sulcus and inferior temporal cortex have strong connections with the OFC (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992), and an effective connectivity study has demonstrated a strong functional connection between amygdala and fusiform gyrus during processing of faces (Fairhall and Ishai, 2007).
In addition to regions within the TOC, posterior parahippocampal cortex only responded transiently to face presentations (Fig. 5C). This result is consistent with a study that found sustained activity in the hippocampus, but transient activity in parahippocampal and anterior fusiform gyri, during delayed matching of faces (Ranganath and D'Esposito, 2001).
Linking the two systems
The active maintenance of social cues requires the cooperation between systems critical for perceptually identifying and processing emotion and identity, and systems critical for actively maintaining this information in memory. In studies where only one dimension, such as identity, is varied, it may be possible for posterior TOC to actively maintain this information in the same system where the information is initially perceptually processed. But in more natural social situations, such as when both identity and emotional expressions can vary, then information from these perceptual TOC regions may be relayed to a memory system that is capable of flexibly binding and maintaining this information together. We suggest this system is comprised of the orbitofrontal cortex, the amygdala, and the hippocampus. We conclude that this system, which is already known to play a critical role in binding together emotional information in long-term memory (McGaugh, 2004), is also critical for actively maintaining, and binding together, salient social cues in working memory.
Footnotes
-
This work was supported by National Institutes of Health Grant P50 MH071702-02, National Center for Research Resources Grant P41RR14075, National Science Foundation Science of Learning Center Grant SBE-0354378, and by the Program in Neuroscience and the Department of Psychology at Boston University. We thank Courtney Horwitz and Matthew Grace for assistance with stimulus preparation and data analysis, and Michael Hasselmo for discussions about this study. We also thank Dr. Ruben Gur's Brain Behavior Lab at the University of Pennsylvania for supplying the face stimuli used in this study. Research was conducted at the Cognitive Neuroimaging Lab, Center for Memory and Brain, Boston University (Boston, MA), and the Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School (Charlestown, MA).
- Correspondence should be addressed to Matthew LoPresti, Boston University, Center for Memory and Brain, 2 Cummington Street, Boston, MA 02215. mattlo{at}bu.edu
