Abstract
The extent to which neural representations of fear experience depend on or generalize across the situational context has remained unclear. We systematically manipulated variation within and across three distinct fear-evocative situations including fear of heights, spiders, and social threats. Participants (n = 21; 10 females and 11 males) viewed ∼20 s clips depicting spiders, heights, or social encounters and rated fear after each video. Searchlight multivoxel pattern analysis was used to identify whether and which brain regions carry information that predicts fear experience and the degree to which the fear-predictive neural codes in these areas depend on or generalize across the situations. The overwhelming majority of brain regions carrying information about fear did so in a situation-dependent manner. These findings suggest that local neural representations of fear experience are unlikely to involve a singular pattern but rather a collection of multiple heterogeneous brain states.
Significance Statement
Much of the debate on the nature of emotion concerns the uniformity or heterogeneity of representation for particular emotion categories. Here we provide evidence that widely distributed activation patterns characteristic of recent neural signatures of fear reflect an amalgam of functionally heterogeneous brain states. Participants completed a novel fMRI task that parametrically examined subjective fear within and across three content-rich and naturalistic situations: fear of heights, spiders, and social threats. Using searchlight analysis and machine learning methods, we show that the overwhelming majority of brain regions that predict fear only do so for certain situations. These findings carry implications for the generalization of findings on fear across species, translational models of fear and anxiety, and developing neural signatures of fear.
Introduction
For over a century, philosophers, psychologists, and neuroscientists have debated the nature of emotions (Dalgleish et al., 2009; Gendron and Feldman Barrett, 2009; Barrett and Satpute, 2019). Much of this debate concerns the uniformity or heterogeneity of representation for particular emotion categories (Lindquist et al., 2013; Mobbs et al., 2019). For example, is there a particular brain state that underlies fearful experiences (Vytal and Hamann, 2010; Celeghin et al., 2017; Nummenmaa and Saarimäki, 2019), or does fear involve a collection of heterogeneous brain states (Wilson-Mendenhall et al., 2011; Satpute and Lindquist, 2019; Doyle et al., 2022)? Addressing this question has broad implications for translational neuroscience models of mood and anxiety disorders (LeDoux and Pine, 2016; Fanselow and Pennington, 2017).
Recent functional magnetic resonance imaging (fMRI) studies have searched for the “brain signatures” of emotion categories (Kassam et al., 2013; Kragel and LaBar, 2015, 2016; Saarimäki et al., 2016, 2018). Brain signatures (sometimes called “neuromarkers” or “neural signatures”) are a type of multivoxel pattern analysis (MVPA) that uses brain data to predict behavior (Kragel et al., 2018). Brain signatures of emotion draw on information that is widely distributed throughout cortical and subcortical areas. However, whether this information is organized into a single prototypical pattern, or reflects an amalgam of heterogeneous functional states, remains unclear because neural signatures do not inherently provide direct insight about neural representation (Kragel et al., 2018; Lindquist et al., 2022). Classification is possible even if none of the individual brain regions carry category-level representations of emotion (for details, see Clark-Polner et al., 2017; Kragel et al., 2018; Azari et al., 2020; Lindquist et al., 2022).
Some findings suggest there is functional heterogeneity in the neural representation of fear. Univariate fMRI studies have found that distinct sets of brain regions are engaged depending on the fear-evocative content (e.g., pictures of spiders, blood, social encounters; Caseras et al., 2010; Lueken et al., 2011; Michalowski et al., 2017). However, two limitations to this work preclude a clear conclusion. First, these studies either compared categories of stimuli with distinct semantic content (e.g., stereotypically fearful stimuli, e.g., spiders or snakes, vs “neutral” stimuli, e.g., ordinary objects), participant groups (phobic vs nonphobic; Caseras et al., 2010), or both (Lueken et al., 2011; Michalowski et al., 2017). Accordingly, observed differences may be due to the semantic content of the stimuli or between individuals that are unrelated to fear. Second, these studies focused on activation magnitude, yet patterns of activation may carry information about psychological states irrespective of differences in activation magnitude.
Here, we tested the extent to which functional activity throughout the brain predicts fear in a situation-general or situation-dependent manner. Participants viewed ∼20 s clips depicting spiders, heights, or social encounters and rated fear after each video. We selected these situations because they span a wide variety of properties. For instance, while fear is often studied in a predator–prey context, fear of heights is potent and yet does not involve a predator. Critically, video stimuli were also curated to evoke a wide range of fear within each situation (Fig. 1). This design enabled us to systematically examine the neural predictors of fear within and across each situation. We used searchlight MVPA (Kriegeskorte et al., 2006) with least absolute shrinkage and selection operator (LASSO)-PCR (Tibshirani, 1996; Wager et al., 2011; Chang et al., 2015) to identify brain regions with functional information that predicts fear ratings. A brain region may carry fear-predictive information that generalizes across all situations using the same “neural codes” (i.e., shared model parameters) or using different neural codes (i.e., unshared model parameters). Thus, we trained our models in two distinct ways to test these possibilities. Our findings suggest that, regardless of the training approach, functional activity that predicts fear is widely distributed throughout the brain and largely dependent on the fear-evocative context.
Within-category variation in fear ratings. Each box and whiskers plot shows the mean and variation (quartiles and max/min values) in fear ratings (0, “low fear”; 1, “high fear”) across participants by video rank. Videos (x-axis) are sorted from lowest to highest on subjective fear ratings by participant, regardless of the video identity, since different videos evoked different levels of fear experience across participants. The plot illustrates that the videos were effective at evoking a wide range of fear experiences within each situation and for each person. Thus, MVPA can be used to examine which brain regions predict fear within and across situations. Extended Data: see Extended Data Figure 1-1 for fear ratings by the order of video presentation.
Figure 1-1
Self-reported fear ratings did not show sensitization or habituation effects over time for each situation. Participants watched a total of thirty-six videos across three functional runs in the fMRI scanner, such that each run consists of four videos from each of the three situations. The box and whisker plots show mean and distribution of fear ratings across videos, rank ordered across time. Download Figure 1-1, TIF file.
Materials and Methods
Participants
Neurotypical participants who reported no clinical psychiatric diagnosis were recruited from the Greater Los Angeles area. Exclusion criteria consisted of claustrophobia, psychiatric medication, left-handedness, metal in the body, and age (under 18 years or over 55 years). After excluding three individuals with excessive motion (criteria described below), the sample included 21 participants (11 male; 10 female; ages 22–40 years; mean age, 30.4).
Stimuli
Thirty-six silent videos were used in the experiment (12 videos per situation; duration, 18–22 s/video). While silent videos might be less evocative, they provide a more conservative test of the situation-dependent hypothesis since it has already been shown that neural responses during affective experiences are modality dependent. For ecological validity, all videos depicted naturalistic footage and were shot from an immersive first-person perspective. Videos were selected to be relatively stable (i.e., did not involve dramatic changes or “jump scares”) to mitigate motion artifacts and maintain the consistency of psychological experience across the duration. Video stimuli were obtained and normed in an independent online sample (Extended Data Stimulus Norms). Stimuli were curated to elicit a wide range of variation in self-reported fear across three distinct situations such that models could be estimated to predict fear within each situation (Fig. 1). In the heights condition, for example, a normatively high-fear video depicts first-person footage of walking along the edge of a sheer cliff, whereas a normatively low-fear video depicts first-person footage of walking downstairs. While norms were used to select stimuli for inclusion, analyses were conducted using subjective reports. A short description of the content and the normative ratings from the independent online sample for each video are available online: https://github.com/yiyuwang/AffVids_mvpa/tree/main/video_info.
Experimental task
Video stimuli were presented across three functional runs (12 videos/run) in the MRI scanner. Each run included an equal number of videos from each situation category, with an equal number of high- and low-fear videos (based on median normative ratings) within each category. The order of video stimulus presentations was pseudorandomized to ensure uniformity of stimulus types over time (Extended Data Fig. 1-1). Videos of a given category were preceded by videos of the same and different categories equally often, and videos with a given normative fear rating were preceded by videos of higher- and lower-normative-fear ratings equally often. Participants were instructed and reminded in between scans “to immerse yourself in the situation shown” and also to “respond according to how you, in particular, feel in response to viewing the videos.” After each video, participants consecutively rated experienced fear, arousal, and valence, on a sliding scale, ranging from “low” to “high” for fear and arousal and from “negative” to “positive” for valence. Participants used a trackball to move a cursor along a continuous scale and then clicked a button under their right thumb to log their rating. Four seconds were allotted to make each rating (12 s total). The task included an anticipatory period before each video, wherein the word (“heights,” “social,” or “spider”) corresponding to the category of the upcoming video was presented for 3 s, followed by a jittered fixation interval of 3–5 s, during which participants rated their expected fear on a sliding scale anchored by “low” to “high.” The purpose of this period was to mitigate effects pertaining to semantic updating that would otherwise occur when transitioning from a fixation cross to a rich visual image and to address other research questions regarding anticipatory activity prior to video watching. This period was not analyzed to address the present hypotheses. Trials were presented across three 9 min runs. Participants failed to provide fear ratings within the allotted time on a small proportion of trials. Missing fear ratings were interpolated (Extended Data Interpolation) and included in analyses. Stimuli were presented using MATLAB (MathWorks) and the Psychophysics Toolbox, and behavioral responses were recorded using a scanner-compatible trackball.
fMRI data acquisition and preprocessing
MRI data were collected using a 3 T Siemens Trio MRI scanner. Functional images were acquired in interleaved order using a T2*-weighted multiband echo planar imaging (EPI) pulse sequence (transverse slices; TR, 1,000 ms; TE, 3,000 ms; flip angle, 60°; FOV, 200 mm; 2.5 mm; thickness slices; voxel dimension, 2.5 × 2.5 × 2.5 mm; phase encoding direction anterior to posterior (AP); multiband acceleration factor, 4). Functional scans included coverage of the amygdala and orbitofrontal cortex (Extended Data Figs. 3-4 and 3-5). Anatomical images were acquired at the start of the session with a T1-weighted pulse sequence (TR, 2,400 ms; TE, 2,600 ms; flip angle, 8°; FOV, 256 mm; 1-mm-thickness slices; voxel dimension, 1 × 1 × 1 mm).
Image volumes were preprocessed using fMRIprep (Esteban et al., 2019). Preprocessing included motion correction, slice-timing correction, removal of high frequency drifts using a temporal high-pass filter (discrete cosine transform, 100 s cutoff), and spatial smoothing (6 mm FWHM). For analysis, functional volumes were downsampled to a 3 mm space to speed up searchlight analyses and registered to participants’ anatomical image and then to a standard template (MNI152) using FSL FLIRT (Jenkinson et al., 2002). Participants with at least two runs without excessive head motion (defined as >2 mm maximum framewise displacement) were included in the analysis yielding 18 participants with three runs of data and 3 participants with two runs of data.
General linear model
A general linear model (GLM) was used to model the neural data. The GLM included a separate boxcar regressor for each video stimulus, convolved with a canonical hemodynamic response function from SPM12. Nuisance regressors included six regressors corresponding with motion parameters, three regressors for physiological noise artifacts (CSF, white matter, framewise displacement), and nonsteady states outliers (stick function per outlier). Three regressors were included to model low-level visual properties of the stimuli. Specifically, luminance, contrast, and the complexity of each extracted frame were calculated using MATLAB scripts (https://github.com/yiyuwang/AffVids_mvpa/tree/main/calculate_visual_property). Pixel values were extracted from each frame occurring at the beginning of each TR. Luminance was calculated as the mean value of the grayscale image of the frame. Contrast was calculated as the difference between the maximum luminance and the minimum luminance. Complexity was calculated as the entropy of the grayscale image of the frame. The GLM was conducted using custom scripts in the Python nilearn module. Beta maps for each video and participant were used for training the searchlight LASSO-PCR (see below).
Searchlight LASSO-PCR
For the searchlight multivariate pattern analysis, betas from the GLM were extracted from voxels within the voxel’s searchlight neighborhood using a 15 mm (five voxel) radius. Because voxel data is nonindependent, we first run a principal component analysis (PCA) with the same number of components as the number voxels in the searchlight. The PCA transforms nonindependent activity across voxels as a set of orthogonal components. The components are then used as regressors in a LASSO regression (Tibshirani, 1996; Chang et al., 2015) to predict continuous fear ratings. A relatively lenient penalty term of 0.05 was used since the searchlight analysis already constrains the dimensionality of the fMRI data. The analysis was performed using modified functions from the scikit-learn and nilearn Python module (Pedregosa et al., 2011). All code is publicly available at https://github.com/yiyuwang/AffVids_mvpa.
Cross-validation
Prior studies that examined the neural predictors of fear trained and tested their model across groups of individuals (Kragel and LaBar, 2015; Zhou et al., 2021). Here, we follow suit with this approach by combining data across participants and training and testing our models using threefold, leave-whole-subject-out, cross-validation for statistical robustness (Poldrack et al., 2020). Participants were randomly divided into three groups (folds) of seven participants each (we selected three since it evenly divides the 21 participants in the sample). Models (i.e., searchlight with LASSO-PCR) were iteratively trained on two groups and tested on the left-out group. The dot product of the model weights from LASSO-PCR from the training data, and activation data from the testing sample yields predicted fear ratings. Pearson’s correlations between the predicted fear ratings and the actual fear ratings were calculated and assigned to the center voxel of the spherical searchlight. After iterating the searchlight across the whole brain, the analysis resulted in a whole brain map of the correlation values between the predicted ratings and the observed ratings of the testing sample for each of the threefold. We averaged the maps from the threefold as our final result.
Model training and testing
In the across-situation training method, models were trained on data across all three stimulus categories. In each fold, the model was trained on data corresponding with all 36 videos from 14 participants (i.e., 504 samples) and was then tested on data for each stimulus category from the 7 left-out participants. In the situation-by-situation training method, models were trained on data from one stimulus category at a time. For example, the model was trained using data corresponding with 12 heights video stimuli from 14 participants (i.e., 168 samples) and then tested on data for heights video stimuli from the 7 left-out participants. The across-situation training method has the advantage of more training samples, which yields more robust results. Thus, we performed analyses with more balanced training sets, too, and found that this training advantage for the situation-general model is unlikely to impact our conclusions (Extended Data Fig. 3-3).
Permutation testing and statistical correction
Permutation testing (N = 1,200/voxelwise neighborhood) was used to identify voxels with nonzero predictions. Models were trained and tested using shuffled data to generate a null distribution of correlation values. A familywise error (FWE) rate of 0.05 was used to threshold the permutation test (Nichols and Holmes, 2002; Nichols and Hayasaka, 2003).
Results
Behavioral findings confirmed the central aim of the task design, namely, that fear ratings varied from low to high levels within each content condition and within participants (Fig. 1). Thus, we proceeded to examine whether and which brain regions contained functional activity that predicts fear ratings within and across situations. There are three possible outcomes. First, population activity of neurons in a brain region may code for fear in the same way across situations. If so, then functional activity for a given brain region may predict fear ratings across situations using the same “neural codes” (model parameters). Second, population activity may code for fear in different ways for different situations, for example, if a brain region contains segregated neural pathways or the same pathway functionally organizes into distinct configurations. If so, then functional activity for a given brain region may predict fear ratings across situations, but the neural codes depend on the situation. Finally, a third possibility is that population activity may code for fear in one or two situations, but not all three, suggesting that both the brain regions and neural codes that predict fear are situation dependent. These hypothetical possibilities dictate two different model training and testing approaches, as described below.
Across-situation model training: shared neural codes
If a brain region carries situation-general neural codes, then model training should include instances across situations to enable the model to best learn which signals are, indeed, situation general. However, if a brain region contains situation-dependent neural codes, then even upon training the model with data across situations, it may only predict fear in one or two situations. To investigate these possibilities, we first trained the searchlight MVPA using data across all three situations and tested how well the model predicted fear for every situation using held-out data. As in prior studies, fear-predictive functional activity was widely distributed throughout cortical and subcortical areas (Fig. 2). A breakdown of the situation-general map showed that 1.9% of voxelwise neighborhoods met criteria of having model parameters that predicted fear across situations. Of the remaining voxelwise neighborhoods, 48.2% predicted fear in one situation, whereas 49.9% predicted fear in two of three situations. The reported findings use FWE-corrected significance tests; the proportions of voxels classified as situation general and situation dependent did not substantially change when using a more lenient threshold (Extended Data Fig. 2-1). Voxelwise neighborhoods that predicted fear across situations were located in the right posterior insula and the right superior temporal cortex (Fig. 2, red; for a list of ROIs, see Extended Data Table 2-1). Yet, the overwhelming majority of brain regions that predicted fear did so in only one or two situations.
Across-situation model training: shared neural codes. Models were trained using data across all three situations. A, The pie chart illustrates the percentage of voxels that predicted fear in held-out data in one (dark blue), two (light blue), or all three (red) situations. Only 1.9% of voxelwise neighborhoods predicted fear across all three situations. B, The maps illustrate which brain regions carried situation-general and situation-dependent information in predicting fear. Situation-dependent areas (both the dark blue and the light blue) are distributed across the whole brain. The situation-general with shared neural codes areas (red) included the superior temporal cortex, posterior insula areas, and the somatosensory cortex. C, A detailed breakdown of the neural pattern by each situation and their combinations. Color codes signify the specific situation or combination of situations predicted by the voxelwise neighborhood. Note, even with across-situation training, some areas only predicted fear for spider-, social-, or heights-related stimuli. Extended Data: see Extended Data Table 2-1 for the list of situation-general ROIs in red. See Extended Data Table 2-2 for MVPA studies of fear signatures; Extended Data Figure 2-1 for results using a more lenient threshold; and Extended Data Figure 2-2 for Pearson’s correlation values in the identified brain regions.
Table 2-1
Situation general ROIs from the Across-situation training. ROIs and its center of mass for the situation general areas in the Across-situation training (red areas in Figure 2). Download Table 2-1, DOCX file.
Table 2-2
MVPA studies of fear signatures comparing methods, stimulus modality, and ROIs. Download Table 2-2, DOCX file.
Figure 2-1
Results from the Across-Situation training using uncorrected p < 0.05: The proportion of voxels classified as situation general and situation dependent or their general locations did not significantly change when using a more lenient threshold. Download Figure 2-1, TIF file.
Figure 2-2
Across-situation Pearson Correlation Values. The observed, cross-validated r values ranged from 0.18 (the FWE corrected threshold based on permutation testing) to 0.46. Download Figure 2-2, TIF file.
Situation-by-situation model training: unshared neural codes
If a brain region carries situation-dependent neural codes, then model training should occur situation by situation. We trained and tested the searchlight with LASSO-PCR models situation by situation (i.e., trained and tested using data from the heights condition only and the same for spider and social conditions). This approach resulted in a Pearson’s correlation map per situation. We performed a conjunction analysis (Nichols et al., 2005) to identify which voxelwise neighborhoods predict fear across situations. The conjunction map may reveal areas with functional activity that predicts fear across all three situations but only when using unshared neural codes. This is a more lenient approach for identifying brain regions that predict fear across situations since the model parameters are allowed to vary by situation. It may also reveal areas that predict fear in one or two situations, but not all three. A breakdown of the conjunction map showed that only 4% of fear-predictive voxelwise neighborhoods carried information in all three situations (Fig. 3, in brown instead of red shading to distinguish these areas that predict fear across situations but with unshared neural codes from those in Fig. 2; for a list of ROIs, see Extended Data Table 3-1). Of the remaining voxelwise neighborhoods, 66.4% predicted fear in only one situation, and 29.5% predicted fear in two of the three situations. We present our findings using FWE-corrected significance tests; however, we performed analyses across a range of lenient and stringent statistical thresholds to ensure that conclusions were robust across thresholding. The proportions of voxelwise neighborhoods classified as situation general or situation dependent did not meaningfully change when using a more lenient threshold (Extended Data Fig. 3-1).
Situation-by-situation model training: unshared neural codes. Models were trained and tested on data from only a single situation, resulting in three maps. A conjunction analysis was used to identify which brain regions carried information that predicted fear across situations. This is a more lenient approach than across-situation training since model parameters are allowed to vary, situation-by-situation, in predicting fear. A, The pie chart illustrates the percentage of voxels that predicted fear in held-out data in one (dark blue), two (light blue), or all three (brown) situations. Four percent of voxelwise neighborhoods predicted fear across all three situations. B, The maps illustrate which brain regions carry situation-general and situation-dependent information in predicting fear. The situation-general areas but with unshared neural codes areas (brown) included the superior temporal cortex and posterior insula areas, as before, but note these areas may involve model parameters that still vary by situation. C, A detailed breakdown of the neural pattern by each situation and their combinations. Color codes signify the specific situation or combination of situations predicted by the voxelwise neighborhood. Extended Data: see Extended Data Table 3-1 for the list of situation-general ROIs (in brown); Extended Data Figure 3-1 for results using a more lenient threshold; Extended Data Figure 3-2 for Pearson’s correlation values in the identified brain areas; Extended Data Figure 3-3 for results comparing different training samples; Extended Data Figures 3-4 and 3-5 for demonstration of signals in amygdala and orbitofrontal cortex; and Extended Data Figure 3-6 for stimulus constant analysis to investigate the effect of visual properties.
Table 3-1
Situation general ROIs from Situation-by-situation training. ROIs and its center of mass for the situation general areas in the Situation-by-situation training (brown areas in Figure 3). Download Table 3-1, DOCX file.
Figure 3-1
Results from the situation-by-situation training using uncorrected p < 0.05: The proportion of voxels classified as situation general (brown) and situation dependent (blue) or their general locations did not significantly change when using a more lenient threshold. Download Figure 3-1, TIF file.
Figure 3-2
Situation-by-situation Pearson Correlation Values. The observed, cross-validated r values again ranged from 0.18 (the FWE corrected threshold based on permutation testing) to 0.46. Download Figure 3-2, TIF file.
Figure 3-3
Correlation between different numbers of training samples in the across-situation model. The training schemes in the situation-by-situation training strategy included 12 samples, while the across-situation training strategy included 36 videos(“SG36”). To make sure that the difference between the sample sizes of the two training strategies did not influence the results, we performed a random sub-sampling and compared the results with the full sample results. First, twelve videos were randomly sampled, four from each situation as the training sample. We repeated the random sub-sampling and training 200 times and calculated an average model (“SG12”). (A) The correlation values between the SG12 and SG36 are 0.97, 0.95, 0.96 for the situation of Heights, Social, and Spiders respectively, suggesting that the patterns are comparable between SG12 and SG36. (B-C) Whole brain patterns for SG36 and SG12. Noticeably, and as might be expected, the r values are lower in the SG12 than in the SG36, which could be due to fewer shared video stimuli for SG12 with the testing sample. Only four videos could be shared between the training and the testing sample in SG12, whereas both the SG36 and the situation-dependent training strategy have all twelve video stimuli shared between the training and the testing sample. Since our hypothesis is not about comparing the model performance between the situation-dependent and the situation-general model, it does not make sense to present a situation-general model with a reduced number of training samples to match the other model. As such, we decided to keep the situation-general model that included all the data. Future studies with more extensive sample size could investigate the relationship between training sample size and effect size. Download Figure 3-3, TIF file.
Figure 3-4
The figure presents a sample of time-averaged EPI scan for assessing coverage. The cursor is centered on the medial OFC (panel A), and the amygdala (panel B). The sequence captured functional activity in both regions. Download Figure 3-4, TIF file.
Figure 3-5
Time series during video watching period, averaged over participants and all videos. 0 TR indicates the video onset. Time series suggest that functional activity was captured in (A)the orbitofrontal cortex and (B)the amygdala. Download Figure 3-5, TIF file.
Figure 3-6
Stimulus Constant Analysis. To test for brain regions with functional activity that predicts fear ratings while holding the stimulus constant, we calculated predictive models of fear for each stimulus separately. Specifically, we applied LASSO-PCR to data for each video to predict fear ratings across subjects with three-fold cross-validation (i.e., data from 14 subjects were used to train the model and data from 7 subjects were used to test the model iteratively across folds). We then averaged the cross-validated Pearson r-values (i.e., out-sample predictions) across the folds. This process was repeated for each video, which produced 12 r-maps per situation. The r maps for each situation were Fisher Z transformed and submitted to a one-sample non-parametric t-test using FSL’s randomise function for each situation. This analysis identifies brain regions with functional activity that predicts fear ratings across individuals, even while holding the stimulus constant (i.e., results from this analysis cannot be attributed to visual differences between the stimuli), per situation. Panels A-C show that functional activity that predicted fear, even while controlling for stimulus properties, was widely distributed throughout the brain for each situational context. We then performed a conjunction analysis as in the main manuscript (Panels D and E). Areas colored in brown (1.7% of significant voxels) in refer to centroids of voxel-wise neighborhoods with functional activity that predicted fear across all three situations, albeit with flexible neural codes (i.e., the specific model parameters that relate activity patterns to fear ratings were allowed to vary, video-by-video). Areas in light (18.4% of significant voxels) and dark (79.9% of significant voxels) blue refer to areas with functional activity that predicts fear in one or two situations, respectively, but not all three (again, model parameters were allowed to vary, video-by-video, in this analysis). Fear-predictive information is widely distributed across the brain and situation dependent. These overall findings parallel the main findings even while holding the stimulus constant. Download Figure 3-6, PNG file.
Discussion
In this study, we characterized each brain region based on whether it contained functional activity that predicted fear ratings across situations using either the same neural code (i.e., situation general, shared parameters) or flexible neural codes (i.e., situation general, unshared parameters) or, alternatively, whether it only predicted fear in some but not all situations (i.e., situation dependency). For the overwhelming majority of brain regions, models of functional activity for predicting fear were situation dependent (∼98%; Fig. 2). A small portion of voxelwise neighborhoods (∼2%) predicted fear across all three situations using the same model parameters. Even upon allowing the model parameters to flexibly vary by situation, few areas (∼4%; Fig. 3) carried information that predicted fear across all three situations. These results suggest that regional representations of fear are dominated by functionally heterogeneous, situation-dependent signals.
These findings have important implications for understanding the neural representations and “brain signatures” of fear. The term brain signature seems to imply uniformity of representation (Kassam et al., 2013; Peelen and Downing, 2023). However, algorithms commonly used to estimate brain signatures aim to identify the best functional mapping between patterns of brain activity and emotion categories, given the inductive bias of the estimation technique. In the case that there are multiple solutions that leverage different brain representations (Edelman and Gally, 2001; Price and Friston, 2002; Friston and Price, 2003; Marder and Taylor, 2011), different estimation techniques can converge on different solutions, suggesting that there may be multiple signatures of the same emotion category (for details, see Clark-Polner et al., 2017; Kragel et al., 2018; Khan et al., 2022; Lindquist et al., 2022). Correspondingly, “brain signatures” should be interpreted as an analytical approach wherein brain data are used to optimally predict behavior but for which additional considerations are required to test theories regarding the neural representations of emotion (Kragel et al., 2018; Čeko et al., 2022; Lindquist et al., 2022).
Questions regarding how the brain represents emotions lie at the crux of emotion theory (Ekman, 1992; Lindquist and Barrett, 2008; Panksepp, 2011; Lindquist et al., 2013; Barrett and Satpute, 2019; Mobbs et al., 2019). Constructionist theory posits substantial within-category heterogeneity in neural representations of emotion (Barrett, 2006, 2017a,b; Lindquist et al., 2012; Wilson-Mendenhall et al., 2015; Doyle et al., 2022). According to this view, fear refers to a population category constituted from instances with diverse and heterogeneous features (Barrett, 2017a,b; Siegel et al., 2018). Fear occurs when incoming sensory input is made meaningful with respect to similar previous instances in a predictive processing neural architecture (Barrett, 2017b). We reasoned that instances from similar situations are more likely to share features in common with each other, and thus, the situation may provide a useful heuristic to guide whether and which instances serve as priors for conceptualizing future sensory inputs as instances of fear (Satpute and Lindquist, 2019). By this account, fearful situations are not necessarily organized into “types” (e.g., a predator type, a heights type) with type-specific brain states. Rather, some instances of fear involving spiders, for example, may be similar to those involving heights, depending on the constituent features (Barrett, 2013; McVeigh et al., 2023). Consistent with this notion, a substantial portion of brain regions contained fear-predictive codes that generalized across two situations even though few brain regions predicted fear across all three situations. These findings coincide with recent theoretical and empirical approaches wherein context is integral to representation rather than modulating a core response profile (Wilson-Mendenhall et al., 2011; Skerry and Saxe, 2015; Tamir et al., 2016; Satpute and Lindquist, 2019). Constructionist theory also proposes that brain representations of emotion categories will depend on the person, including one’s cultural background (Immordino-Yang et al., 2016; Immordino-Yang and Yang, 2017; Lindquist et al., 2022; Pugh et al., 2022). While our study cannot address this aspect of the theory due to sampling limitations, person-dependent predictive models may be tested in future work. Such work may benefit from using functional hyperalignment to help mitigate variation in functional neuroanatomy across individuals (Haxby et al., 2011).
Appraisal theories suggest that emotions result from evaluating an event's significance to one's well-being and goals (Lazarus, 1991; Moors et al., 2013). If fear involves a particular appraisal configuration (Roseman and Smith, 2001) and specific appraisal dimensions involve the functioning of specific neural circuits or networks (e.g., amygdala for relevance appraisals, hippocampus/amygdala for novelty appraisals; Brosch and Sander, 2013; Smith and Lane, 2015), then one might expect activity in those circuits to generalize in predicting fear across situations. Alternatively, some appraisal models have proposed that fear is associated with many, heterogeneous appraisal patterns (Meuleman and Scherer, 2013) wherein appraisals are not necessarily causal antecedents of a “core fear” state but rather are descriptive features of emotion (Ellsworth and Scherer, 2003; Ortony and Clore, 2015). Our findings showing substantial functional heterogeneity in fear suggest that many different appraisals may take place during instances of fear, although the causal role of the information captured by decoding remains to be tested.
Functionalist models posit that fear refers to a goal (e.g., prevent harm from a predator) that may be achieved by different defensive behaviors (e.g., running, freezing, fighting; Fanselow, 1994; Fendt and Fanselow, 1999; Anderson and Adolphs, 2014; Mobbs et al., 2019). These behaviors are thought to involve a circuit that traverses the amygdala, hypothalamus, and periaqueductal gray, among other primarily subcortical structures. Different configurations of this circuit may drive different defensive behaviors, depending on the situation (e.g., the imminence of the predator). It remains contested as to whether this circuit underlies both defensive behaviors and fearful experiences in a one-system model (Panksepp, 2011; Panksepp et al., 2011) or whether survival behaviors and fearful experiences involve distinct neural systems in a two-system model (LeDoux and Pine, 2016; LeDoux and Brown, 2017).
Notwithstanding issues of spatial resolution with standard 3 T fMRI (Satpute et al., 2013) and constraints of the searchlight approach (Kragel et al., 2018; Zhou et al., 2021), our findings suggest that single-system accounts may not fully account for “fear” (Kragel and LaBar, 2015; Taschereau-Dumouchel et al., 2020). For instance, functional activity in the amygdala predicted fear in some but not all situations—even when the neural codes were allowed to vary to accommodate the idea that a single circuit may engage in different functional configurations to support fear. Establishing boundary conditions for generalization would be a critical avenue for future work. For instance, macrolevel architecture associated with defensive behavior may generalize in predicting fear in situations that share features of a predator–prey interaction, such as predatory imminence (Fanselow, 1994), or, alternatively, when there are similar allostatic demands, regardless of whether the situation resembles predator–prey (Schulkin, 2004; Barrett and Finlay, 2018).
Notably, functional activity in some brain regions, including the posterior temporal cortex and posterior insula, may support situation-general representations. The presence of these representations, although less frequent than situation-dependent signals, suggests that the brain may contain circuitry that processes fearful events (or aspects of fear) in ways that generalize across situations. These areas have been inconsistently implicated in prior MVPA studies on emotion experience (Extended Data Table 2-2). A better understanding of the nature of generalizable signals may help address these inconsistencies. Areas overlapping with (Skerry and Saxe, 2015) or contralateral to (Peelen et al., 2010) the posterior superior temporal sulcus have been implicated in emotion categorization in the context of emotion perception. The posterior insula receives sensory inputs from the body and may play a more general role in arousal or interoception that is shared across mental phenomena (Damasio, 1999; Craig, 2002, 2009; Damasio and Carvalho, 2013; Kleckner et al., 2017; Satpute et al., 2019). The mere processing of emotion words (e.g., the word “fear”) in the absence of an evocative stimulus also involves functional activity that is widely distributed throughout the brain, including in the lateral temporal cortex (Lee and Satpute, 2024), suggesting that conceptual information may also play a factor in explaining whether and which brain regions carry generalizable neural representations of fear. Future work may focus on these areas to replicate these findings; determine if they carry generalizable, and specific, neural codes that predict fear; and understand the nature of this information.
Our findings underscore the importance of testing for external validity and generalizability of a given brain–behavior relationship (Shackman and Wager, 2019; Lee et al., 2021). Many studies in affective neuroscience preclude tests for external validity by examining fear in a single context or averaging findings across trials. Yet, our findings suggest that generalizability may be strongly constrained by the situation. To effect, and perhaps owing to the lack of robust predictive models of valence (for a review, see Lee et al., 2021), recent theoretical models in affective neuroscience incorporated modality as an organizing factor (Chikazoe et al., 2014; Chang et al., 2015; Satpute et al., 2015; Kim et al., 2017; Miskovic and Anderson, 2018; Kim et al., 2019; Lee et al., 2021). For instance, recent work has advanced a “visually induced fear signature” (Zhou et al., 2021). Yet, we only used visual stimuli and yet we still found robust evidence of context dependence. These findings suggest that representations of emotion categories are not necessarily organized into modality-dependent “types” but rather that the sensory modality is just one aspect of a broader interpretation of context, wherein context could be characterized in terms of predictions and prediction errors that are derived from prior experience (Lee et al., 2021; Barrett, 2022).
One potential explanation for our findings is that the visual features that drive higher fear in the context of spiders vary from those that drive higher fear in social or heights contexts. These differences in visual features could be viewed as part of the emotion representation or auxiliary to it (depending on one’s theoretical perspective), and their role may be investigated in future work in which these features are explicitly modeled. We also conducted an additional analysis wherein we estimated predictive models of fear while holding the stimulus constant and found that even when doing so, functional activity that predicts fear ratings was widely distributed throughout the brain and varied by situation (Extended Data Fig. 3-6).
Insofar as fear holds a central position in emotion theory, it stands to reason that other emotion categories, too, are likely to exhibit degeneracy, or many-to-one relationships between brain states and psychological constructs (Friston and Price, 2003; Barrett and Satpute, 2019; Doyle et al., 2022; Khan et al., 2022). Notably, our findings converge with recent work showing strong evidence of situation dependence in the peripheral autonomic correlates of fear, too (McVeigh et al., 2023). Modeling this variation may be key to developing a fundamental understanding of complex mind–brain–behavior relationships alongside personalized treatments in clinical populations.
Data Availability
Anonymized data will be deposited in OpenNeuro (https://openneuro.org/) after publication. Analysis scripts are available in Github at https://github.com/yiyuwang/AffVids_mvpa.
Footnotes
Research reported in this publication was supported by the National Science Foundation Division of Graduate Education (NCS 1835309).
The authors declare no competing financial interest.
- Correspondence should be addressed to Yiyu Wang at wang.yiyu{at}northeastern.edu or Ajay Satpute at a.satpute{at}northeastern.edu.
