Abstract
The necessity of conscious awareness in human learning has been a long-standing topic in psychology and neuroscience. Previous research on non-conscious associative learning is limited by the low signal-to-noise ratio of the subliminal stimulus, and the evidence remains controversial, including failures to replicate. Using functional MRI decoded neurofeedback, we guided participants from both sexes to generate neural patterns akin to those observed when visually perceiving real-world entities (e.g., dogs). Importantly, participants remained unaware of the actual content represented by these patterns. We utilized an associative DecNef approach to imbue perceptual meaning (e.g., dogs) into Japanese hiragana characters that held no inherent meaning for our participants, bypassing a conscious link between the characters and the dogs concept. Despite their lack of awareness regarding the neurofeedback objective, participants successfully learned to activate the target perceptual representations in the bilateral fusiform. The behavioral significance of our training was evaluated in a visual search task. DecNef and control participants searched for dogs or scissors targets that were pre-cued by the hiragana used during DecNef training or by a control hiragana. The DecNef hiragana did not prime search for its associated target but, strikingly, participants were impaired at searching for the targeted perceptual category. Hence, conscious awareness may function to support higher-order associative learning. Meanwhile, lower-level forms of re-learning, modification, or plasticity in existing neural representations can occur unconsciously, with behavioral consequences outside the original training context. The work also provides an account of DecNef effects in terms of neural representational drift.
Significance Statement
This study examined the role of conscious awareness in human learning by using functional MRI decoded neurofeedback. These techniques enabled participants to self-regulate their brain activity to align with the perceptual representations generated by a real-world entity (i.e., dogs), without awareness of the content they represented. We demonstrated that established brain conceptual representations can be unconsciously modified, influencing visual search behavior for the targeted perceptual content. Nonetheless, our research suggests that conscious awareness plays a role in more advanced forms of associative learning. Further, this study offers methodological insights for improving DecNef protocols and suggests potential for personalized interventions, including guidance to correct maladaptive conceptual representations.
Introduction
The acquisition of word meanings depends on associative learning of symbols and episodic referents (Smith, 1978; Blewitt, 1982). A long-standing question in psychology and neuroscience concerns whether conscious awareness is necessary for higher-level associative learning.
Prior research provided evidence of unconscious associative learning between subliminal visual items and conscious rewards (Pessiglione et al., 2008; Seitz et al., 2009; Skora et al., 2021), as well as fear conditioning responses to stimuli not consciously perceived (Raio et al., 2012). Studies using evaluative learning paradigms have suggested that emotional value can be conditioned into neutral visuals, like nonsense syllables, and that these associations can be made implicitly, even without participants awareness (Staats and Staats, 1957; Tryon and Cicero, 1989; Hofmann et al., 2010). However, the aforementioned evidence is controversial and has faced substantial criticism (Shanks and Dickinson, 1990; Shanks and John, 1994; Field, 2000; Mertens and Engelhard, 2020), raising the possibility that the results were due to experimental demand characteristics and participants’ awareness of the experimental goals, not from non-conscious knowledge (Page, 1974; Field, 2000; Stahl et al., 2016).
Research on unconscious associative learning is limited due to the low signal-to-noise ratio of the brief and masked unconscious stimulus. Learning may falter in these conditions. Addressing this challenge requires an increase in signal strength in the unconscious condition (Lau, 2022). Here, we used functional MRI decoded neurofeedback (fMRI-DecNef) to enable participants generate brain patterns matching a target state without awareness of the content represented by those brain patterns (Watanabe et al., 2017). Previous DecNef studies mainly centered on modulating affective responses to faces (Shibata et al., 2016), fear (Koizumi et al., 2016; Taschereau-Dumouchel et al., 2018) or confidence (Cortese et al., 2016). Here we used DecNef training to shape brain representations and form novel conceptual knowledge. Specifically, we aimed to incept the perceptual meaning of a real-world entity into a Japanese hiragana that had no inherent meaning for our participants, by reinforcing related neural patterns. To achieve this, we developed an open framework for DecNef that integrates several enhancements to the standard pipeline, namely, a customized decoder that provides more fine-grained and less binomial/overconfident predictions during DecNef training (Margolles et al., 2023).
First, we trained a classifier to differentiate between brain patterns associated with perceiving dogs or scissors images (Fig. 1A). During several DecNef training sessions, participants self-regulated their brain activity while viewing a hiragana to maximize the size of a feedback disk and monetary reward (Fig. 1B). Participants were unaware that the feedback depended on how closely their brain activity patterns matched the classifier’s representation of dogs in the bilateral fusiform. We chose the fusiform due to its role in object recognition and imagery (Stokes et al., 2009; Reddy et al., 2010; Lee et al., 2012; Dijkstra et al., 2021). We anticipated that this approach would create associative knowledge, imbuing the hiragana with the meaning of dogs. The behavioral significance of the DecNef was evaluated using a visual search task involving dogs and scissors, with the search target being pre-cued by the DecNef hiragana or by a control hiragana (Fig. 1C). We hypothesized that associating the hiragana to the dogs concept would lead to automatic attention shifts toward the matching search item, resulting in faster performance and distinct neural responses compared to the control hiragana (Soto et al., 2007, 2008). Additionally, we compared the search behavior of the DecNef group to a control group to understand better how DecNef training influenced the target category’s representation itself. This contrast explores the potential of DecNef training to unconsciously modify the neural representation of the target concept, resulting in distinct search performance for dogs targets compared to scissors.
Overview of the experimental design. A, Participants viewed pictures of dogs and scissors. We trained a custom logistic regression decoder using perceptual data from the bilateral fusiform to distinguish between the multivoxel patterns associated with each class. B, Over four DecNef sessions, our goal was to covertly associate a neutral hiragana with the perceptual meaning of dogs using reinforcement learning. Participants self-regulated their brain activity in the bilateral fusiform while looking at an hiragana. Monetary rewards were given based on the alignment of their brain activity patterns with the target class (i.e., dogs) as determined by the classifier. The decoded volumes matched the three fMRI volumes acquired between 5 and 11 s after hiragana onset, aligning with the peak of the Hemodynamic Response Function during self-regulation. Importantly, participants remained unaware of the experiment’s true objective and the existence of a target category. C, Upon completion of the four DecNef sessions, a visual search task was administered to evaluate the behavioral and neural impact of the DecNef training. Participants looked for dogs or scissors targets, which were either pre-signaled by the hiragana used during DecNef training or by a new hiragana. We anticipated changes in search times due to the unconscious priming of dogs’ perceptual representations driven by the targeted hiragana.
Materials and Methods
Participants
The DecNef training involved a group of 18 young adults (23–36 years old, mean ± SD age: 28.19 ± 3.73 years, 6 males) with normal or corrected-to-normal vision, who were right-handed and unfamiliar with the Japanese language. A sample size of 18 is well above average in DecNef studies (Cortese et al., 2016; Koizumi et al., 2016; Taschereau-Dumouchel et al., 2018). The experimental procedures were conducted in accordance with the Declaration of Helsinki and approved by the BCBL Research Ethics Board. All participants provided written informed consent and at the end of the experiment, received a fixed monetary compensation of 105
€ for their participation in the study. Additionally, we included a control group of 36 young participants to double the DecNef group sample size and improve the statistical power of our analysis for detecting any differences between groups (20–29 years old, mean ± SD age: 23.16 ± 2.62 years, 11 males). One of the participants had to be replaced due to a high number of errors (i.e.,
We also note here that the decoded neurofeedback paradigm is a special case when it comes to the selection of a control group. Standard fMRI neurofeedback studies are based on explicit approaches in which participants are aware of the strategy that should be used to modulate brain activity in the targeted region, and hence standard fMRI neurofeedback approaches use control groups in which participants receive feedback of a non-target brain region, sham feedback, or perform mental strategies without neurofeedback signaling (Sorger et al., 2019). In the case of DecNef, and specifically in our associative DecNef study, the above is not applicable (Sorger et al., 2019) because the neurofeedback training occurs at an implicit level, namely, participants are unaware of (i) the target brain region, (ii) what the neurofeedback signal represents, and (iii) the strategy that may be used to regulate activity in the target region. Because of this, we elected to include a control group of participants that did not undergo DecNef in order to test that the behavioral pattern of search performance in the DecNef group was indeed specific to DecNef training. Note that a similar approach was used in prior associative DecNef studies (Amano et al., 2016; Knotts et al., 2019).
Apparatus
Our DecNef pipeline required three high-performance computers: the computer controlling the MRI scanner; a server computer, which analyzed the data in real time; and a client computer for feedback and stimuli presentation during MRI scanning. The three computers were connected through an ethernet local area network.
MRI settings
Whole-brain data were acquired on a 3T Siemens Magnetom Prisma-Fit whole-body MRI scanner with a 64-channel head-coil at the BCBL facilities. A gradient-echo echo-planar imaging sequence was used with the following parameters: time-to-repetition (TR) =2000 ms, time-to-echo (TE) =28 ms, flip angle (FA)
fMRI scans for decoder construction
The decoder construction session involved four perceptual runs and two mental imagery runs, each lasting approximately 8 min.
First, participants completed four perceptual runs in which they were presented with pictures of dogs and scissors from the THINGS database (Hebart et al., 2019). Each perceptual run was composed of seven dogs and seven scissors trials, and every trial had a duration of 22 s. A trial started with a white fixation point lasting 1500 ms (
Furthermore, we sought to consider that any representation induced during DecNef, resembling the target representations, would be internally generated. Consequently, following the perceptual runs, participants underwent two mental imagery runs. During these runs, they were instructed to vividly imagine a picture of their choice from each of the visual categories. This image was chosen at the start of the perceptual runs based on its ease of imagination for the participant. Each mental imagery run comprised nine imagery trials for every category, with each trial lasting 22 s. These trials began with a 1500 ms white fixation (
fMRI training data preprocessing
The fMRI volumes used for training the decoder were preprocessed using a customized pipeline in Analysis of Functional NeuroImages software v21.0.05 (AFNI, Cox, 1996), which was created through the Nipype v.1.4.1 (Gorgolewski et al., 2011) interface module for Python 3.6 (see Margolles et al., 2023).
The first experimental volume from the decoder construction session, obtained after the heat-up volumes, served as a functional reference for the realignment of volumes for each participant. That reference Digital Imaging and Communications in Medicine (DICOM) file was converted to Neuroimaging Informatics Technology Initiative (NifTi) format using Dcm2Niix (Li et al., 2016) and de-obliqued to match cardinal coordinate orientation with AFNI 3dWarp command.
The subsequent perceptual and mental imagery volumes were co-registered to the reference volume using AFNI 3dvolreg ,nd using heptic (7th order) Lagrange polynomial interpolation (Oakes et al., 2005). The OpenSesame task logs were used to label all volumes, indicating their corresponding stimulus category, trials onset times, and fMRI run. For each run, we linearly detrended the fMRI time-series to remove drift before training the decoder. To match the characteristics of the hemodynamic response function (HRF) and capture the peak neural activity associated with the presented stimuli, we selected these fMRI volumes occurring between 5 and 18 s following the initial picture/auditory prompt presentation. This spanned seven volumes per trial in both the perception and mental imagery runs. The selected perceptual and mental imagery samples were stacked independently for region of interest (ROI) delineation and decoder training. Next, the stacked perceptual volumes were z-scored by subtracting the samples’ mean value at each voxel and dividing by the standard deviation. The standardization of the scans in the imagery runs was performed using the voxels mean and standard deviation from the perceptual samples. This approach was utilized to establish a uniform and comparable scale for the data from both conditions and investigate the extent to which information could generalize from perception to mental imagery.
Algorithm for decoding
To establish the ROI for DecNef training, our initial step was to outline the regions where perceptual representations, triggered by the pictures, converged with internally generated representations during mental imagery. Previous research had identified these overlapping regions in the fusiform gyrus (Stokes et al., 2009; Reddy et al., 2010; Lee et al., 2012; Dijkstra et al., 2021). The rationale behind this approach is that the brain activity patterns used for training the classifier are mainly externally triggered through the presentation of the images, while the relevant brain activity patterns during DecNef training may stem from internal mental simulation. Therefore, our goal was to enhance the cross-domain generalization of our machine learning classifier and facilitate DecNef training by focusing on these voxels located at the overlap between perception and imagery in the fusiform region.
Accordingly, a searchlight decoding analysis was conducted for each participant. The initial step involved creating a binarized functional brain mask from the skull-stripped reference brain volume. Next, z-scored volumes from the perceptual and mental imagery runs were masked to exclude non-brain voxels with zero values. Then, we performed a cross-domain searchlight analysis in the subject’s native space using Brain Imaging Analysis Kit (BrainIAK v. 0.9.1; http://brainiak.org; Kumar et al., 2021). A searchlight of 9*9*9 voxels was defined by a cube cluster of 4-voxel (3 mm*4 voxels = 12 mm) radius. We employed the scans from the perceptual runs, excluding data from visual noise samples, to train a standard logistic regression classifier with L2 regularization within each searchlight. This classifier was then applied to the imagery data in order to decode the content of mental imagery volumes as either dogs or scissors categories. We evaluated the decoder’s performance by computing the area under the receiver operating characteristic curve (AUC) and subsequently assigned this performance score to the center voxel of each searchlight. This resulted in a comprehensive whole-brain map of cross-domain decoding results.
Following this, we manually generated a spherical mask in each participant’s original functional space. The center of this mask was positioned around the voxels demonstrating the highest level of generalization from perception to mental imagery. In doing so, we analyzed the resulting searchlight maps across the two mental imagery runs to draw our fusiform ROIs. The goal was to guarantee that the ultimate ROI for DecNef encompassed areas where the overlap between perception and imagery remained consistent over time. Visualizing the searchlight maps for each run allowed us to achieve a balance between selecting an overly narrow ROI and an overly broad ROI. Accordingly, this approach serves to address potential issues, such as neglecting spatial variability during DecNef training due to a narrow ROI or the risk of overfitting our decoder because of an excessively broad ROI. Furthermore, the fusiform gyrus encompasses various regions with distinct functional and representational roles (see Schwarzlose et al., 2005; Pitcher et al., 2009 in relation to faces, bodies, and objects). Opting for selecting an overly broad ROI and overly distributed voxels across the fusiform would pose a challenge to interpreting the outcomes of DecNef training. Therefore, we chose to utilize searchlight maps as a method to pinpoint fusiform areas characterized by more consistent functional, spatial, and temporal patterns.
The brain areas showing better classification performance varied to some extent across participants. Although our a priori target was on the fusiform region, the first participant exhibited noteworthy cross-domain decoding, particularly in the parietal cortex. Consequently, we incorporated this region alongside the bilateral fusiform into the DecNef training. The second and third participants exhibited greater decoding performance in the left fusiform gyrus compared to the right, leading us to choose the left fusiform as the target for DecNef. However, for subsequent participants, we opted to utilize a uniform region for DecNef training, centered in the bilateral fusiform gyrus. Across these three participants, this region consistently demonstrated a high degree of decoding accuracy. We aimed to ensure uniform ROI dimensions for both the left and right fusiform areas across all participants. The masks, on average, covered 561.35 voxels per hemisphere, with a standard deviation of ±150.29 voxels. The spheres encompassing the left and right fusiform regions were combined to form a single bilateral fusiform mask. The stacked perceptual and mental imagery data were ultimately masked to preserve only the voxels located within the targeted bilateral fusiform ROI.
Subsequently, for each participant, we trained a custom decoder on the ROI mask obtained through the aforementioned searchlight analysis. The perceptual data served as the training set, and the classifier hyperparameters were optimized using the mental imagery data as the validation dataset. This method enhanced the decoder’s ability to generalize to the DecNef training sessions. During the development of our DecNef pipeline, we observed that sparse logistic regression classifiers (Yamashita et al., 2008), as commonly employed in previous DecNef training studies (Cortese et al., 2016; Koizumi et al., 2016; Cortese et al., 2017; Taschereau-Dumouchel et al., 2018), tended to produce highly bimodal/overconfident probability distributions for out-of-sample distribution predictions in the imagery or DecNef runs (Bai et al., 2021). Moreover, prior DecNef research did not explicitly address representational noise during DecNef training (Watanabe et al., 2017), which encompasses neural patterns that do not directly correspond to the specific trained representations or categories. This factor may reduce the capacity of the decoder to discriminate target from non-target representations, constraining the feedback signal and thereby hinder participants’ learning. These issues were addressed by developing a custom logistic regression classifier using Keras (Chollet, 2015) (v2.7) and Tensorflow (Abadi et al., 2016) (v2.7). The classifier consisted of a fully connected input and a softmax output layer, specifically designed for the binary classification of samples into dogs or scissors categories. To prevent overfitting, we applied regularization to the kernel weights and layer outputs using an L2 kernel regularizer of 0.1 and an L1 activity regularizer of 0.001, respectively. We used a default stochastic gradient descent optimizer with a learning rate of 0.0001. Specifically, we employed binary cross-entropy (BCE) loss as the loss function with scissors labeled as 0, dogs as 1, and visual noise images as 0.5. By adopting this strategy, we achieved a more continuous representational gradient between categories and decreased the network’s susceptibility to representational noise during testing. As a result, the network exhibited improved discrimination between the relevant classes by generating more moderated predicted probabilities. To evaluate the performance of our decoder, we utilized the AUC. We set a tolerance of 0.0001, which represents the minimum change in the validation loss required to qualify as an improvement. We also established a patience of 50 epochs; if there was no improvement in the validation loss after these 50 epochs, the training would be halted. Finally, we utilized a batch size of 64 to compute the gradient during each iteration. A comparison among the standard and sparse regression decoders, as well as our custom classifier using BCE loss can be found in Margolles et al. (2023).
DecNef training procedure
The DecNef training consisted of four fMRI sessions, each lasting approximately 70 min, performed on consecutive days at a rate of one session per day in the early morning (8 A.M.). In the first three sessions, there were a total of eight fMRI runs. These runs had a duration of approximately 6.5 min each and consisted of 18 trials per run, with each trial lasting approximately 19 s. In the final session, only two fMRI runs were carried out, both of the same duration as those in the earlier sessions. Subsequently, a visual search task was performed within the MRI scanner to serve as a behavioral assessment for DecNef training. A 2-min rest period followed the completion of each run.
During the DecNef training sessions, participants encountered one of two gray Japanese hiragana characters (の or せ), with the presentation of each character being counter-balanced across participants to ensure equal distribution. Since the participants were not familiar with the Japanese language, these characters had no inherent meaning for them. Our goal was to establish a novel unconscious association between the perceptual representation of dogs and these neutral hiragana characters through reinforcement and associative learning.
Considering that distinct stimulus categories, such as living and non-living entities, are associated to distinct multivoxel patterns and neural substrates (Pitcher et al., 2009), attempting to counterbalance the target classes (i.e., dogs and scissors) would have led to the formation of two smaller groups, each comprising only nine subjects. In such circumstances, it becomes notably more difficult to derive meaningful conclusions or discern significant effects, largely due to the limited statistical power associated with smaller sample sizes. Opting to concentrate on a single category across all participants would ensure a more robust evaluation of DecNef’s neural effects. Additionally, this approach would contribute to reducing potential sources of variability in participants’ behavioral responses. This approach has been utilized in previous associative DecNef studies (Amano et al., 2016; Knotts et al., 2019). Due to these considerations and the known responsiveness of the bilateral fusiform gyrus ROI to living entities (Downing et al., 2006; Connolly et al., 2012; Sha et al., 2015), indicating that targeting dogs representations might be more effective in inducing neural changes, we chose to assign dogs as the target category for all participants.
The entire experimental procedures were carried out implicitly. Participants were aware that they were participating in a neurofeedback study aimed at self-regulation of brain activity, but they were kept entirely unaware of the true purpose of the intervention, which included the existence of a target category. To prevent participants from associating the prior fMRI session used for the decoder construction with the subsequent DecNef training sessions, a new experimenter was assigned, and they were explicitly informed that they were taking part in a different experiment. Additionally, participants were asked to complete different consent forms and were given separate information sheets.
We instructed participants to engage in self-regulation of their brain activity with the goal of maximizing their monetary rewards by enlarging the size of a red circle displayed as feedback. At the beginning of the DecNef training, participants were told that they would receive a fixed compensation of 85 € for their participation in the experiment. Additionally, they had a chance to receive a performance-based bonus of €25, with the potential to earn up to €0.10 for each trial. Participants were explicitly encouraged to freely explore various mental strategies and choose those that proved most effective in achieving their goal. Nonetheless, we emphasized the importance of avoiding strategies that could introduce potential artifacts to the fMRI analysis, such as changes in breathing or body movements. After each session, participants were given an estimated report of their earnings designed to maintain their motivation. During the final DecNef session, participants were engaged in only two fMRI runs and were encouraged to employ their most effective mental strategies from previous sessions to maximize the size of the red circle. However, in order to fulfill potential ethical considerations, all participants were given a fixed amount of 110 € at the end of the study, which included the 85 € compensation for their participation and the 25 € bonus reward.
At the outset of each trial (Fig. 1B), the hiragana character was displayed on a black background for 6 s, subtending a visual angle of
The fMRI scans during DecNef training were co-registered with the functional reference volume acquired during the decoder construction session, using the same preprocessing pipeline utilized for training the decoder (Margolles et al., 2023). To ensure timely processing in real-time and prevent delays in analyzing incoming volumes, it was crucial that the co-registration times were consistently shorter than the fMRI TR. Here, co-registration of each volume took approximately 0.7 s (vs. TR = 2 s). Then, we applied the functional mask of the bilateral fusiform ROI extracted for each participant. Spatial smoothing was not performed. Linear detrending was applied to the ROI data, encompassing all scans available from a particular run up to the current volume, and including an extra set of 20 baseline scans collected at the onset of each run. Previous neurofeedback studies used z-scoring procedures based on run/session baselines or prior data from the current run (Oblak et al., 2019). In contrast, we utilized the mean and standard deviation parameters for each voxel obtained from the scans used in decoder construction, specifically the perceptual data, for standardizing the volume data. This method does not impose any data leakage on the information that is used for decoding, because this procedure does not use any information regarding the classes. Moreover, the classification analysis is performed on the multivariate pattern across voxels and the voxel-wise standardization procedure is blind to these patterns. Using the mean and standard deviation from the decoder construction data for voxel-wise standardization in the DecNef scans merely aligns the distributions of the train and test sets at the single voxel level, providing several benefits. For example, this approach enhances result interpretability by maintaining consistency across all runs and sessions with a common reference data distribution. Additionally, it ensures that the test data scale to match the distribution of the training data, potentially enhancing the decoder’s performance during DecNef training (see Margolles et al., 2023 for a comprehensive comparison of how different DecNef standardization techniques affect decoding performance). Real-time decoding focused on volumes captured between 5 and 11 s after the hiragana presentation, resulting in the decoding of three volumes per trial. In previous DecNef research, a common practice involves utilizing volume averaging to improve the signal-to-noise ratio during decoding (Watanabe et al., 2017; Taschereau-Dumouchel et al., 2018). However, as noted earlier, standard and sparse logistic regression decoders tend to yield highly bimodal or overconfident category probabilities when applied to single volumes in cross-domain classification. We reasoned that providing participants with a more moderated and continuous feedback signal might enable them to refine their strategies and learn how to self-regulate their brain activity more effectively. To illustrate, consider a scenario where a specific strategy does not accurately reflect the intended pattern (i.e., the probability of decoding the target class is only marginally significant). However, if the classifier tends to provide extreme or overly confident predictions, then this strategy would appear to be just correct for the participants. In our pilot studies, we investigated the issue of standard and sparse decoders showing overly confident predictions for out-of-sample distributions. We conducted a specific analysis to evaluate the generalization from perception to imagery data used in the decoder construction. Our goal was to assess whether the average decoding probability, computed by independently decoding volumes and then averaging their probabilities, demonstrated reduced bimodality compared to the probabilities obtained using the conventional method of averaging all volumes before decoding in each trial. We elected to decode each volume and then compute the average probability as feedback signal because the analysis indicated that the probability distributions were less binomial/overconfident in this case (see Margolles et al., 2023 for a comparison of both averaging techniques).
Visual search task
After completing the final two DecNef training runs, we conducted a visual search task while the participants were still inside the MRI scanner. The purpose was to evaluate the behavioral and neural effects of DecNef training regarding the creation of new conceptual associations. The experiment involved five fMRI runs, each with a duration of approximately 6 min and encompassing 56 trials per run.
Each trial started with a white fixation mark displayed on a black background for 500 ms (
The participants in the control group performed the same visual search task outside the fMRI scanner, but we ensured that stimulus presentation and event timing were consistent with the experimental conditions of the DecNef group. Also, we matched the exposure to the images in both experimental groups prior to performing the search task. Accordingly, the control group was presented with a block-wise design where images of dogs and scissors were shown, similar to the session in the DecNef group used to build the classifier. Next, they practiced the visual search task without any hiragana presentation. Finally, they completed the visual search task using the same timings and jitters as those used by the fMRI participants. Additionally, they listened to scanner noise through headphones while carrying out the task. Previous studies assessing the processing advantage for living items (i.e., animals) over non-living stimuli indicate that low-level visual features, such as power spectrum, luminance, and contrast, cannot account for this processing advantage (Wichmann et al., 2006, 2010) and that higher-level semantic categorization processes are most critical (He and Cheung, 2019; Moon et al., 2022). However, Thorpe et al. (2001) showed how the detection of animal items dropped with increasing eccentricity levels in peripheral vision. Hence we controlled for the visual angles of the search stimuli across the DecNef and control groups.
Questionnaires
After completing the DecNef training but before the visual search task, participants responded to a series of questions related to the effort required to maximize monetary rewards and their perceived ability to manipulate the circle’s size. We used a Likert scale from 1 to 10, where the lowest value signified minimal effort/control and the highest value represented significant effort/control. We also asked them about the strategies they used during DecNef sessions and any hypotheses they had concerning the experimental objectives.
After the visual search task, we revisited the participants’ understanding of the experiment’s goals and asked them to make a guess about the category targeted by DecNef, which could be either dogs or scissors. They were also asked to indicate their level of confidence in their answer using a four-point Likert scale (0—chance; 1—not so sure; 2—sure; and 3—really sure). Lastly, participants were given a debriefing regarding the true objectives of the experiment.
Off-line whole-brain searchlight analysis of DecNef training
We used searchlight analyses to examine the probability of detecting the target category throughout the DecNef sessions. The goal of this analysis was to assess the role of the bilateral fusiform gyrus during DecNef training, in addition to the involvement of other brain regions in representing the target class. Similar to previous searchlight analyses, we utilized a standard logistic regression classifier with L2 regularization to train on the perceptual data and calculate the probability of decoding the dog class across all DecNef trials. While our custom logistic regression classifier with BCE loss would have been the most suitable option for this analysis, given that it was utilized during DecNef training in our ROI, we decided against using it for searchlight analyses. This decision was based on the substantial computational expenses and the extended training duration it would demand for each searchlight. The searchlight analysis was conducted using the BrainIAK library (Kumar et al., 2021) for each participant. The resulting searchlight maps were then registered from the functional to the participant’s T1-weighted structural space using epi_reg tool in the FMRIBL Software Library (FSL, Jenkinson et al., 2012), and from the structural space to the Montreal Neurological Institute (MNI) standard space (MNI-152 template, 2*2*2 mm3) using FMRIB’s Linear Image Registration Tool (FLIRT, Jenkinson et al., 2002, 2012). A non-parametric one-sample t-test was used to identify the brain regions where the average probabilities were significantly above the chance level of 0.5 across participants. This contrast was calculated using FSL randomize with threshold-free cluster enhancement (cluster correction) and 10,000 random permutations (Smith and Nichols, 2009). To increase the statistical power of the analyses, a spatial full-width-half maximum (FWHM) Gaussian kernel of 5 mm was used for variance smoothing. The family-wise error correction was applied to adjust for multiple comparisons. We obtained a whole-brain corrected statistical image, which was then thresholded to display only statistically significant clusters (p < 0.05).
Linear mixed-effects modeling analyses
Mixed-effects modeling was used to examine participant’s learning effects throughout the DecNef training, namely, whether participants could progressively enhance their ability to induce neural patterns related to the target category over the course of training. The design involved repeated measures for each participant and a hierarchical structure comprising 2 groups of hiraganas, 18 participants, 4 sessions, 8 runs, and 18 trials. In cases of non-independence of the data, multilevel models, specifically linear mixed-effects models (LMMs), have gained popularity in psychological research. LMMs offer several advantages, such as accounting for both within-subject and between-subject variability, handling missing data, and examining relationships between variables at different hierarchical levels. Traditional analysis methods, such as repeated-measures ANOVA, often aggregate data within a single condition or hierarchical level, assuming data independence, which can lead to a loss of information and statistical power. In comparison, LMMs can capture variability in the observations by leveraging all levels of data, using both fixed and random effects modeling (Singmann and Kellen, 2019; Meteyard and Davies, 2020). To perform the linear mixed model analysis, we used the lme4 package (Bates et al., 2014) in the R programming language (R Core Team, 2021). Furthermore, we employed the lmerTest package (Kuznetsova et al., 2015) to determine the statistical significance of our results. This package estimated p-values for mixed models using Satterthwaite’s method to approximate the degrees of freedom for t and F tests.
LMM analysis of DecNef induction probabilities
Here we used the decoding probability of each trial as the dependent variable and the global trial index (1–468) as a fixed effect predictor. This independent variable captures the progression of performance across trials and is critical for assessing the learning effects. We also sought to account for the potential variability in the data arising from the hierarchical structure of our study. To achieve this, we introduced a random intercept for every distinct combination of the hiragana group, participant (1–18), session (1–4), and run (1–8) levels. Incorporating additional time-dependent information from the run or session serve as a broader contextual factor to partial out learning contributions that are trial specific from those that develop more gradually, including fatigue effects unrelated to the trial-wise learning trend. Specifically, we used the LMM formula: lmer(trial_probability ∼ global_trial_index + (1 | hiragana group/participant/session/run)) to conduct our analyses in R. The dependent variable was trial_probability, the independent variable was global_trial_index, and the term (1 | hiragana group/subject_nr/session/run) represented the random intercepts. Importantly, the global trial index variable was treated as a continuous numeric variable, while the hiragana group, participant, session, and run variables were categorical variables with discrete levels.
LMM analysis of behavioral data from the visual search task
To examine the effects of the DecNef training on the visual search task, a second LMM was utilized to analyze the reaction times (RTs) of the correct responses. RTs outside of three standard deviations from each participant’s average RT were excluded. The factors hiragana (targeted vs. control), search target (dogs vs. scissors), and their interaction were set as fixed effects. Additionally, we included each participant in the DecNef group as a random effect to account for inter-subject variability with the following formula: lmer(RTs ∼ hiragana + search target + hiragana * search target + (1 | participant)). In this model, the control hiragana level was used as the reference level for the hiragana factor, while the scissors level was used as the reference level for the search target factor.
Moreover, in order to compare the RTs between the DecNef and control groups, we combined the RTs across the levels of the hiragana factor (targeted vs. novel), as this factor did not apply to the control group. Then, we conducted a similar LMM using the following formula: lmer(decision RTs ∼ search target + experimental group + search target * experimental group + (1 | participant)). In this model, the scissors level was used as the reference level for the search target factor, while the control group level was used as the reference level for the experimental group factor.
Univariate fMRI analyses performed on the visual search task
Although the bilateral fusiform gyrus was our a priori ROI, it is possible that DecNef-related neural changes might also occur in various regions within the ventral visual pathway. Hence, our initial approach involved conducting whole-brain analyses to explore group effects during the visual search task.
The FSL v6.0 suite tools were used for conducting univariate analyses (available at fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSL, Jenkinson et al., 2002; Smith et al., 2004). First, the T1-weighted structural image was cropped by employing the FSL robust FoV method to remove the neck and lower head slices. The resulting image was then skull-stripped using the brain extraction tool (BET). Next, raw functional data in NifTi format were preprocessed using fMRI expert analysis tool (Woolrich et al., 2001, 2004b). To ensure signal stabilization, the first five volumes of each functional run were discarded and the MCFLIRT algorithm was applied to correct for head-motion (Jenkinson et al., 2002). Non-brain voxels were removed through functional brain extraction, and the data were smoothed with a 5 mm FWHM Gaussian kernel. Additionally, high-pass temporal filtering was performed with sigma = 60 s. Finally, the functional data were linearly registered to the high-resolution structural brain image using a boundary-based registration (BBR) cost function. Then, linear affine transformations with 12 parameters were used to transform the data into the standard MNI-152 space (2*2*2 mm3).
Following the preprocessing of the data, we carried out hierarchical general linear modeling using FMRIB’s improved linear modeling. For each visual search run, the first-level analysis modeled the time-series of each voxel by incorporating four regressors (Woolrich et al., 2001): targeted hiragana–dogs target, targeted hiragana–scissors target, control hiragana–dogs target, and control hiragana–scissors target. Explanatory variables in the general linear model (GLM) were specified based on the onset of hiragana in each trial across the different experimental conditions. These onset times were adjusted to account for the five heat-up volumes that were discarded in the preprocessing stage. A double-gamma canonical HRF was used to model each trial in the GLM beginning with the onset of the hiragana. To address motion-related confounding factors, the standard and extended motion parameters estimated by motion correction using FMRIB's Linear Image Registration Tool (MCFLIRT) algorithm were included as covariates. Additionally, to account for temporal autocorrelation in the voxel time-series, prewhitening was performed on the data. To improve the model, we incorporated temporal derivatives and filtering to the design matrix, which helped to reduce the effects of lags in the onset of the HRF and minimize the presence of low-frequency noise over time. Four regressors were used: control hiragana–dogs target, control hiragana–scissors target, targeted hiragana–dogs target, targeted hiragana–scissors target. For each run, we calculated four contrasts of parameter estimates (COPEs), namely, control hiragana > targeted hiragana (i.e., 1, 1, − 1, − 1); control hiragana < targeted hiragana (i.e., −1, − 1, 1, 1); control hiragana–dogs target > targeted hiragana–dogs target and control hiragana–scissors target < targeted hiragana–scissors target (i.e., 1, − 1, − 1, 1); control hiragana–dogs target < targeted hiragana–dogs target and control hiragana–scissors target > targeted hiragana–scissors target (i.e., −1, 1, 1, − 1). Then, we combined the first-level COPEs files into a second-level fixed-effects model. Finally, group-based, third-level analyses (Woolrich et al., 2004a,b) were performed using the second-level COPEs files obtained in the previous step for each individual. To address inter-subject variability within each group, a mixed-effects analysis was conducted using the FMRIB local analysis of mixed effects 1 + 2 algorithm. Group-based analyses used cluster-based thresholding, with a voxel-wise threshold of Z = 3.1 and a cluster significance threshold of P = 0.05, corrected for multiple comparisons using Gaussian random field theory. The uncorrected contrast of parameter estimates and zstats for the different contrasts are available at https://pedromargolles.github.io/pyDecNef/.
After completing the whole-brain analyses, we narrowed our focus to the bilateral fusiform gyrus as our target ROI. This was achieved by applying the GLM to binary masks of the fusiform gyrus in MNI space. These masks were generated based on the off-line searchlight analyses discussed earlier, which identified the precise areas surrounding the fusiform gyrus that demonstrated increased probabilities of the dogs category across the DecNef sessions. Additionally, we assessed the activation levels within each individual’s fusiform masks employed during DecNef training. For each participant, we calculated the mean activation within these fusiform masks across the four different conditions (control hiragana–dogs target, control hiragana–scissors target, targeted hiragana–dogs target, and targeted hiragana–scissors target). We then analyzed these averages using a repeated measures ANOVA.
Results
First, we report the results of the searchlight analyses carried out on perception as well as on the the cross-domain generalization from perception to imagery data to draw an individual mask involving the fusiform cortex for DecNef training (see Methods). The stimulus class (dogs vs. scissors) could be reliably decoded through the entire ventral visual pathway and beyond during the perceptual scans (see Fig. 2A). The cross-domain searchlight analysis revealed significant decoding clusters in the bilateral fusiform gyrus, calcarine cortex, precuneus, supramarginal gyrus, and neighboring regions (Fig. 2B).
A, Searchlight decoding results during perception. B, Cross-domain searchlight decoding results from perception to mental imagery. The average AUC across participants is illustrated in MNI space.
Then, we assessed the participants’ ability during DecNef training to induce brain activity patterns in the target bilateral fusiform ROIs corresponding to a living world referent, which unbeknownst to them referred to the dogs pictures. A LMM analysis demonstrated significant learning effects across DecNef sessions regarding the induction of the target pattern in the fusiform cortex (Fig. 3) [β = 7.833 × 10−05, SE = 3.281 × 10−05, 95% CI lower = 1.338 × 10−05, 95% CI upper = 1.428 × 10−04 (units: proportion points), p = 0.019]. This and subsequent analyses were repeated by excluding the three subjects who did not have a bilateral fusiform ROI, and the main experimental results were replicated (see “Control analyses” section for more details).
Learning effects throughout DecNef training. The figure represents the average decoding probability of dogs category in the bilateral fusiform at the run level across participants. *p = 0.019. Error bars represent 95% CIs, calculated using the Cousineau–Morey method (Cousineau et al., 2005; Cousineau and O’Brien, 2014).
We also investigated whether brain regions beyond the bilateral fusiform played a role in the induction of the relevant target category during DecNef. We performed an off-line searchlight decoding analysis using the entire brain dataset collected during the DecNef sessions. The findings revealed the involvement of additional brain regions in inducing the target pattern, including the early visual cortex (specifically, the lingual and calcarine sulcus), the precuneus, and the posterior cingulate gyrus (see Fig. 4). These brain regions overlap with the areas showing better cross-domain searchlight generalization from perception to imagery during the construction of the classifier.
A, Searchlight maps in MNI space where the probability of inducing the target category (i.e., dogs) across DecNef sessions exceeded the nominal 0.5 chance level. B, The heatmap illustrates the intersection of ROIs targeted across participants, calculated in MNI space.
Visual search task: behavioral results
To assess the impact of DecNef training on the visual search task, we conducted a LMM model on the search RTs of the correct responses (see Methods). On average, 2.53% of trials were removed across participants due to response errors (SD
Results of the behavioral assessment in the visual search task. A, DecNef group: visual search RTs as a function of the hiragana and search target factors. B, Search RTs based on the search target across DecNef and control groups. Error bars represent 95% CIs, calculated using the Cousineau–Morey method (Cousineau et al., 2005; Cousineau and O’Brien, 2014). **p < 0.005, ***p < 0.001.
The slower search RTs for dogs items in DecNef participants are surprising, since previous research has demonstrated the faster recognition of living categories compared to non-living counterparts (New et al., 2007; Wang et al., 2015; He and Cheung, 2019; Moon et al., 2022). This prior evidence provides a critical baseline to assess and interpret the behavioral effects of DecNef on the visual search task. In line with this, a control group (N = 36) that did not undergo DecNef but performed the same task under matching conditions (see Methods) showed the opposite search bias. Namely, faster search for living (i.e., dogs) compared to non-living targets (i.e., scissors). This dissociation indicates that the effect of DecNef is selective to the target class.
Data analyses for the control group followed the same pipeline. RTs of correct search responses were (mean error
We also analyzed the search error rate in the DecNef group by means of a repeated measures ANOVA in Jeffreys's Amazing Statistics Program (JASP, JASP Team, 2023) with hiragana and search target as factors. No significant effects of DecNef were observed on the error rate of the search responses, namely for the hiragana [F(1, 17) = 0.029, p = 0.868;
Lastly, we run a correlation analysis between the individual mean probability of inducing the target category during DecNef training (Amano et al., 2016; Cortese et al., 2016) and the individual effect of DecNef on visual search performance (RT for dog targets—RT for scissor targets). We run three different analyses: (1) a correlation involving the mean probability across all DecNef training runs (Pearson: r = 0.004, p = 0.984; Spearman: ρ = 0.091, p = 0.717); (2) a correlation utilizing the mean likelihood from the last two sessions (Pearson: r = −0.079, p = 0.753; Spearman: ρ = −0.077, p = 0.760); (3) a correlation based on the mean likelihood from the final session (Pearson: r = −0.086, p = 0.7344; Spearman: ρ = −0.007, p = 0.977). All these analyses resulted in non-significant correlations.
Visual search task: fMRI results
Univariate analysis
Potential changes in brain activity driven by DecNef training were assessed through univariate analyses of the fMRI data acquired while participants performed the visual search task. It is well established that perceptual expectations or implicit memory templates have a strong effect on neural responses during perceptual decisions and visual search tasks (Soto et al., 2007, 2008; Summerfield et al., 2008). These effects have been linked to the phenomenon of neural repetition suppression, whereby neural responses decrease following the presentation of an expected item or following the presentation of an item that matches an implicit memory trace. Accordingly, based on prior studies assessing the interplay between memory and visual selection (Soto et al., 2007, 2012), here we hypothesized that if the hiragana was incepted with the perceptual representations of the dogs class, then a neural repetition suppression response would be triggered by the subsequent presentation of the dogs image in the search task, relative to when a neutral (non-incepted) hiragana preceded the search display. Nonetheless, we did not observe any statistically significant clusters of activity that aligned with our hypothesis, both in terms of whole-brain analysis and within a bilateral fusiform region ROI in MNI space, as determined by the off-line searchlight analyses of DecNef training data. Furthermore, we computed the average activity in the bilateral fusiform ROI for each participant in native space, and performed a repeated measures ANOVA with the following factors: hiragana (targeted vs. control) and search target (dogs vs. scissors). There were no significant main effects of the factors associated with the hiragana [F(1, 17) = 0.181, p = 0.676,
Multivariate analysis
We further conducted a decoding analysis on the visual search fMRI data. We employed the previously trained perceptual classifiers to predict the probability of the dogs class for each individual trial in the search task. This process involved calculating the probability of the volume aligning with the corresponding hemodynamic peak during each presentation of hiragana, specifically within the temporal range of 4–6 s after the hiraganas’ onset. The fundamental premise underlying this analysis is that successful conditioning through DecNef should enhance the probability of categorizing a trial as dogs when presented with the targeted hiragana cue, as opposed to the control hiragana cue. To quantify this, we computed the average probability of dogs across runs per participant, and conducted a paired t-test to compare these probabilities between the targeted and control hiragana conditions. We observed no significant differences between the targeted hiragana (mean = 0.769, SE = 0.011) and the control hiragana (mean = 0.773 SE = 0.012) [t(17) = 1.082, p = 0.294].
Principal component analysis
So far, the results indicate that DecNef training did not result in associative knowledge between the hiragana and the perceptual content represented in the decoded multivoxel activity pattern in the fusiform. However, the behavioral results indicate that the representation of the target concept changed following DecNef training. We then further examined representational changes induced by DecNef training by means of a principal component analysis (PCA) of the fusiform activity patterns across DecNef runs (Tipping and Bishop, 1999). This approach has been previously used to characterize the brain representation of both sensory (Mazor and Laurent, 2005; Deitch et al., 2021) and motor information (Churchland et al., 2010; Ahrens et al., 2012) in cortex.
The analysis was performed in Scikit-learn (Pedregosa et al., 2011) on an individual basis. Specifically, principal components (PCs) were identified from the dogs perceptual samples used to train the decoder. Prior to performing the PCA, trial examples were standardized by subtracting the mean and dividing by the standard deviation. Subsequently, the DecNef data were projected onto the space identified by the PCs. Then, a linear support vector regression (SVR) was used to test for linear trends in the representation defined by the PCs across the runs of DecNef training for each participant. The linear SVR used the PCs of the DecNef data to predict the indices of the DecNef training runs. The SVR was cross-validated using a 100-fold cross-validation procedure before deriving the average variance explained across folds. We run the analysis with 25 PCs based on prior analysis of DecNef induction dataset (Shibata et al., 2019) and also with 10 and 3 PCs in order to estimate the robustness of the results.
The results of the PCA/SVR analysis in the fusiform ROI targeted by DecNef showed significant changes in the representation of the PCs across DecNef runs. This result was consistent across different analyses performed with different numbers of PCs [25 PCs: t(17) = 2.953, p = 0.004; 10 PCs: t(17) = 2.34, p = 0.016; 3 PCs: t(17) = 1.758, p = 0.048].
The same PCA/SVR analysis was performed in four control ROIs. The inferior frontal and inferior parietal ROIs were selected as control regions because their activity patterns did not significantly predict the target class in the searchlight analysis (Fig. 4). We also included a second pair of control regions from the searchlight analysis, namely, posterior cingulate and precuneus, that showed above chance probability of decoding the target class.
We directly compared the variance explained in the fusiform and the control ROIs by means of 5 (ROI) × 3 (number of PCs) repeated measures ANOVA using Greenhouse–Geisser for the sphericity correction correction. There was a main effect of the ROI factor [F(4, 68) = 4.235, p < 0.025,
Variance explained by the linear SVR analysis as a function of ROI and number of PCs. The ROIs include fusiform, inferior frontal gyrus (IFG), inferior parietal lobe (IPL), paracingulate gyrus (PCG), and precuneus (PCUN). Error bars represent 95% CIs, calculated using the Cousineau–Morey method (Cousineau et al., 2005; Cousineau and O’Brien, 2014). *p < 0.05, **p < 0.005.
Control analyses
We repeated a new set of LMMs analyses after excluding the three participants who did not specifically engage the bilateral fusiform. Our analysis revealed significant learning effects across DecNef sessions in terms of inducing the target patterns in the fusiform cortex (β = 8.150 × 10−05, SE = 3.750 × 10−05, 95% CI lower = 9.458 × 10−06, 95% CI upper = 0.000 (units: proportion points), p = 0.034). In the visual search task, we replicated too the main experimental results. The DecNef group was significantly slower for dogs targets compared to scissors targets [β = 20.392, SE = 6.238; 95% CI lower = 8.168, 95% CI upper = 32.617 (units: ms), p = 0.001], but not significant main effects on RTs were found for the hiragana [β = 1.702, SE = 6.236; 95% CI lower = −10.519, 95% CI upper = 13.923 (units: ms), p = 0.785] nor interaction effects between hiragana and search target factors [β = −9.435, SE = 8.833; 95% CI lower = −26.746, 95% CI upper = 7.874 (units: ms), p = 0.285]. Upon comparing the experimental groups, we observed that there were no statistically significant differences in RTs between the DecNef and control groups [β = 58.955, SE = 30.996; 95% CI lower = −1.725, 95% CI upper = 119.638 (units: ms), p = 0.063]. However, we found an interaction effect of search target and group factors [β = 23.104, SE = 4.840; 95% CI lower = 13.617, 95% CI upper = 32.591 (units: ms), p < 0.001]. Analysis of the simple effects revealed that control group participants respond faster to images of dogs than to scissors [contrast: scissors–dogs, β = 7.42, SE = 2.61 (units: ms), p = 0.004]. In contrast, those in the DecNef group exhibited the reverse trend [contrast: scissors–dogs, β = −15.69, SE = 4.08 (units: ms), p < 0.001]. Moreover, the DecNef participants had slower RTs for dogs when compared to the control group [contrast: control–DecNef, β = −82.1, SE = 31 (units: ms), p = 0.008]. However, there was no significant difference in RTs for scissors between the two groups [contrast: control–DecNef, β = −59.0, SE = 31 (units: ms), p = 0.057]. There were no significant differences in search accuracy between the DecNef and control groups. The univariate and multivariate fMRI analyses also revealed no significant activity differences as a function of the different trial types with the targeted vs. neutral hiragana.
Participants’ subjective reports
Participants were queried about the techniques they employed to manipulate the size of the feedback disk during DecNef training. Their responses included various strategies like mental humming, mathematical calculations, visualizing different scenarios, evoking emotions, and more. Many participants indicated that they needed to adapt and employ a range of strategies to maintain their performance over time. They expressed moderate to high levels of effort in enlarging the feedback disk, but the majority believed they had gained a high degree of control throughout DecNef training. Moreover, participants’ reports indicate that they were unaware of the specific research goals. Out of 18 participants, 12 participants guessed that the target category was dogs, but a Chi-squared test showed the observed frequencies were consistent with a uniform distribution and therefore unreliable identification of the target class [χ2(1) = 2, p = 0.157]. Confidence reports indicated low confidence levels in their guesses, even among those who guessed correctly. Tables with details regarding the individual strategies and participants beliefs about experimental goals are available at https://pedromargolles.github.io/pyDecNef/.
Discussion
This study aimed to covertly manipulate conceptual representations using fMRI-DecNef, thereby introducing a novel perceptual meaning into a neutral hiragana without participants’ awareness. Participants progressively improved in their ability to induce neural patterns that were similar to the target concept (i.e., dogs). This observation is noteworthy because previous DecNef studies that used a similar amount of training sessions as here (Amano et al., 2016; Cortese et al., 2016; Koizumi et al., 2016; Taschereau-Dumouchel et al., 2018; Knotts et al., 2019) did not reliably observe significant learning effects, despite observing DecNef effects on behavioral performance. Here we used an improved decoder to provide a better calibrated and more fine-grained feedback to participants (Margolles et al., 2023). This improvement contrasts with the logistic regression classifiers used in previous DecNef studies, which tended to produce overly confident and highly binomial predictions as feedback (Bai et al., 2021). These improvements in the classifier could enhance participants’ capacity to identify subtle neural changes through feedback, potentially resulting in improved self-regulation during DecNef training. These improvements represent a step forward toward refining the precision of DecNef. However, there were inter-individual differences in learning during DecNef, as commonly observed in neurofeedback studies, due to differences in attention, motivation (Kadosh and Staunton, 2019), or metacognitive abilities (Muñoz-Moldes and Cleeremans, 2020; Lubianiker et al., 2022; Stirner et al., 2022). Future DecNef studies would benefit from developing personalized training protocols to enhance learning within each individual.
Despite the consistent learning effect detected during the DecNef training, which resulted in strong non-conscious neural representations of the target perceptual content, the behavioral results do not support the view that the hiragana was associated with the target concept. Search performance for the target category (i.e., dogs) was similar when search was primed by the hiragana targeted by DecNef, or by a control hiragana. Previous studies provided robust evidence that memory effects on visual search performance can occur when the number of items in the search display is low, namely, even with two items (Soto et al., 2007, 2010, 2012; Pan et al., 2012). Thereby, it is unlikely that the absence of effects of the incepted hiragana were due to the small set size of the search display. Furthermore, both univariate and multivariate fMRI analyses in the search task did not show any neural modulation driven by the matching between the targeted hiragana and the target category. While some prior studies indicated that associative learning can operate on visually masked and subliminal representations (Pessiglione et al., 2008; Seitz et al., 2009; Raio et al., 2012), mounting evidence has indicated otherwise (Shanks and John, 1994; Field, 2000; Mertens and Engelhard, 2020; Skora et al., 2021). Prior research supporting unconscious associative learning faced methodological criticisms, particularly regarding experiment demand characteristics and limitations in assessing participant awareness (Shanks and Dickinson, 1990; Field, 2000). Our findings indicate that while robust neural representations of the target perceptual content are induced, forming associative knowledge between these patterns and a neutral stimulus likely requires conscious access to the represented content. However, this does not necessarily exclude the other forms of associative learning, namely, involving emotional content and fear conditioning (LeDoux and Pine, 2016) may occur outside participants’ awareness (Lovibond and Shanks, 2002).
A previous study found evidence of orientation-color associations in early visual cortex induced by DecNef (Amano et al., 2016). However, our current study investigated higher-order associative learning involving an abstract symbol and a perceptual class, compared to the lower-order associations observed in the previous study. More recently, Collin et al. (2022) showed that contextual representations of faces or houses could be induced through DecNef in the visual cortex, while participants memorized a different set of objects. The implicit neural context induced by DecNef (i.e., face or house) facilitated associative memories between the study items and the specific encoding contexts. However, in Collin and colleagues’ study, participants were explicitly instructed to think about either faces or houses during DecNef training, thereby likely implicating awareness of the association between the feedback and the mental context (i.e., whether the feedback related to faces or houses).
Notably, DecNef training significantly impacted the representation of the target category. In contrast to the control group, the DecNef group exhibited impaired selection of the target category during the search. While the control group showed a visual search advantage for living (i.e., dogs) over non-living targets (i.e., scissors), consistent with prior research (e.g., New et al., 2007; Wang et al., 2015; He and Cheung, 2019; Moon et al., 2022), the DecNef group displayed the opposite pattern of results. Specifically, they showed slower search performance for dogs compared to scissors. This dissociation suggests that DecNef’s influence is specific to the targeted category. We propose that DecNef training can lead to a reconfiguration of the neural template associated with the target category, which in turn affects search behavior. Importantly, previous studies demonstrated that visual imagery, working memory, and longer-term memory contents have positive influences in the guidance of attention, enhancing selection, and search performance for visual targets that match the contents held in memory (Soto et al., 2008; Tartaglia et al., 2009; Reinhart et al., 2015; Nobre and Stokes, 2019). For instance, it has been shown that explicit imagery training of a visual feature facilitates its subsequent selection in a visual search task (Tartaglia et al., 2009; Reinhart et al., 2015; Nobre and Stokes, 2019). Therefore, the impairment in search performance observed for the target representation of DecNef (i.e., dogs) is not predicted by an account based on participants consciously rehearsing the target representations during DecNef.
One possible explanation is that DecNef induced representational drift of the target concept, causing a shift in alignment of neural activity patterns within a high-dimensional space over time (Rule et al., 2019). DecNef protocols require participants to explore a broad search space to find the target representation by means of diverse cognitive strategies (Watanabe et al., 2017). These strategies are conceptually distinct from the target and are also accompanied by various sources of interference, such as the presence of the hiragana in this particular case. Hence, the brain is confronted with a many-to-one mapping problem often encountered in machine learning, where a unique target can potentially be mapped from multiple patterns in multidimensional space. The repeated induction of noisy neural patterns that partially overlap with the neural representation of the target class can consequently result in the corruption of the target representation through Hebbian learning and operant conditioning (McClelland et al., 1999). The PCA analyses demonstrated that the fusiform activity patterns underwent changes in representation throughout the DecNef training runs compared with several control regions. This result provides further evidence that effects of DecNef training were more pronounced and localized within the specific targeted region, in keeping with a targeted neural plasticity model of DecNef effects (Shibata et al., 2019).
Representational drift has been observed in the hippocampus (Ziv et al., 2013) and also in cortical regions (Driscoll et al., 2017), including sensory cortex (Deitch et al., 2021; Aitken et al., 2022), when observers are exposed to similar stimuli across sessions and days. Representational drift is higher for more complex representations associated with naturalistic stimuli relative to simple items (i.e., oriented gratings) (Marks and Goard, 2021). Importantly, representational drift can occur despite sustained decoding accuracy of the target content from neural responses by a linear classifier (Aitken et al., 2022), as we observed in the present study.
Previous theoretical work proposed that representational drift may afford computational benefits, namely, for continual learning (Driscoll et al., 2022). However, the causal effects of representational drift on behavior remain to be elucidated (Sadeh and Clopath, 2022). The current findings suggest that representational drift could potentially result in undesired behavioral outcomes. Specifically, representational drift may lead to a mismatch between internal representations held in memory and the perceptual input from the external environment, thereby influencing visual selection and search behavior.
In summary, this study elaborates on the impact of conscious awareness on different types of learning. It indicates that higher-order associative learning is more likely to depend on conscious awareness, whereas lower-level forms of re-learning, modification, or plasticity in established representations can occur without awareness, potentially through a mechanism involving neural representational drift. Additionally, this research offers novel insights for the advancement of DecNef protocols, specifically aimed at manipulating conceptual representations. These findings pave the way for further research into the potential of DecNef in personalized interventions, where the guidance provided can be used to target and improve maladaptive mental representations.
Data Availability
The scripts and documentation corresponding to the Python library used for decoded neurofeedback (PyDecNef) can be found at https://pedromargolles.github.io/pyDecNef/ (Margolles et al., 2023). The experimental data can be found in the Open Science Framework (OSF) platform at https://osf.io/54j97/.
Footnotes
P.M. acknowledges support from an Formación de Personal Investigador (FPI) Grant from the Spanish Ministry of Economy and Competitiveness (ref. BES-2017-080796). P.E. acknowledges support from the Basque Government Predoctoral Grant. D.S. acknowledges support from the Basque Government through the Basque Excellence Research Centers (BERC) 2022–2025 program, from the Spanish Ministry of Economy and Competitiveness, through the “Severo Ochoa” Programme for Centres/Units of Excellence (CEX2020-001010-S), and also from project grants PID2019-105494GB-I00. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The authors declare no competing financial interests.
- Correspondence should be addressed to Pedro Margolles at pedromargolles{at}gmail.com or David Soto at d.soto{at}bcbl.eu.
