Decoding of Working Memory Contents in Auditory Cortex Is Not Distractor-Resistant

Stimuli and procedure

The stimuli were complex sounds consisting of three harmonics with varying spectral frequencies. This was true for the stimuli in the perception experiment and for both target and distractor sounds in the working memory experiment. Specifically, the stimuli were composed of a combination of three band-passed noises (a fundamental frequency and two harmonics, each bandpass filtered to a bandwidth of 1/10 octave). The fundamental frequency was chosen from 4 stimulus bins with two bins from the small octave (Cis and A) and two bins from the two-line octave (Cis′′ and A′′). The centers of the stimulus bins were 138.591, 220, 554.365, and 880 Hz, respectively. To prevent encoding into long-term memory, the precise frequency of the stimuli varied slightly around the central frequency of the 4 stimulus bins. Specifically, the frequency randomly ranged from a semitone below to a semitone above the bin center in 5 equidistant, logarithmic steps. For example, the exact stimulus frequencies of the Cis bin were as follows: 130.81, 134.65, 138.59, 142.65, and 146.83 Hz. Thus, 20 (4 stimulus bins × 5 steps) unique frequencies characterized the stimuli in the perception experiment, as well as the target and distractor sounds in the working memory experiment.

All auditory stimuli were processed with an external soundcard (Fireface UC, 192 kHz sampling rate, RME) and presented at a comfortable intensity via MRI-compatible, active noise cancellation headphones (OptoActive, Optoacoustics). Stimulus construction and timing were controlled with MATLAB R2012b (The MathWorks) and the Psychophysics Toolbox (Brainard, 1997). Visual instructions were projected onto a screen outside the scanner. The participants could see the screen through a tilted mirror in a dimly lit room. Participants were instructed to maintain central fixation throughout the experimental runs.

Perception experiment

In each trial, participants had to detect volume reductions within a sequence of pulsating sounds with a fixed frequency composition (see Stimuli and procedure) for 6 s. Each stimulus consisted of 10 pulses of 0.3 s duration with increasing and decreasing volume during the first and last 50 ms, followed by a 0.3 s silent period. The volume level of a standard pulse in each trial was constant, but 1, 2, or 3 target pulses could be reduced in volume. The number and temporal position of target pulses were randomly selected. To ensure attentive stimulus engagement, participants had to respond to volume changes by pressing a button. A response was considered correct if the participant responded within 1 s after the onset of a target pulse. After a 0.5 s delay, performance feedback was provided for 0.5 s. The order and the color of the dots provided feedback to the subjects about their performance on each of the volume reductions. The leftmost dot informed about the first volume change, the neighboring one about the second of up to three volume changes, etc. Green and red dots symbolized hits and misses, respectively. False alarm and correct rejection performance was not fed back to the participants. Trials were separated by an intertrial interval of 2, 4, or 7 s.

We chose a Bayesian psychometric staircase procedure to adapt the difficulty of detecting the reduced volume level (Watson and Pelli, 1983). For every subject, Quest estimated the maximum likelihood of a volume reduction threshold corresponding to a specified performance criterion. We set the target hit rate at 75% to keep the effort constant between participants and to obtain a challenging but motivating task. The prior threshold estimate (prior SD) was set at 0.5, the steepness of the Weibull function (β) was 3.5, the false alarm rate (γ) was set at 0, and the lapse rate (δ) at 0.05. The Quest algorithm takes the response pattern of all previous trials into account and updates the volume reduction threshold on a trial-by-trial basis. The staircase procedure was initialized at the first fMRI session and ran continuously for every participant. The average hit rate across all trials was 76.82% (SD = 4.24%), which did not differ significantly from the targeted 75% (t₍₁₅₎ = 1.72, p = 0.11).

Every fMRI session included one run of the perception experiment with 60 trials. We counterbalanced and randomized the stimulus bins, the stimulus steps, and the number of volume reductions within runs. This procedure resulted in 60 unique trials, each occurring once per run.

Working memory experiment

Participants had to memorize the pitch of a single 0.3 s target sound in anticipation of an upcoming two-alternative forced-choice task (Fig. 1a). After a delay period of 12.8 s, a probe sound was presented for 0.3 s. The fundamental frequency of the probe was manipulated in four steps of ±0.007, ±0.015, ±0.025, or ±0.05 log10(Hz) relative to the frequency of the to-be-probed target sound. The sign and the exact difference between target and probe sound were chosen randomly. Participants had to indicate if the frequency of the probe sound was higher or lower than the target sound by rotating a trackball to the right or to the left, respectively. The duration of the response period was 2.9 s. Afterward, participants received visual feedback for 0.5 s via a colored dot located to the right of the fixation circle. Green and red dots indicated correct and incorrect responses, respectively. Trials were separated by an intertrial interval of 2, 4, or 7 s.

Crucially, we manipulated whether distractors were presented during the delay period or not. When distractor sounds were presented, either participants could ignore them (passive distraction) or they had to detect possible variations in their volume level (active distraction). The no distraction condition was characterized by a blank delay period without distractor sounds. In the passive distraction condition, participants were instructed to ignore the pulsating distractor sound presented during the whole delay period (for a similar approach, see Deutsch, 1970). The distractor sound was similar to the stimulus in the perception experiment described above; that is, a pulsating sound sequence with a fixed frequency composition was presented for 10.8 s starting 1 s after the target presentation. The fundamental frequencies of the distractor and target sounds were drawn from the stimulus pool described above (see Stimuli and procedure) and combined in an orthogonal way. Every combination of target and distractor frequency bin was presented once during each run, but every combination of target and distractor sound frequency described above could possibly occur throughout the experiment. In the active distraction condition, the physically identical pulsating distractor sounds were presented as in the passive condition; however, participants were instructed to detect pulses with a reduced volume level. To ensure that participants actively engaged with the distractor throughout this period, they had to respond to 1, 2, or 3 possible volume reductions via button press. Moreover, we implemented the same Quest procedure with the identical initial parameters as for the perception experiment. The staircase procedure of the active distraction task was initialized at the first run and ran continuously across all fMRI sessions. The average hit rate across all trials (mean = 76.9%, SD = 4.12%) was slightly higher than the targeted hit rate 75% (t₍₁₅₎ = 1.85, p = 0.084). The small SD indicated constant task difficulty between participants. For this active distraction task, participants received additional feedback analogously to the feedback provided during the perception task. It consisted of up to three dots, arranged horizontally to the left of the fixation circle with green and red dots indicating detected and missed volume changes, respectively.

In every fMRI session, participants completed six experimental runs of the working memory experiment comprising 16 trials per run. As all trials within a run were from the same condition, participants completed two runs each of the no distraction, passive distraction, and active distraction conditions per fMRI session. The serial position of the conditions was counterbalanced such that each condition had to occur once during the first three and last three runs of the session. The stimulus bins of the target sound and the distractor sound were counterbalanced within runs. Therefore, each of the resulting 16 target-distractor combinations occurred once per run. The stimulus steps and the ITI durations were counterbalanced and randomized within a session.

Data acquisition

We recorded structural and functional MRI data using a 3-Tesla Magnetom Prisma MR scanner (Siemens), located at the Brain Imaging Center Frankfurt with a 64-channel head coil. We collected functional scans during each run of the perception and working memory experiment and one structural image per session. Functional data were whole-brain EPI (72 slices, 2 × 2 × 2 mm resolution, 0 mm gap, FOV = 20.8 cm, TR = 1 s, TE = 35 ms, flip angle = 56°). Furthermore, we recorded 5 EPI volumes before each experimental run with the same parameters but reversed phase encoding direction to correct for distortions. Structural data were high-resolution T1-weighted images (176 sagittal slices, 1 × 1 × 1 mm resolution, FOV = 25.6 × 25.6 cm, TR = 2.8 s, TE = 2.12 ms).

MRI preprocessing

All imaging data were preprocessed using FMRIB Software Library (FSL) (https://fsl.fmrib.ox.ac.uk) and SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12). Functional images within a run were motion-corrected via FSL MCFlirt with a final sinc interpolation stage. We then applied FSL's topup procedure to correct for distortions using the reversed phase-encoding volumes collected just before each functional run. Next, realignment was performed in two steps using SPM. First, we aligned all functional data with the first volume of the functional run that followed directly on the structural scan. Second, all runs were realigned onto a functional mean image. Finally, the anatomic image was coregistered to the functional mean image.

Anatomical ROIs

Based on our recent work (Czoschke et al., 2021) and previous literature (Linke et al., 2011; Kumar et al., 2016; Uluç et al., 2018), we defined the auditory cortex as our main anatomic ROI comprising Heschl's gyrus, planum temporale, and the posterior division of the superior temporal gyrus. To investigate the auditory cortex ROI in greater detail, we also defined Heschl's gyrus, planum temporale, and the posterior division of the superior temporal gyrus as separate ROIs. In an exploratory analysis, we also tested the possibility that distractor-resistant information was stored in higher-order regions known to belong to a putative ventral processing stream including a temporal pole (TP) ROI (Arnott et al., 2005) and an inferior frontal gyrus (IFG) ROI (Czoschke et al., 2021). The TP ROI comprised the anterior divisions of the superior temporal and middle temporal gyrus and the TP. The IFG ROI included the pars triangularis and pars opercularis of the IFG. We also included the superior parietal lobule (SPL) ROI that includes IPS, as this region was suggested to be involved in working memory processing, including auditory working memory (Czoschke et al., 2021). It has been reported repeatedly to represent distractor-resistant visual working memory information (Bettencourt and Xu, 2016; Lorenc et al., 2018; Rademaker et al., 2019).

Anatomical probability maps were taken from the FSL Harvard-Oxford cortical and subcortical structural atlases (Desikan et al., 2006). We transformed anatomic ROIs from standard into native space using SPM segmentation and normalization functions. Specifically, segmentation was applied to generate a forward transformation matrix containing information to warp anatomic images from native space to standard space. Using the inverse of the matrix, we transformed ROIs from standard to native space.

BOLD signal time course

For data visualization, Figure 1c shows blood-oxygen-level-dependent (BOLD) signal time courses time-locked to the onset of stimulation both for the perception and working memory experiments in the auditory cortex ROI. The BOLD signal was calculated as the percent signal change relative to the average of the two TRs preceding each trial. For every participant, we selected a number of voxels taken from a nested cross-validation procedure based on data from the classifiers trained on perception data (see Multivariate analyses) and averaged the BOLD signal time courses across the selected voxels from Trials 2-16 of each run separately for each distraction condition.

Univariate analyses: voxel selection

Voxels responsive to auditory stimulation were selected with a univariate approach in SPM12. First, we set up a GLM, including all runs in the perception experiment. The GLM included one regressor for the stimulation period of all trials, six motion regressors per run, and constant regressors to model the run means. Next, the stimulation period regressor was convolved with a canonical HRF. Then, we contrasted the stimulus period regressor with the implicit baseline and multiplied the resulting whole-brain t maps with each binary ROI map to receive separate t maps for every ROI and participant. After that, specific voxel sets were selected based on the highest positive t values from the ROI- and participant-specific t maps reflecting the n most auditory-responsive voxels. Based on this procedure, we created 30 sets of n voxels for every ROI and participant, with n varying from 100 to a maximum of 3000 in steps of 100 voxels. The maximum number of voxels in a ROI was determined based on the participant with the lowest number of auditory responsive voxels in that ROI. Finally, multivariate analyses were computed for every voxel set separately before entering a nested cross-validation scheme (see Multivariate analyses).

Multivariate analyses

Multivariate analyses were performed on trial-specific t maps derived from functional data and recorded during runs in the perception and working memory experiment. To create these maps, we set up finite impulse response (FIR) models for every run and participant, including a 1/128 Hz high-pass filter and a first-order autoregressive error structure, using SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12). All models included six motion regressors and a constant regressor to model the run mean. For the perception experiment, we computed t maps in two steps. First, we set up first-level FIR models with nine regressors per trial to model the 9 s from stimulation onset. BOLD signals in each voxel were then fitted with this set of FIR regressors separately for every run. Then, in a second step, we computed trial-specific t maps (i.e., perception t maps) by contrasting trial-specific regressors of FIR bins from 3 to 8 s after stimulation onset against the implicit baseline (i.e., the t maps contained trial-specific data averaged across the stimulation-related TRs). We deliberately chose this time window to account for the hemodynamic delay.

Runs in the working memory experiment were processed in two steps. First, we set up first-level FIR models using 21 regressors to model the signal of 21 time points from target presentation onward. BOLD activity in each voxel was then fitted with this set of FIR regressors separately for every run. Second, trial-specific t maps (i.e., working memory t maps) were computed by contrasting trial-specific regressors of 8-14 s after target presentation against the implicit baseline to cover working memory retention-related activity that was not contaminated by stimulus encoding and retrieval. Again, the t maps contained voxel- and trial-specific data but averaged across the delay-related TRs.

The t maps from the perception and working memory experiment formed the input for ROI-based multivariate pattern analyses. For this purpose, we grouped trials with target sounds taken from the Cis′ and A′ bins (high octave) and stimuli taken from the lower octave Cis and A bins (low octave). Then, we implemented support vector machines to classify whether participants memorized a high or low octave stimulus in a given trial. To ask whether working memory representations are stored in a format similar to perceptual coding, we trained classifiers on the independent perception t maps and tested them on the working memory t maps. This procedure followed previous work on the effects of sensory distractors on working memory representations (Rademaker et al., 2019). A somewhat similar approach was taken in a recent auditory working memory study where classifiers were trained on the probe phase and tested on the maintenance phase (Lim et al., 2022). We repeated the decoding procedure for every participant, ROI, distraction condition, and the up to 30 voxel counts (described above, see Univariate analyses: voxel selection). The resulting classification accuracies were then entered into a nested cross-validation procedure to select optimal ROI sizes and avoid overfitting at the same time (Christophel et al., 2018).

Previous studies have found that content-specific memory information can also be decoded from activity patterns in the visual cortex (e.g., Harrison and Tong, 2009; Serences et al., 2009; Christophel et al., 2012; Emrich et al., 2013; Peters et al., 2015) and in the auditory cortex (Linke et al., 2011; Kumar et al., 2016; Uluç et al., 2018; Czoschke et al., 2021) when the classifiers were not trained on data from a separate perception experiment but trained (and tested) on data the from the delay period in the memory experiment itself using a leave-one-out cross-validation approach. While an analysis approach using perception-trained classifiers captures memory information that is similar to the activity measured in the perception task, the analysis approach that uses memory-delay data itself to train a classifier capitalizes on any signal differentiating remembered sounds during the delay (Iamshchinina et al., 2021). Thus, we trained and tested classifiers on separate data from the working memory experiment by using a leave-one-session-out approach (Czoschke et al., 2021). Specifically, the classifiers were trained on working memory t maps from all but one session and tested on working memory t maps from the remaining session. The procedure was iterated until every session was used for testing. After that, the decoding accuracies of each iteration were averaged. This approach has the advantage that the training and test sets are temporally independent, which reduces the risk of overfitting.

The previous analyses were based on a priori defined ROIs. However, it is possible that distraction-resistant memory information is coded in fMRI activity patterns outside these regions. For this aim, we used a whole-brain searchlight analysis (Kriegeskorte, 2006; Allefeld and Haynes, 2014; Christophel, 2017; Erhart et al., 2021). Specifically, we applied a support vector machine classification procedure, which was trained on perception data and tested on memory data, to a set of voxels within a spherical mask with a radius of 5 voxels (i.e., 515 voxels). The resulting decoding accuracy was then attributed to the central voxel of the sphere, and this procedure was repeated for every voxel in the brain. Decoding accuracy maps for each participant and distraction condition were then normalized using unified segmentation and smoothed with a Gaussian kernel of 4 mm FWHM. Finally, group-level t maps for each condition were calculated using SPM's built-in procedure using voxels with above-chance level decoding accuracy (after correction for the chance level, i.e., 50%).

Statistical analyses

For the analysis of the behavioral data, trials without a valid response were removed, accounting for 0.35% of all trials (no distraction: 0.47%; passive distraction: 0.47%; active distraction: 0.12%). Then, each participant's mean accuracy was calculated separately for the distraction conditions. To test whether distractor presence influenced behavioral accuracy, we computed a repeated-measures ANOVA with the factor distraction condition. Post hoc paired t tests were calculated for significant ANOVA effects to reveal the underlying differences between the individual conditions.

For all fMRI decoding analyses, we tested for above-chance decoding performance by calculating sign permutation tests for every ROI, distraction condition, and decoding approach: the null distribution of group means (10,000 iterations) was generated by subtracting the chance level (0.5) from individual accuracies. Then, we randomly inverted the sign of the resulting differences and computed the group means. Finally, we calculated the proportion of simulated group means equal or larger than the empirical group mean of the accuracies and reported this proportion as the p value. An α level of 0.05 was set with no correction of multiple comparisons across ROIs.

In addition to the permutation analyses, we computed one-tailed Bayesian t tests (Rouder et al., 2009) against chance level decoding of 0.5. We estimated Bayes factors (BF₁₀) for every ROI, distraction condition, and decoding approach. Bayes factors indicate the likelihood of a model with the one-tailed informed alternative hypothesis compared with a model with the null hypothesis given the data. Thus, Bayes factors with BF₁₀ > 1 indicate that a one-tailed informed alternative hypothesis model is more likely than a model with the null hypothesis given the data. Conversely, Bayes factors with BF₁₀ < 1 indicate the opposite. We interpreted BF₁₀ > 3 as evidence for the alternative hypothesis, BF₁₀ < 1/3 as evidence for the null hypothesis, and 1/3 < BF₁₀ < 3 as inconclusive evidence (Kass and Raftery, 1995). All Bayesian t tests were performed in JASP (JASP Team, 2022) with default settings, including a zero-centered Cauchy prior with a width of r = 0.707.

To directly test whether decoding accuracies differed between distraction conditions, we computed repeated-measures ANOVAs with the factor distraction condition. Following up on significant main effects, post hoc paired t tests as well as paired Bayesian t tests were calculated to assess differences between individual conditions. These analyses were performed for every ROI and decoding approach separately.

For the whole-brain searchlight analysis, group-level t maps for each condition were corrected for multiple comparisons using the cluster-size thresholding method (Forman et al., 1995; Goebel et al., 2006) implemented in BrainVoyager. Specifically, for each statistical map, the uncorrected voxel level threshold was set at p < 0.001 (uncorrected) and then submitted to a whole-brain correction criterion based on the estimate of the spatial smoothness of the map and using Monte Carlo simulations for estimating cluster-level false-positive rates. After 1000 iterations, the minimum cluster-size that yielded a cluster-level false-positive rate of 5% was used to threshold the statistical map, thus resulting in a final map with p < 0.05, corrected for multiple comparisons.