Abstract
Selective spatial attention plays a critical role in perception in the daily environment where multiple sensory stimuli exist. Even covertly directing attention to a specific location facilitates the brain's information processing of stimuli at the attended location. Previous behavioral and neurophysiological studies have shown that microsaccades (MSs), tiny involuntary saccadic eye movements, reflect such a process in terms of visual space and can be a marker of spatial attention. However, it is unclear whether auditory spatial attention processes that are supposed to interact with visual attention processes influence MSs and vice versa. Here, we examine the relationship between MS direction and auditory spatial attention during dichotic oddball sound detection tasks with human participants of both sexes. The results showed that the MS direction was generally biased contralateral to the ear to which the oddball sound was presented or that to which sustained auditory attention was directed. The postoddball modulation of MS direction was associated with the behavioral performance of the detection task. The results suggest that the inhibition of stimulus-directed MSs occurs to reduce erroneous orientation of ocular responses during selective detection tasks. We also found that the correlation between MS direction and neural response to the tone originated from the auditory brainstem (frequency-following response). Overall, the present study suggests that MSs can be a marker of auditory spatial attention and that the auditory neural activity fluctuates over time with the states of attention and the oculomotor system, also involving auditory subcortical processes.
Significance Statement
Microsaccades (MSs), tiny involuntary saccadic eye movements, reflect covert visual attention and influence neural activity in the visual pathway depending on the attention state. However, we lack convincing evidence of whether and how MSs reflect auditory spatial attention and/or neural activity along the auditory pathway. Intriguingly, we showed that the MS direction exhibited a systematic stimulus-related change and correlated with auditory brainstem frequency-following response during the dichotic selective attention task. These results suggest that MSs are associated with general spatial attention processes, not restricted to the visual domain, and can be a good tool for accessing fluctuating neural activity that may covary with attention states.
Introduction
Our brain selectively processes sensory input based on its relevance to ongoing cognitive tasks, and such selective processing, often collectively referred to as attention, modulates the neural information processes in sensory modalities [e.g., auditory cortex (Fritz et al., 2007); visual cortex (Gilbert and Sigman, 2007)]. Attention is a dynamic process, and its states vary spontaneously over time for both visual and auditory domains (Esterman et al., 2013; Van Den Brink et al., 2016; Terashima et al., 2021). An objective way to capture fluctuating attention would assist in the understanding of the neural system of attention and track time-varying attention.
Microsaccades (MSs), tiny and involuntary saccadic eye movements, serve as a good candidate for a “window” for evaluating the attentional state. In visual tasks, MS direction correlates with covert attention: central cues indicating the target direction elicit MS toward the cued location (Hafed and Clark, 2002; Engbert and Kliegl, 2003), while peripheral cues elicit an early MS toward and a late MS away from the cued location (Laubrock et al., 2005; Hafed and Ignashchenkova, 2013; Hafed et al., 2021). In a simple detection task of a peripheral visual target without cueing, the short (or long) manual reaction time (RT) is accompanied by MS toward (or away from) the target (Tian et al., 2016). These temporal dynamics of MSs may reflect temporal shifts in attention states and are associated with relevant neural substrates (Hafed et al., 2021). MS bias can also persist over a relatively long period, depending on sustained visual attention (Xue et al., 2020), and can occur even for memorized visual items (Liu et al., 2022). Neurophysiological evidence suggests that the firing rates of visual cortical neurons increase following MS toward attended visual stimuli (Lowet et al., 2018).
In the auditory domain, however, links between attention and MS have been explored far less in phenomenology and neural substrates. The present study attempts to address the following issues. (1) Although general tendencies of cue-oriented MS have been reported during an auditory cueing task (Rolfs et al., 2005), it is unknown whether the MS reflects moment-by-moment fluctuation of attention states during auditory tasks and to what extent the shift of MS direction is linked to behavioral states during auditory attention tasks. (2) In contrast to the visual domain, there is a lack of evidence on how MSs are associated with neural activity in the auditory pathway. Exploration in the auditory domain should provide insights into the significance of MSs in supra- or cross-modal attention processes, as well as the utility of MSs as a biomarker for tracking auditory attention.
To address the two points raised above, we conducted a dichotic selective detection task: participants were asked to attend to the left or right ear during the presentation of a prolonged sequence of standard sounds and were required to detect the oddball sounds presented to the attended side. To observe the fluctuation of attention states over time, we used a sound sequence with a relatively long duration (2–3 min) compared with the previous study that examined the transient effect of the auditory cue on MS (Rolfs et al., 2005). We examined auditory neuronal activities by focusing on the auditory brainstem frequency-following response (FFR). Recent studies including source-level analysis of EEG responses showed that attention affects FFR to continuous speech stimuli (Forte et al., 2017; Etard et al., 2019; Price and Bidelman, 2021), while other studies reported a large individual variation in the effect of attention on the FFR (Lehmann and Schönwiesner, 2014) or even found that attention had no significant effect (Galbraith and Kane, 1993; Varghese et al., 2015). We hypothesized that attention-related brainstem activities, if any, are susceptible to the spontaneous temporal variations of the attention states, which could explain previous contradictory findings. We expected that if an MS property reflects instantaneous attentional states, we might observe attention-related FFR associated with the MS property.
Materials and Methods
Participants
Human adults with normal hearing participated in three experiments; Experiment 1, 19 participants (three males) with ages ranging from 22 to 49 years (mean = 38.53); Experiment 2, 12 participants (five males) with ages ranging from 20 to 48 years (mean = 37.08); and Experiment 3, 20 participants (13 males) with ages ranging from 20 to 28 years (mean = 22.1). The experimental protocols were approved by the Research Ethics Committee of Nippon Telegraph and Telephone Communication Science Laboratories. All listeners gave written informed consent before the experiment.
Apparatus
The eye data were recorded with an Eyelink system (ER Research) at a sampling rate of 1,000 Hz. Visual stimuli were generated with MATLAB (R2016b) and were presented on a monitor with a resolution of 1,920 × 1,080 pixels and a refresh rate of 60 Hz. Participants sat on a chair and put their heads on a chin rest. The distance between the chin rest and the center of the display was ∼70 cm. Auditory stimuli were synthesized with MATLAB at a sampling frequency of 44.1 kHz and were presented binaurally through ER-3A insert earphones (Etymotic Research).
We used the ActiveTwo system (Biosemi) to record EEG signals, from which FFRs were derived. The electrophysiological responses were recorded differentially between Cz and both ear lobes (A1 and A2) with Ag–Ag/Cl active electrodes and sampled at 16,384 Hz. The ActiveTwo system replaces the ground electrodes used in conventional EEG systems with two separate electrodes: common mode sense and driven right leg electrodes. FFRs were measured only in Experiments 1 and 3 (see below).
Stimuli and procedure
In a typical experimental session, the participant's task was to detect oddball sounds embedded in a sequence of standard sounds (complex tones) with alternating fundamental frequencies (F0), presented to the left and right ears (Fig. 1). Table 1 summarizes the conditions of the session tested in the experiments. All three experiments examined the Dich_TargetL and Dich_TargetR conditions (see below), and there were additional tasks (as necessary) depending on the experiment. Experiment 1 included five conditions that differed in stimulus presentation and task: In the Dich_TargetL and Dich_TargetR conditions, the stimuli were presented dichotically, and the participant was asked to maintain attention toward the left or right ear only, respectively, and respond as quickly as possible only to the oddball sounds presented to that ear. In the Dich_Passive condition, although the stimuli were again dichotically presented, participants were asked to listen to the stimuli passively, i.e., not attending to a particular ear and not responding to oddballs. In the Mono_TargetL and Mono_TargetR conditions, the stimuli were presented to the left or right ear only, respectively, and the participant was asked to detect oddballs in that ear (thus, no selection between the ears was required). In one block of five sessions (corresponding to the five conditions), the conditions were ordered pseudorandomly. We conducted 18 blocks per participant in total.
Experimental design. A, Schematic illustration of a dichotic oddball detection task. The participants were asked to detect oddball sounds (white noise) embedded in a sequence of standard sounds (complex tones) with alternating fundamental frequency (F0), presented to the left and right ears. This figure illustrates the Dich_TargetL condition, which is only shown as an example. Please see Table 1 for the summary of experimental conditions. We measured behavioral responses by button press and the eye metric information (gaze position and pupil size) using a camera-based eye tracker (Eyelink). In Experiments 1 and 3, we measured EEG to evaluate auditory brainstem FFRs in addition to the eye metrics (data not shown). B, Example of the time course of MS amplitudes for one participant’s data from Experiment 2 (only showing the Dich_TargetL condition with medium oddball intensity). The gray dots represent individual MS amplitudes. The red line represents the running-averaged value of the amplitudes. MSs are discrete events that occur randomly, and their amplitude fluctuates over time. C, Example of the time course of RT of the oddball detection (the data correspond to the data shown in panel B). Red circles represent the RTs obtained for responses to random-timed oddballs in one session. Red crosses indicate the trials where the participant failed to respond to the target (miss). Blue circles indicate the trials where the participant responded to the oddball sounds presented to the unattended ear (false alarm). The number of oddball sounds was 64 (32 for each ear) in one session for Experiment 2. RT also fluctuates over time, sometimes reaching ∼1 s, indicating that the participant’s attention state varied during the task.
Summary of tasks for the three experiments
In Experiment 2, the Dich_TargetL, Dich_TargetR, and Dich_TargetBoth conditions were investigated. The former two conditions were the same as in Experiment 1, while we tested three oddball levels (soft, mid, and loud) to examine the effect of the intensity of the oddball sound on MS. We considered that the oddball intensity would be associated with the salience of the oddball, as well as the difficulty of the task. In the Dich_TargetBoth condition, participants were asked to attend to both ears and respond to oddballs presented in either ear. This means that they were not required to facilitate or suppress responses to a particular ear. In one block of nine sessions (corresponding to the nine conditions, i.e., three attention directions times three intensities), the conditions were ordered pseudorandomly. We divided one block into three sub-blocks and ordered the conditions so that each sub-block contained the Dich_TargetL, Dich_TargetR, and Dich_TargetBoth conditions in a pseudorandom order. We conducted six blocks per participant in total.
In Experiment 3, we focused only on the Dich_Target L and Dich_Target R conditions to yield a large number of repetitions for the same condition within a limited time. This experiment was conducted to ensure the reliability of the scalp-recorded subcortical responses, which are susceptible to the number of trials.
In Experiments 1 and 2, standard sounds were harmonic complex tones with F0 s of 315 Hz and 395 Hz, presented to the left and right ears, respectively. In Experiment 3, the F0 s of the complex tones were 375 Hz and 470 Hz, and the presented ears were counterbalanced between participants. The number of harmonic components was five (including F0) for all experiments. The starting phase of the harmonic components was fixed across presentations. We chose these harmonic complex tones because FFR (regarded as originating from the brainstem) can be reliably obtained for F0s up to ∼500 Hz (Hoormann et al., 1992; Tichko and Skoe, 2017) while avoiding confounds caused by a slow cortical following response (Coffey et al., 2016; Tichko and Skoe, 2017). The A-weighted sound pressure level of standard sounds was 78 dB, calibrated with an artificial ear with a coupler by IEC 60318-1. The oddball sound was white noise and presented in place of the standard. The duration of each standard and oddball sound was 40 ms, and the interval between the sounds was 50 ms for Experiments 1 and 2. In Experiment 3, the duration of each sound was 50 ms, and the interval between the sounds was jittered from 50 to 63 ms with uniformly distributed random numbers. In Experiment 1, the A-weighted sound pressure level of the oddball was the same as that of the standard sounds. In Experiment 2, we tested three intensities to vary the detectability of oddballs: 2.5, 5, and 10 dB relative to the detection threshold of the oddballs measured prior to the main experiment for each participant. In Experiment 3, the intensity of oddball sounds was the same as that of the 5 dB condition in Experiment 2. Thus, the saliency of oddball sounds was relatively low compared with Experiment 1. The total number of sounds, including the standards and oddballs, in Experiments 1–3 was 500, 800, and 700, respectively. Of these, the oddballs made up 4, 8, and 2%, respectively.
For all experiments, we presented a visual fixation spot at the center of the display (0.63 cd/m2). We wanted to investigate the relationship between FFR and the baseline pupil size, which may reflect the participant's arousal state, as well as MSs. The pupil size, however, reflects not only arousal states but also the brightness of visual stimuli. Thus, background luminance was manipulated only for Experiment 1 to dissociate the effects of brightness on the baseline pupil size: for half sessions, 10.15 cd/m2, and for the other half, 2.80 cd/m2. For Experiments 2 and 3, the background luminance was 10.15 cd/m2 throughout the experiment.
Detection of MSs
We analyzed the gaze data as the eye-tracker output for both eyes. To detect MSs, we defined the onset as the time when the velocity of gaze movement exceeded a certain threshold. Specifically, we used a threshold that was six times the standard deviation of the velocity in one trial (Engbert and Kliegl, 2003). We excluded large saccades whose amplitude exceeded 1.5° in the visual angle to extract only MSs. In the present study, we only focus on binocular MSs (Engbert and Kliegl, 2003), defined by a temporal overlap between the left-eye and right-eye MSs: If the difference between the onsets of left and right MSs exceeded 10 ms, we excluded those events from the analysis. We extracted only the horizontal component of MSs because we focused on the attention effect, which is expected to be observed in the horizontal direction (i.e., left or right). Raw gaze positions were converted from the pixel values obtained by a five-point grid calibration into degrees of the visual angle using the distance from the center of the display to the participant's eyes. We excluded eye data during blinks and 100 ms before and after the onset and offset of blinks, as well as unsuccessful recordings, from the analysis. These criteria for excluding data are common to all analyses related to eye metrical measures (i.e., gaze or pupil analysis).
Analysis of MSs
We analyzed the peritarget MS rate time course to examine the transient impact of the oddball (i.e., target) presentation on MS. To calculate the time course, the MSs that occurred within the −0.5 s to 2 s time range around each target presentation were found and sorted into those featuring movement toward or away from the direction of the oddball stimuli (i.e., the left or right ears). Over multiple presentations of targets for a condition of interest, the MSs at discrete occurrence times relative to the target were averaged with a moving Gaussian window (σ = 60 ms; duration, 300 ms). We applied a cluster-based permutation test, which has been commonly used for EEG and magnetoencephalographic data (Maris and Oostenveld, 2007), to compare the time courses for MS rate around oddball presentations and evaluate if there were specific time points when MSs occurred toward or away from the stimulus. We also calculated the ratio of rightward MS counts throughout a trial (right MS ratio, #MSr/#MSall) to assess the overall bias of MSs during an attention task.
Analysis of behavior
We evaluated moment-by-moment behavioral performance for target detection in terms of RT. We sought button response events with a time window from 0.15 to 1 s after oddball presentations. For example, if the participant failed to press the button within 1.5 s after an oddball, that trial was excluded from the analysis as a “miss” trial.
Analyses of subcortical activity (FFR)
We first applied a bandpass filter (passband, 70–2,000 Hz) to the raw EEG signal. EEG recordings corresponding to each presentation of a standard sound were extracted using a rectangular window. If the root-mean-squared amplitude of the segmented signal exceeded 35 µV, that extract was excluded from further analyses (Skoe and Kraus, 2010). Then, the amplitude spectrum of the extracted segment was calculated by fast Fourier transformation with a 40 ms Hanning window. Finally, we derived FFRs as the amplitude of the spectral component of the EEG signal at the F0 of the standard sound by averaging the FFR value for all extracted segments (Krizman and Kraus, 2019).
Peri-MS FFR functions
We also examined the relationship between MS and FFR in an MS-by-MS manner. The general procedure is illustrated in Figure 9A. First, for a given MS toward the direction of interest, standard sounds (separately for the left- and right-ear sounds) within 300 ms before and after MS onset were found, and EEG responses to these sounds were fast-Fourier-transformed. We extracted the spectral components as complex numbers only at the stimulus F0 frequency. Both the real and imaginary parts (i.e., amplitude and phase information) were necessary to improve sensitivity to the stimulus-related response by focusing on the phase-locked component relative to the stimulus waveform. Then, we averaged the FFR data before and after the MS onset (pre- and post-MS FFRs). We computed the FFR amplitude not only during the stimulus presentation but also during the prestimulus period (i.e., the silent period between −40 and 0 ms relative to the sound onset) to evaluate the baseline level of FFR for each participant. If the adjacent MS occurred within 500 ms before or after the target MS, that trial was excluded from the analysis because the adjacent MS may affect the FFR time course. After averaging, the data were transformed into absolute values (spectral amplitudes), and the grand average across participants was calculated.
We evaluated the MS-linked variation as the difference in FFR amplitude between post- versus pre-MS and compared the FFR difference between the two cases when the MS direction was congruent and incongruent with the stimulus ear for which FFR was evaluated (referred to as the congruent and incongruent cases, respectively). The data were summarized for all participants and for the Dich_TargetL and Dich_TargetR conditions.
Results
Stimulus-related MS bias: the occurrence of MS toward and away from the oddball stimulus systematically varied after the oddball presentation
Generally, the MS occurrence clearly decreased within 0.5 s after the oddball presentation (Figs. 2A, 3A), suggesting MS inhibition following sudden auditory stimulation (Rolfs et al., 2008; Zhao et al., 2024). We further asked whether the MS rate shows a systematic change after the presentation of oddball sounds dependent on the stimulus direction, as shown in previous visual studies (see Hafed et al., 2021, for a review). To describe the MS bias relative to the oddball direction, we compared the occurrence of MS toward and away from the oddball stimuli (Fig. 2A, blue and red lines), as shown schematically in the top part of Figure 2. MSs were generally biased away from the oddball ∼1 s after the oddball presentations in the Dich_TargetL/R condition, commonly observed in both Experiments 1 and 2 (Fig. 2B). Interestingly, for the Dich_TargetBoth condition in Experiment 2, we found an early MS bias toward the oddball followed by an opposite bias (Fig. 3B, right panel). The horizontal blue and red lines for each figure represent clusters with significantly large amounts of MSs toward and away from the oddball, respectively.
MS rate time courses around the oddball presentation in the Dich_TargetL/R condition. A, MS rate for Experiments 1 (left panel) and 2 (right panel; averages of the three intensity conditions). The detected MSs were averaged using a moving Gaussian window (see Materials and Methods). We focused on the MS occurrence only around the oddball presentation, from −0.5 to 2 s relative to the oddball onset. As shown in the schematic illustration at the top, MSs were sorted into those featuring movement toward or away from the oddball, respectively. We found a drastic decrease in the MS rate within 0.5 s for MSs both toward and away from the oddball. B, The difference in the MS rate between the toward and away from the oddball groups. The red horizontal bars indicate the clusters with significantly large amounts of MSs away from the oddball. The result shows that the MS was biased away from the oddball direction ∼0.5–1 s after the oddball presentation for both Experiments 1 and 2.
MS rate time courses around the oddball presentation in the Dich_Passive (Exp1), Mono_TargetL/R (Exp1), and Dich_TargetBoth (Exp2) conditions. A, MS rate inhibition was observed in all conditions, but it was small in the passive condition in Experiment 1, suggesting the effect of the task engagement and action requirement on the MS rate. B, The difference in the MS rate between toward and away from the oddball. The blue and red horizontal bars indicate the clusters with significantly large amounts of MSs toward and away from the oddball, respectively. We found the early MS bias within 0.5 s toward the oddball only for the Dich_TargetBoth condition, where the participants were not required to perform selective detection between left-ear and right-ear sounds.
Stimulus-related MS bias associated with RT in a dichotic selective detection task
We next asked to what extent the MS reflects the behavioral task performance, as an index of attention states that fluctuate over time. The analysis was inspired by the fact that the MS toward the oddball sounds was observed for the Dich_TargetBoth condition, but not for the Dich_TargetL/R conditions. The Dich_TargetL/R conditions require a selective response to left or right ears, respectively, while a task in the Dich_TargetBoth condition requires simple reactions, i.e., no need to suppress responses to sounds in the nontarget ear. We hypothesized that such a selective reaction process would somehow be reflected in MS bias: The early reflexive MS toward the stimulus (orienting MS; Hafed et al., 2021) would be suppressed during the dichotic selective detection task to reduce the risk of reflexively responding to the oddballs. More specifically, we expected that the orienting MS would occur toward the oddballs when the participant failed to inhibit the orienting ocular response toward the salient sound, even during the Dich_TargetL/R conditions, due to, for example, the participants’ low alertness level at that moment. Under this hypothesis, we focused on RT as an indicator of the alertness level. Generally, the RT was significantly shorter for the Mono_TargetL/R conditions and the Dich_TargetBoth condition (simple reaction tasks) than for the Dich_TargetL/R conditions in Experiments 1 and 2, respectively (Fig. 4A). These results indicate that the Dich_TargetL/R conditions required additional processing load for selective sound detection, compared with the Mono_TargetL/R and Dich_TargetBoth conditions. In Experiment 2, we also found a significant effect of oddball intensity on RTs (RTs were longer and shorter for soft and loud oddballs, respectively).
The effect of RT on MS rate time courses. A, Mean RT for Experiments 1 and 2. RT was significantly longer for the Dich_TargetL/R conditions than for the monoaural condition in Experiment 1 or the Dich_TargetBoth condition in Experiment 2. We also found a significant effect of oddball intensity on RTs (right panel; longer and shorter RTs for soft and loud oddballs, respectively). B, C, MS rate time courses around oddballs for the short- and long-RT trials for Experiments 1 and 2, respectively. We found early MS bias toward the oddball in long-RT trials (i.e., low performance) for the Dich_TargetL/R conditions, not in short-RT trials (i.e., high performance). The temporal pattern of the MS bias function in long-RT trials was similar to that for the Dich_TargetBoth condition (Fig. 3C), suggesting a failure to inhibit the orienting MS.
Does MS bias reflect trial-by-trial variation in RT during the Dich_TargetL/R task? To answer this question, we divided the trials for each condition and participant into halves with RTs shorter and longer than the median (referred to as short- and long-RT trials, respectively). We found an early MS bias toward the oddball sounds in long-RT trials (i.e., low performance) for the Dich_TargetL/R conditions for both Experiments 1 (∼ 0.3 s) and 2 (∼0.5 s), which was not observed in short-RT trials (Fig. 4B,C, bottom panels). In summary, we consistently found that the early MS bias toward the oddballs occurred when task performance was low (i.e., long-RT trials). These results lead us to infer that the orienting of the MS toward the salient stimulus tends to occur when participants' alertness levels are low during a sound detection task and when they engage in a simple reaction task to respond to targets in the dichotic sound sequence, which requires no selective attention between ears.
Relationship between MS and the direction of sustained auditory attention
We first asked whether the MS direction was biased toward or away from the direction of voluntary and sustained auditory attention, regardless of the oddball presentation (see Xue et al., 2020, for the visual domain). The MS bias was quantified by computing the rightward MS ratio, the number of rightward MS divided by the total number of MS for each condition and participant. There was a tendency that the rightward MS ratio would be higher when the participant attended to the left ear (Dich_TargetL) than when they attended to the right ear (Dich_TargetR) in both Experiments 1 and 2 (Fig. 5A; comparison between the blue and red symbols). This relationship between the directions of attention and MS was opposite to the one observed for the visual sustained attention task (Xue et al., 2020).
Effect of sustained attention and gaze position on MS bias. A, Mean rightward MS ratio (the number of rightward MS was divided by the total number of all MSs). For both Experiments 1 and 2, MS tended to be biased away from the target side (e.g., the ratio of rightward MS was greater for the Dich_TargetL condition than for the Dich_TargetR condition). B, Mean gaze position at the time immediately before the MS onsets. A positive value for gaze position indicates a rightward gaze. Note that the overall patterns of the gaze positions were mirror-symmetric to the patterns of the rightward MS ratio. C, The relationship between the baseline gaze position and MS in z-score. Each point corresponds to one MS occurrence. The color of the dots represents the participant. D, The distribution of correlation coefficients for the gaze position and MS amplitude, derived from individual participants. All participants showed a negative correlation, implying that the MS away from the target included the gaze-correcting MS.
As a statistical analysis of Experiment 1, we conducted a two-factor repeated–measures ANOVA for rightward MS ratio in the Dich_TargetL, Dich_TargetR, Mono_TargetL, and Mono_TargetR conditions. The factors were the presentation method (monaural vs dichotic) and target laterality (left- vs right-ear targets). We found a significant effect of laterality (p < 0.00001; F(1,18) = 40.509; left panel in Figure 5A; blue > red symbols), showing the MS bias away from the target direction. We found that the presentation method had no significant effect (p = 0.494). This indicates that the MS bias away from the target direction occurs even when selective attention is not required (Mono_TargetL/R conditions). It is interesting that the rightward MS ratio, when pooled across all the conditions, was generally distributed above 0.5, indicating that the MS direction was biased toward the right, regardless of the target side. Note, however, that the ratio for the Dich_Passive alone, where the listener was not required to pay attention to the stimuli, was not significantly above 0.5 (p = 0.3680).
We conducted the same analysis for Experiment 2, ANOVA for rightward MS ratio in the Dich_TargetL and Dich_TargetR conditions. In this case, the factors were the target laterality (left and right) and sound intensity (soft, middle, and loud). We found that the rightward MS ratio was larger for the Dich_TargetL condition than for the Dich_TargetR condition (p = 0.0020; F(1,11) = 16.303; Fig. 5A, right panel, blue > red symbols), indicating that the MS was biased away from the target direction, a finding consistent with Experiment 1. We found no significant effect of the sound intensity (p = 0.293) and no significant interaction between the two factors (p = 0.619). This time, the rightward MS ratio was generally below 0.5 when pooled across the three conditions (i.e., Dich_TargetL, Dich_TargetR, and Dich_TargetBoth) or when focused on only the Dich_TargetBoth condition (Fig. 5A, black symbols in the right panel; p = 0.0867 or 0.0183, respectively, by one-sample t test). We have no explanation for this discrepancy in the general bias between Experiments 1 and 2.
The gaze shift also reflects the direction of sustained auditory attention and its relation to MS
We next checked whether the spontaneous gaze shift also reflects the sustained auditory attention. We sampled the gaze position data immediately before the MS onsets. Therefore, all gaze data in this analysis were linked with the MS timing. As expected from Gopher (1973), the averaged gaze direction was biased toward the attended direction (Fig. 5B). The overall patterns of the gaze positions (Fig. 5B) appeared to be mirror-symmetric to the patterns of the rightward MS ratios (Fig. 5A). That is, in conditions with higher rightward MS ratios, for example, the gaze tended to be toward the left side. The trial-by-trial correlation between the gaze amplitude and the MS amplitude, including direction information (rightward, positive; leftward, negative) for each participant separately (Fig. 5C), exhibited a consistent tendency, i.e., negative coefficients for all participants (Fig. 5D). This indicates the compensatory relationship between MS and gaze amplitudes and is consistent with the study showing the gaze-correcting saccades (Cornsweet, 1956). We additionally examined the MS bias separately for trials with and without gaze deviation beyond the fixation box (0.3° in size). The MS bias was only observed in trials with gaze deviation, indicating a primarily motoric (i.e., gaze correction) rather than attentional origin of the effect. These results suggest that the MS bias away from the target side during sustained auditory attention may be accounted for, at least in part, by corrective saccades for an error between the fixation point and the gaze position (Costela et al., 2014). In contrast, for the stimulus-driven modulation of MS bias, we observed a bias away from the oddball even when the gaze shifted opposite to the attended direction (Fig. 6B), indicating that such modulation cannot be explained by corrective saccades.
MS rate time courses around the oddball presentation in the Dich_Target L/R conditions of Experiment 2, for trials in which gaze shifted toward the left or right sides from fixation. We categorized data based on baseline gaze position (−1 to 0 s) into two types: (A) trials with incongruent gaze-shift and MS directions (i.e., MS bias toward fixation) and (B) trials with congruent gaze-shift and MS directions (i.e., MS bias further away from fixation). Insets in panel B show the trial counts for GazeL-AttendR, GazeL-AttendL, GazeR-AttendR, and GazeR-AttendL, respectively, indicating that congruent gaze–attention trials were more frequent than incongruent trials, although the latter were not rare.
The gaze shift cannot account for the link between the stimulus-related MS bias and task performance
One may doubt that the oddball-related MS bias described in earlier sections (Figs. 2–4) is influenced by the gaze modulation due to the oddball presentation which, in turn, possibly affects MS direction in a compensatory manner as shown in Figure 5. To address this, we calculated the gaze bias toward oddball direction around the presentation of oddballs (Fig. 7A,B, right panels) and examined its contributions to the association between the RT and MS bias Figure 6, left panels, which are replots of the bottom panels of Figure 4, B and C. The overall positive bias in gaze position is a replication of the observations in Figure 5B that the gaze direction was shifted toward the direction of sustained attention (Fig. 7, right panels). Comparing low- and high-performance trials (i.e., short- and long-RT trials), we found no significant difference in the gaze time course between short- and long-RT trials. To examine the effects of MS and gaze on RT in a single statistical model, we applied a linear mixed-effects model (LMM) using lme function from the nlme Package in R to the data from Experiments 1 and 2 as follows:
Comparison of the MS rate difference (toward–away; left panes) and gaze position (right panels) between short-RT and long-RT trials for Experiments 1 (A) and 2 (B). Green and purple colors represent short-RT and long-RT trials, respectively. To derive the results in the right panels, the gaze position was extracted from the onset of detected MS. To examine whether the gaze drift has an impact on RT, we applied a LMM using a time range including the significant cluster determined by a cluster-based permutation test for the MS time course (boxes filled with gray lines). Red horizontal lines represent clusters showing significant differences between short-RT or long-RT trials, respectively. See the text for details.
where MStoward and MSaway represent MS toward and away from oddballs, respectively, Gaze indicates the gaze position at MS onset, and SoundLevel corresponds to the stimulus level for Experiment 2. The subject factor was included as a random effect. We focused on the main effects, using a time range including the significant cluster determined by a cluster-based permutation test for MS between short- and long-RT trials (Fig. 7, boxes filled with gray lines). We divided the trials into two groups based on whether the RT was shorter or longer than the median because if we used single trials to predict the RT, there would be few valid trials with MSs. The results for Experiment 1 showed a significant effect of MStoward (t = 2.079; p = 0.0425), whereas MSaway was not significant (t = −1.681; p = 0.0987). Importantly, Gaze had no significant effect on RT (t = 0.217; p = 0.829). The results for Experiment 2 generally showed the same tendency. We confirmed significant effects of MStoward (t = 3.165; p = 0.0019) and MSaway (t = −3.376; p = 0.0010) on RT. SoundLevel also significantly influenced RT, consistent with Figure 4C (t = −5.302; p < 0.0001). Gaze had no significant effect on RT (t = 0.899; p = 0.370). These results indicate that although the MS bias generally correlated with gaze position in a compensatory manner (Fig. 5), the oddball-induced effect on MS direction associated with task performance for an auditory selective attention task cannot be explained by gaze drift.
Auditory subcortical response (FFR): no significant effect of the attentional set
Finally, we asked whether auditory processing (specifically, the brainstem FFR) covaries with the attentional set and/or MS bias during the dichotic oddball detection task. Figure 8A shows bandpass-filtered EEG responses (i.e., FFRs) to a pair of consecutive left-ear (315 Hz) and right-ear (395 Hz) sounds, averaged across all conditions in Experiment 1. We measured the FFRs, which are responses corresponding to the stimuli in time (with some delay) and frequency. We can identify responses to the left- and right-ear sounds by referring to the EEG energy at the frequencies of the respective sounds. Thus, the FFR amplitudes for the left- and right-ear sounds were defined as the spectral EEG amplitudes at 315 and 395 Hz, respectively, and are plotted in Figure 8B.
No significant effect of dichotic attention on average FFR strength. A, Grand-averaged waveform and spectrogram of FFR obtained in Experiment 1. The black bars under the waveform indicate the duration of stimulus presentation. The spectrogram shows the power at the frequencies (315 and 395 Hz) of stimuli used in Experiment 1. B, Comparison of the averaged spectral amplitude of FFR at the stimulus frequencies among the conditions. Multiple comparisons revealed significantly greater responses for the Dich_Mono condition. This result can be explained by the difference in neural adaptation at the brainstem level due to the higher presentation rate in the Dich_TargetL/R conditions or Passive conditions compared with the Dich_Mono condition.
FFR amplitudes varied between left- and right-ear sounds and between some conditions related to the physical conditions of the stimuli (described later). What is critical for the scope of the current study is the difference between the Dich_TargetL and Dich_TargetR conditions within a left- or right-sound case. There was no significant difference between the Dich_TargetL and Dich_TargetR conditions in either right- or left-ear FFRs (Fig. 8B). This result is consistent with the previous studies, which found no significant effect of attention on the averaged FFR (Galbraith, 1994; Varghese et al., 2015).
Less critically, we found that FFR amplitude was largest when the sound was presented monaurally (Mono_TargetL and Mono_TargetR conditions) compared with the binaural presentation conditions (Dich_TargetL, Dich_TargetR, and Dich_Passive conditions) (Fig. 8B). In the Mono conditions, the stimulus presentation rate (collective of the two ears) was lower than in the Dich conditions, which may induce less neural adaptation (thus, greater activity) in the auditory nuclei integrating the binaural information, e.g., in the inferior colliculus (IC), which is the main source of FFR (Chandrasekaran and Kraus, 2010; Bidelman, 2018).
Generally, FFR was larger for right-ear sounds than for left-ear sounds. This difference can be explained by the asymmetry in FFR amplitude between left and right ear sound presentations (Ballachanda et al., 1994) or the frequency dependency in FFR amplitude (Hoormann et al., 1992; Tichko and Skoe, 2017).
FFR strength varied systematically around the MS onset
Next, we examined the relationship between the moment-by-moment FFR strength and the MS direction. This analysis was driven by our expectation that the MS direction may reflect the state of auditory attention at the moment of MS occurrence, as suggested by the earlier sections. In this analysis, for a given MS toward the direction of interest, standard stimuli (separately for the left- and right-ear sounds) around the timing of the MS were found. The data were classified into two categories: congruent and incongruent, e.g., for the case of rightward MS, the right-ear sound was congruent, and the left-ear sound was incongruent (Fig. 9A). We averaged the FFRs with several certain windows (200, 300, 400, and 500 ms) before and after MS onset (pre-MS and post-MS, respectively; Fig. 9B). Our primary interest was whether the FFR changed in association with the MS direction, depending on its congruency with the stimulus ear for which the FFR was evaluated (referred to as the congruent and incongruent cases, respectively). In this analysis, we conducted a mixed-design ANOVA with three within-subject factors and one between-subject factor. The within-subject factors were MS type (congruent or incongruent), time range (pre- or post-MS), and window for average (200, 300, 400, and 500 ms). The between-subjects factor was Experiments (Exp1 or Exp3). Figure 9B shows the pre-MS and post-MS FFR amplitudes for the congruent and incongruent cases in Experiments 1 and 3. The ANOVA result showed an interaction between the MS type and time range (p = 0.0372; η2 = 0.0151). Post hoc analyses revealed that FFR amplitude was significantly smaller for the post-MS period than the pre-MS period for congruent MS (p = 0.0141; η2 = 0.0340), while not for incongruent MS (p = 0.2911). We found no effects of other factors. Overall, we found that FFR strength tended to decrease after MS onsets only for the congruent case. This result suggests that subcortical processes in the auditory system correlate with the instantaneous fluctuation of attention states, probably linked with MS direction, which may not have been captured by a whole-trial average in previous studies (Galbraith and Kane, 1993; Varghese et al., 2015). We believe it is unlikely that the MS–FFR relationship observed above was a by-product of gaze shift during an auditory attention task via the MS’s compensatory relationship with gaze position (Fig. 5). There was no clear correlation between FFR amplitude and the gaze position when we divided the FFR data into the left-sided and right-sided gaze trials.
FFR changes around the MS onset. A, Procedure for calculating pre- and post-MS FFRs. The top and bottom panels show the cases for rightward and leftward MSs, respectively. We first detected MS and sorted the FFR data around MS according to the ear of sound presentations. Red and blue bars represent congruent and incongruent sounds (e.g., right-ear and left-ear sounds for rightward MS), respectively, within the analysis range. The sorted FFR data were averaged within a 200, 300, 400, and 500 ms window relative to the MS and then averaged across trials. B, Comparisons of the FFR between pre- and post-MS for congruent (red bars) and incongruent cases (blue bars). The four panels represent results for different averaging windows. The results indicate that FFR strength tended to decrease after the MS occurrence for the congruent case.
One limitation of our analysis of the relationship between FFR and MS is the exclusion of trials in which adjacent MSs occurred within 500 ms before or after the target MS, potentially removing many typical MSs that occur at a frequency of 1 to 2 Hz. Indeed, the mean proportion of valid MS trials relative to the total MS count was 0.264 (SD, 0.107), with a maximum of 0.489 and a minimum of 0.107, suggesting that the trials included in this analysis may not be fully representative of MS occurrences. We observed a similar trend without excluding adjacent MSs; however, the effect did not reach statistical significance. It remains unclear whether the observed FFR–MS relationship reflects a specific phenomenon occurring only in rare, isolated MS events (i.e., single MSs without adjacent MSs) or is generally observed for all MSs, with other MSs obscuring the effect.
Discussion
We found that the direction of MS was generally biased contralateral to the ear to which the oddball sound was presented or that to which sustained auditory attention was directed. The stimulus-related modulation of MS bias after the oddball presentation was associated with performance in the sound detection task. MS bias away from the location of sustained auditory attention included a component that compensates for the gaze shift. This compensation effect, however, cannot explain the relationship between stimulus-related MS bias and task performance. We also found that there was a relationship between subcortical FFR and MS. The results suggest that auditory neural activity, even in the subcortical stage, fluctuates over time, probably with the states of attention/oculomotor systems.
Temporal dynamics of auditory-induced MSs
The microstructure of the MS time course after the presentation of auditory oddballs showed an early bias toward the stimulus and a late bias away from it. The early bias was only observed when the participants were required to act on oddball sounds without selective attention. Comparable characteristics of MS time course had been reported for the peripheral visual cue: MS direction tended to be biased toward the cued location in an earlier time range and biased away from the cued location in a later time range for both humans and monkeys (Rolfs et al., 2004; Laubrock et al., 2005; Hafed and Ignashchenkova, 2013; Tian et al., 2016; Hafed et al., 2021). The early and late MS biases may be mediated by different neural circuits, involving the superior colliculus (SC) and frontal eye field (FEF), respectively: The inactivation of the monkey SC diminished the early MS bias toward the peripheral cue, while the late opposite bias remained (Hafed and Ignashchenkova, 2013; Hafed et al., 2021). On the other hand, the inactivation of the monkey FEF influenced the late MS bias away from the cue, while the early MS bias toward the stimulus remained (Peel et al., 2016; Hafed et al., 2021). Our results extended this theory to the auditory domain. It is important to note that the latencies of the early and late biases appear longer for the auditory target (∼0.5 and 1 s for the early and late biases, respectively) than for the visual target (∼0.2 and 0.4 s). The latency difference between visually- and auditory-evoked MS bias suggests that the auditory-evoked MS bias may involve an indirect pathway from the auditory signal input to an oculomotor command, for example, through the connection between IC and SC, or the auditory input to the FEF (Kirchner et al., 2009).
Suppression of early MS bias and its relationship to task performance
Why was the early MS bias toward the oddball sounds observed only in conditions when the target was presented to both ears, given that previous visual studies have generally reported MS biases toward the visual cue? In the dichotic selective detection task, where sustained attention was directed to one side, participants might have employed a strategy to suppress stimulus-driven orienting responses to reduce the probability of attentional capture by oddballs on the unattended side. The absence of MSs toward the oddball sound under selective listening conditions may thus reflect “success” in inhibiting unnecessary orienting responses. However, participants were informed in advance of the location where they should attend. If the above hypothesis is correct, this suggests that the brain cannot selectively suppress orienting responses toward the unattended side (e.g., leftward MSs in the Dich_TargetR condition).
Our findings on the link between the MS bias and task performance (i.e., RT) are consistent with this idea. The MS tended to be biased toward the position of oddballs for the low-performance trials (long-RT trials), suggesting that the suppression of the orienting MS failed when the participants' alertness levels were low. Such an inhibition of orienting responses might be controlled by the top–down signal from FEF (Van der Stigchel et al., 2012). The result implies that the MS can be an index of time-varying attention states relevant to spatial processing in the auditory domain.
Interpretation of MS bias away from the target direction
Our consistent observation of the late MS bias away from the target direction may be explained by mechanisms proposed in the visual domain. One is the inhibition of return, typically associated with the late MS bias (Galfano et al., 2004), which may be preserved even when the early reflexive response related to attentional capture was suppressed. As discussed above, the early and late MS biases likely involve distinct neural mechanisms—specifically, the SC and the FEF, respectively—suggesting that the effects of attentional modulation on MS emerge differently across temporal phases. Alternatively, the late MS bias may reflect an imbalance in the saccade system induced by the suppression of the early MS. Rolfs et al. (2004) reported a consistent late MS bias in the direction opposite to an exogenous visual cue, even in the absence of an early cue-directed bias, in the Posner task using an informative cue. They interpreted this finding as a consequence of active oculomotor suppression engaged to maintain central fixation.
Does the MS direction reflect stimulus or gaze direction?
The MS modulation observed in our study is likely to reflect a mixture of two components: (1) stimulus-driven responses associated with attention-related dynamics of the saccade system and (2) gaze-correcting responses that return the eye to fixation. Distinguishing between these components is a key interpretive challenge. In our experimental design, we presented the auditory stimuli via earphones to both the left and right ears, such that the direction of sustained auditory attention was assumed to lie along the horizontal axis. In this configuration, the axis of the eye movements that break and return to fixation aligns with (or at least overlaps with) that of auditory attention, which raises ambiguity about whether a given MS reflects a stimulus-driven shift of attention or a return-to-fixation movement following gaze drift toward the attended location. Interestingly, however, we observed a late MS bias away from the oddball (i.e., further away from the fixation), even when gaze drifted toward the side opposite the attended target (e.g., leftward gaze in right-attention trials; Fig. 6). Since a gaze-correcting mechanism alone cannot account for this pattern, the observed directional bias likely reflected a stimulus-driven component and was not merely the result of returning the eye to fixation. Future studies may be able to separate the above factors by placing auditory targets off the horizontal axis (e.g., bottom-left and bottom-right). This would allow MS directions to be mapped onto attended versus unattended locations along different spatial axes, assuming that auditory spatial coordinates are roughly comparable to those used for oculomotor targets.
Link between MS occurrence and auditory subcortical FFRs
Finally, we asked whether the MS can be an index of fluctuating neural activity, which may covary with the attention state. We measured the auditory subcortical responses (FFRs) during a dichotic selective detection task. When the data were averaged over the entire period of stimulus presentation, there was no significant effect of attentional set on the FFR to standard sounds for both left- and right-ear sounds. This result is consistent with the null effect of attention on FFR reported by previous studies (Galbraith and Kane, 1993; Varghese et al., 2015). Through careful inspection, however, we found that the FFR at the stimulus frequency was linked with the onset and direction of MS during a selective detection task. FFR may be susceptible to temporal variation in the attention states (Esterman et al., 2013; van den Brink et al., 2016; Terashima et al., 2021), which covaries with the eye activities revealed by the current study. The current result may shed light on the discrepant results of studies about the effect of attention on FFR. The temporal fluctuation of the subcortical activity may covary with the attention states, which cannot be captured by a simple whole-trial average in conventional FFR studies but might be indexed by the MS. Recent studies found the effect of attention on the FFR signal to a running speech stimulus (Forte et al., 2017; Etard et al., 2019) and by applying source-level analysis of EEG data to emphasize the subcortical activity (Price and Bidelman, 2021), which may be able to capture the time-varying fluctuation of the effect of attention.
Footnotes
The authors declare no competing financial interests.
- Correspondence should be addressed to Shimpei Yamagishi at shimpei.yamagishi{at}gmail.com.
