Abstract
The human ability to track overlapping and asynchronous time intervals is crucial for a myriad of tasks, from engaging in conversation to driving a car. Additionally, unexpected events can trigger rapid, on-the-fly adjustments, necessitating quick updating of both timing intervals and action planning. Such events require immediate recalibration of decision variables to allow the system to promptly adapt to new stimuli and update the timing mechanisms accordingly. In this study, we assessed human male and female participants’ ability to track two simultaneous and asynchronous beep trains and determine which one ended first. Due to the stochastic nature of the beeps, participants frequently had to reorient their intended actions in order to identify which train was more likely to have ended. We found that they were able to do this accurately, demonstrating timing performance that was comparable with that of a single train. At the neural level, we recorded slowly evolving EEG potentials that encoded a single interval, the one associated with the currently intended action. Upon an intention switch, when participants had to reorient to a previously unintended action, the EEG response amplitude was reset to reflect the new intended interval. In contrast, when participants were instructed to disregard one of the beep trains, EEG responses solely reflected the intervals of the sequence they attended to. This flexibility in response highlights the brain's ability to dynamically reconfigure cognitive processes in real time, ensuring that actions remain contextually appropriate despite sudden changes in the environment.
Significance Statement
The human brain exhibits a remarkable ability to track temporal patterns and rapidly adjust timing and action plans in response to unexpected events. Using a novel task, we show that humans can flexibly process two independent, asynchronous sound trains and accurately determine which ends first. The unpredictable nature of the stimuli required frequent shifts in participants’ intentions and adjustments in timing. EEG recordings reveal neural signals that mirror this adaptability, dynamically aligning with behavioral changes. These findings highlight the brain's capacity for real-time cognitive flexibility in response to sudden environmental changes, offering new insights into the mechanisms of complex timing behavior.
Introduction
The brain's capacity to follow temporal patterns and produce well-timed motor commands is crucial to a wide variety of tasks and behaviors, from playing a musical instrument to holding a conversation (van Rijn, 2018). One of the elementary components of temporal processing is sensorimotor interval timing, which requires participants to respond to a given stimulus only after a certain amount of time has elapsed (Buhusi and Meck, 2005; Paton and Buonomano, 2018).
By design, responses in such tasks are self-initiated rather than tied to immediate sensory events. Furthermore, humans can keep track of multiple overlapping yet not necessarily synchronized intervals, rapidly adjusting to changes in interval length, variability, and other task demands. Interval timing, therefore, relies on internally generated, highly flexible time-keeping mechanisms (van Rijn and Taatgen, 2008; Salet et al., 2022; Tsao et al., 2022).
Previous studies investigating the neural mechanisms of temporal processing in humans uncovered EEG signatures relating to and predictive of time perception and timing behavior (Walter et al., 1964; Volberg and Thomaschke, 2017; Hassall et al., 2022). For example, in a temporal bisection task, in which participants are required to categorize intervals as “short” or “long,” an EEG potential building up from the interval offset to the participants’ motor response are correlated with the interval's duration and reflects subjective time perception (Ofir and Landau, 2022).
Additionally, the potential evoked by a tone depends on the duration elapsed since the previous tone (the interstimulus interval, ISI; Nelson and Lassman, 1973; Kononowicz and van Rijn, 2014). ISIs also modulate slower evolving potentials in the EEG recording. For example, the contingent negative variation (CNV) is a monotonic negative ramp that is locked to each stimulus and whose slope correlates with the expected time of an upcoming event (Breska and Deouell, 2017). Furthermore, contextual effects, which involve encoding information about the probability distribution of event timing, also influence temporal processing (Grabenhorst et al., 2019). In the absence of prior knowledge, the expectation of an upcoming event typically increases with elapsed time. However, when certain intervals are more probable, expectations peak around these more likely intervals. EEG (Herbst et al., 2018; Tavano et al., 2019) and MEG (Todorovic and Auksztulewicz, 2021) recordings have demonstrated that neural activity encodes both the passage of time since an event and contextual probabilities derived from experience.
Finally, in self-paced response tasks, a related EEG component is the readiness potential (RP), which manifests as a persistent ramp-like activity and has been observed preceding self-initiated actions (Libet, 1985). RPs are typically viewed as neural antecedence of voluntary actions, reflecting motor preparation (Schurger et al., 2021). In the framework of decision-making, it was posited that the RP represents a decision variable in tasks involving perceptual judgment and action timing, particularly in ramp-to-threshold models of decision-making (Schurger et al., 2012; Schurger, 2018).
However, the majority of research has concentrated on the mechanisms of single time intervals. Consequently, our understanding of how multiple intervals are simultaneously timed and the underlying mechanisms responsible for rapid adjustments in action planning and timer resetting is limited.
To study the human ability to track multiple intervals, we developed a simultaneous-timing version of the stop-reaction-time task (Rousseau and Rousseau, 1996), in which participants listen to two independent beep trains and are required to identify the first train to end. To minimize temporal regularities that might enable the prediction of the next beep's timing, the duration of each train and the ISIs were both exponentially distributed. This approach enabled us to uncover a prominent frontocentral EEG pattern, whose dynamics reflected the subjective propensity to respond, stretching for late responses and contracting for early ones. By contrasting the resetting of this EEG decision variable when two sensory streams were being tracked, and when one of them is to be ignored, we show the flexible dynamic nature of the underlying mechanism as depending on both sensory input and task requirements.
Materials and Methods
Participant details
Forty individuals participated in this study (aged 26.45 ± 2.5, mean ± SD; 16 women). The participants were recruited from the university community and were compensated for their time with either money (∼10 Euro per hour) or class credit. All procedures were approved by the institutional review board of ethical conduct of the Hebrew University.
Experimental design and procedure
The participants sat in a dimly lit, sound-attenuated room in front of a BenQ XL2420Z monitor with a refresh rate of 100 Hz and two speakers on either side. This study involves three different tasks, and each participant performed at least two of them. Each task started with verbal and written instructions, followed by 10 familiarization trials, and proceeded to the test phase, which consisted of 140 trials. Each task was divided into four blocks with short breaks in-between that were self-terminated by the participants. Task order within a session was random.
Single timing task
At the beginning of the “single” task, the participants were informed that on each trial they would hear a random beep train and were instructed to press a button whenever they thought no more beeps would be played. The response was delivered through a keyboard button press (on the QWERTY US keyboard)—either “.” using the right hand for right stimuli or “x” using the left hand for left stimuli. After the participants made their response, a feedback of “Correct” or “Too early” was displayed depending on whether the response was given before or after the end of train (Fig. 1A,C).
Probabilistic auditory change detection task. A, Trial schematic in the “single” task. Each trial is composed of an initial “wait” period with a duration drawn from an exponential distribution followed by a “go,” or response window. During the “wait” period, Poisson beeps are played, and the participants are instructed to respond when they believe the beep train has ended (i.e., as soon as the “go” period starts). Response times (RTs) are defined as the intervals between the last beep and the response. B, “Single” session behavior from one example participant. Each row corresponds to a single trial. Rows are sorted according to last-beep-times and split according to side. Green dots mark right beeps, orange dots left beeps, and black dots mark responses. C, An optimal decision variable (DV) in this task measures elapsed time since the last beep. The DV ramps until either another beep is played, which causes it to reset, or reaching threshold and triggering the response. D, Average performance as a function of last beep times for empirical (black) and shuffled (red) data (n = 19). Inset, Single participant regression coefficients for performance as a function of last beep times for empirical and shuffled data. Asterisks indicate significant difference between datasets: ***p < 0.005, paired t test. E, Average RT distributions for different binned last-beep-times (n = 19). F, Average RTs as a function of last-beep-timed for empirical (black) and shuffled (red) data (n = 19). Inset, Single participant regression coefficients for RT as a function of last beep times for empirical and shuffled data. Asterisks indicate significant difference between datasets: ***p < 0.005, paired t test.
The single beep train was either played through the left speaker in low pitch pure tone (440 Hz) or the right speaker in high pitch pure tone (1,046 Hz). Left and right trials were interleaved but within a trial all stimuli had the same pitch and came from the same side. To generate a train, an exponentially distributed train length was drawn (5 s mean) and filled with 40-ms-long beeps (with a 5 ms ramp up and down). One beep was positioned at the start of the train, and additional beeps were randomly added from a Poisson process, such that interbeep intervals were exponentially distributed with a 1 s mean (but could not be shorter than 50 ms to avoid beep overlap). Therefore, the stimulus was an inhomogeneous Poisson process, in which beeps occurred at a rate of one per second during the train and zero after it.
At the start of each trial, the participants fixated their gaze on a red point at the center of the screen and a word matching the current stimulus side (“Left” or “Right”) appeared on the corresponding side of the fixation point, this side cue was present throughout the trial. Then, the auditory train was played and the participants would make their temporal decision using the relevant button press—right or left depending on the single-stimulus train side.
Cued timing task
This task was similar to the “single” task, except that both low-pitched left and high-pitched right stimuli were played simultaneously and independently. At the beginning of each trial, the participants were instructed to attend to only one of the streams, either low-pitched left or high-pitched right, and ignore the other. Similar to the single task, at the start of each trial, a word matching the cued stimulus side (“Left” or “Right”) appeared on the corresponding side of the fixation point and was present throughout the trial. Then, different from the single task, auditory trains started on both sides and the participants would make their temporal decision using the relevant button press—right or left depending on the cued side.
Dual timing task
This task was similar to the “cued” task, except that both the words “Left” and “Right” appeared to the left and right of the fixation point, respectively, at the beginning of each trial. The participants were instructed to respond when they thought one of the trains has ended, so that no more beeps would be played from it, even if the other side's train was still ongoing. Note that while the participants were instructed to respond to the first train to end, responses to the second train were also considered correct if they were produced after the end of the second train.
To keep experiment time reasonable, all participants performed the “dual” task and (at least) one of the other two: 18 performed the “single” task, 20 performed the “cued” task, and one participant performed both.
EEG acquisition
A g.GAMMAcap and a g.Hlamp amplifier (g.tec) were used to record EEG signals during task performance. The cap had 62 active electrodes distributed over the scalp with two active earlobe electrodes. Electrodes were positioned according to the extended 10–20 system. For consistency P9, P10, or F9 and F10 electrodes were removed prior to analysis since part of the participants were recorded with P9 and P10 positioned while the rest had F9 and F10 positioned. Vertical electrooculogram (EOG) was recorded for all participants using passive electrodes (g.tec) placed above and below the left eye and the horizontal EOG using electrodes placed at the outer canthi of both eyes. Both EEG and EOG were continuously sampled at 512 Hz. Online anti-aliasing filter was applied to the EEG data with a cutoff frequency of 19.2 kHz. Signals were stored for offline analysis using a Simulink model (MathWorks).
EEG preprocessing
All offline preprocessing and analyses were done using Matlab 2020a employing functions from FieldTrip (Oostenveld et al., 2011), EEGLAB (Delorme and Makeig, 2004), Unfold (Ehinger and Dimigen, 2019), and Ept-TFCE (Mensen and Khatami, 2013) toolboxes as well as custom code. The EEG was referenced offline to the average of both earlobe electrodes. Slow drifts in the EEG were removed per electrode and per block (in segments between breaks) using the spline-based baseline removal function msbackadj() included in the Matlab Bioinformatics toolbox. This function was used to estimate baseline using a 4 s wide sliding window with a 0.75 s step. For each window, the baseline value was set as the median. A spline was interpolated for each block using its window's baseline (median) values and then subtracted from the data to detrend it. This method was compared with detrending with a 15 s spline window, which produced qualitatively similar but noticeably noisier results due to significant drift, and with a bandpass filter (0.1–200 Hz, two-sided Butterworth filter). Reanalyzing the data presented in Figures 4⇓⇓–7 using the filtered data produced results identical to those obtained with 4 s spline-based detrending.
Next, independent component analysis was performed on each participant’s dataset from which components corresponding to eye movements and blinks were visually selected and rejected.
For each participant, noisy electrodes were removed by visual inspection using the summary statistics of Fieldtrip’s ft_rejectvisual() function: one participant’s “T7” electrode was removed, and another participant’s “FC5” and “FC6” electrodes were removed.
In the next step of preprocessing, we sought to identify channel clusters that correspond to either side or action-time coding using the “single” dataset (n = 19). To do so, we fitted a GLM with response side as a single predictor to each participant's data. Each trial was aligned to the motor response and regression was evaluated at successive time points ranging from 7 s before to 1 s after the response (this was done using the unfold toolbox). Hence, a significant predictor indicated that EEG responses at that time point are different for the two sides, and a significant intercept indicated that EEG responses are different from 0. This latter case was used to identify the action time-dependent component. Visual inspection revealed noisy regression in one participant's data, which were then discarded from the next step.
Next, the resulting sets of intercepts and regression coefficients (each containing 4,571,136 values; 18 participants × 512 samples per second × 8 s × 62 electrodes) was fed to the ept-tfce toolbox and underwent the Threshold Free Cluster Enhancement permutation (TFCE) test. The goal of the TFCE method is to enhance initial statistic (t values) using information from neighboring data points to decrease noise. Generally, this is done by assigning a TFCE value to each data point, accounting for its intensity and factoring in the signal intensity of its neighbors. The neighborhood is both spatial and temporal neighborhood, taking into account activity from adjacent channels and time points. We applied this procedure using the TFCE default settings regarding weighting of neighborhood and intensity (“E” = ⅔, “H” = 2, cluster alpha = 0.05). This procedure can be viewed as a generalization of the cluster-mass approach (Smith and Nichols, 2009). The null distribution was approximated randomly by permuting the labels of the coefficients in the 4D sets (shuffling the labels for random participants in each permutation) and recalculating the TFCE population score for each voxel (2,500 permutations). This procedure revealed significant effects in both the intercept set and the regression coefficients set. The response side analysis revealed a significant lateralized cluster starting at ∼0.18 s prior to the response (Fig. 3A,B; peak significance found at channel “C4” −0.14 s before the response, p < 0.001). The intercept analysis revealed a significant frontocentral cluster appearing 1.35 s before the response as a general negative deflection lasting until the response (Fig. 3C; peak significance found at channel “C4” 0.06 s before the response, p < 0.01).
In light of the cluster results, most analyses focused on average activity from the frontocentral cluster (Fz, FCz, Cz, F1, F2, FC1, FC2, C1, and C2). Trials containing artifacts were removed following visual inspection through Fieldtrip's ft_rejectvisual() function. On average, 3.8 ± 1.9% (mean ± SD; max 9.6%) of recorded trials were removed per participant. Lastly, electrodes rejected due to artifacts were interpolated with spline interpolation using FieldTrip's ft_channelrepair() function.
Participant exclusion
In this study, we analyzed both behavioral and EEG datasets, implementing exclusion criteria based on behavioral performance and recording quality. Participants with faulty behavior or poor recording quality were excluded from subsequent analyses. Following the behavioral analysis, one participant was excluded from the “cued” dataset and eight participants from the “dual” dataset due to misunderstandings of the respective tasks. Additionally, one participant was excluded from all analyses due to significant EEG artifacts, with 10 channels showing variance an order of magnitude higher than the other channels throughout the session. Finally, one participant was excluded only from the regression analysis (described above and in Fig. 3) as their regressor value (averaged across channels) exceeded 3 standard deviations from the population mean within a 2 s window before the response. Overall, in the main text and figures, we analyzed “single” data from 19 participants, “cued” data from 20 participants, and “dual” data from 31 participants.
Statistical analyses
We used t tests for comparisons between two data groups or one-way ANOVA in cases in which more groups were compared. To evaluate the effects of continuous variables, we used linear regression models. When comparing regression coefficients between groups (empirical vs shuffled data in Fig. 1), we first fitted a model to each participant's empirical and shuffled behavior and then compared the regression coefficients of all subjects using paired t tests. In the analysis of EEG signals, in which the data was not grouped, we used a mixed-effects model, treating participants as random effects and the other variables as fixed.
All analyses were performed using custom code written in Matlab (MathWorks). In all figures, average data and error bars, or shaded patches around curves, represent mean ± SEM.
EEG simulations
To simulate EEG signals, we first generated synthetic trials for the three tasks (140 trials per task per subject). We then simulated for each trial a decision variable (DVs) that ramped linearly until reaching a threshold to trigger a response or reset if another beep occurred first. Resetting was task-specific: in the “single” condition, the DV reset after every beep; in the “cued” condition, it reset only after attended beeps; while in the “dual“ condition, partial resetting occurred following switch beeps. To account for response time variability, we fitted each subject's RT distribution to a Gaussian using MATLAB's normfit() function. The fitted distribution was then used to generate a random DV slope after each reset, defined as slope = 1/X, where X is a Gaussian-distributed random variable sampled from the fitted RT distribution. By setting the threshold to 1, the simulated RTs matched the distribution of the real data (Fig. 6A).
EEG responses were then simulated using a variable-duration template based on the analysis in Figure 4. This template consisted of three phases: an early stimulus-locked ERP lasting 0.5 s, a plateau of variable duration, and a linear negative deflection (RP, or readiness potential) lasting 1.5 s. The early ERP was visually fitted as the sum of three Gaussians (means: 0.12, 0.2, and 0.3 s; standard deviations: 0.025, 0.05, and 0.1 s; amplitudes: −5, 2, and −1 μV), while the RP was modeled as a linear negative deflection ranging from 0 to −4.1 μV over 1.5 s. The RP's duration and threshold were determined based on the regression analysis presented in Figure 4F.
Between these two fixed-duration components, the plateau maintained a constant amplitude of 0 μV, with its duration determined by RT variability. Specifically, plateau duration was set to X − 2 s, where X is the Gaussian-distributed variable defined above. The 2 s offset accounts for the combined duration of the ERP (0.5 s) and RP (1.5 s), ensuring that the total length of the simulated EEG signal from onset to threshold matched that of the decision variable (DV), both following the same distribution as the real data. Additionally, since nonresetting beeps also evoked early ERP signals (Fig. 7), these responses were added to the simulated signal but without resetting it.
All simulations were performed using custom code written in Matlab.
Results
Behavioral performance
To study multiple timing behavior in a dynamic environment, we designed a novel probabilistic auditory change detection task. This task had three versions: the primary task in which participants had to track two asynchronous stimulus streams (which we term the “dual” task) and two control tasks, one presenting only one stream (the “single” task) and the other which required tracking a single stream in the presence of a second distracting stimulus stream (the “cued” task). Each participant performed the “dual” task and at least one of the control tasks (see Materials and Methods).
For clarity, we start by describing the simpler “single” task (Fig. 1A,B). In the “single” task, participants were presented with a train of Poisson beeps, and the participants were instructed to press a button when they believed no more beeps would occur. On each trial, the beeps were either low-pitched tones from a left speaker, requiring a left button press, or high-pitched tones from a right speaker, requiring a right button press.
Since the end of the stimulus train was not explicitly signaled, the task required participants to infer an underlying “hidden” state (i.e., whether the beep train has ended or not) from noisy observations (i.e., beeps). Each beep indicated with certainty that the train was still going on, but periods of silence were ambiguous, since they occurred both before and after the beep train ended.
Due to the memoryless property of the exponential distribution, from which we drew both train lengths and interbeep intervals, the elapsed time since the last beep was sufficient to infer the current state, regardless of the time that has passed since the train start time. Therefore, an ideal observer should solve this task by measuring the elapsed time since the last beep and reset this timer after each new beep. Action can be readily implemented by setting a threshold to this timer, whose value determines the interval between the last beep and the response. To give a mechanistic intuition to this process, we visualized the timer as a dynamic variable that ramps linearly toward threshold in between beeps and resets after each beep (Fig. 1C).
A defining hallmark of a resetting strategy is that both response times (RTs), measured as the interval between the last heard beep and the response (regardless of whether the response occurred before or after the state change), and timing accuracy, defined as the probability of responding after the state change, should not depend on the trains’ length. In contrast, if the participants were timing their responses from the start of each train without resetting after beeps, we would expect both RTs and accuracy to decrease with train length. We therefore measured the correlation between train length and timing performance as well as RTs. We then compared these results to those of a shuffled dataset, created under the assumption of a nonresetting strategy. In this dataset, the interval between train start and response was paired with a randomly selected beep train, ensuring that the distribution of time elapsed from the start of each train to the response was identical to that of the empirical dataset, but the RT (i.e., the interval between the last beep and the response) was random. Consequently, shuffled RTs were essentially the same as those predicted under a nonresetting strategy and therefore expected to result in negative correlations between the length of the train and both RTs and accuracy.
Using a linear model to estimate the effect of train length on each participant's performance in the empirical dataset, we found that the slopes of the regression were clustered around zero, whereas the shuffled dataset's coefficients were significantly lower (empirical data vs shuffled data regression slopes: 0.0018 ± 0.0022 vs −0.078 ± 0.007, mean ± SEM; t(18) = 12.12, p = 6.56 × 10−12, paired t test, n = 19; Fig. 1D). Furthermore, RT distributions in different-length trials were overlapping (Fig. 1E), and the estimated regression coefficients were significantly higher (i.e., closer to zero) than those observed in the shuffled data (empirical data vs shuffled data regression slopes: −0.052 ± 0.018 vs −0.67 ± 0.043, mean ± SEM; t(18) = 15.54, p = 5.90 × 10−12, paired t test, n = 19; Fig. 1F). We conclude that in agreement with a resetting strategy, timing behavior was only marginally influenced by train length.
We now turn to describe the other two tasks. The “cued” task was similar to the “single” one, only that here, two beep trains of different frequencies (440 Hz for the right choice and 1,046 Hz for the left choice) were played simultaneously on each trial, and the participants were cued at the beginning of each trial to base their decisions on only one of the trains (the attended stimulus) while ignoring the other (the unattended stimulus). Finally, in the “dual” task, both stimuli were also presented simultaneously and the participants were instructed to attend to both trains and respond to the side in which the beep train ended first (Fig. 2A).
Simultaneous timing performance. A, Trial schematic in the “dual” task. Each trial is composed of two, independent, exponentially distributed “wait” periods, each followed by a corresponding “go” period. Poisson beeps of differing pitches are played during both wait periods, and the participants are instructed to respond when they believe the first beep train has ended. Responses are produced through either a left-hand button press if the left beep train ended first or a right-hand button press if the right beep train ended first. Left/right response times are defined as the intervals between the last beep from the corresponding side and the response. B, Average psychometric curves showing the probability of choosing “left” as a function of the difference between RT-left and RT-right, as defined in panel A in the “dual” (n = 31) and “cued” (n = 20) tasks. C, Average RT distributions in the three tasks measured from the last chosen beep ('single”: n = 19, “cued”: n = 20, “dual”: n = 31). D, Average RT distributions in the “dual” task measured from the last unchosen beep (n = 31). E, Average RTs in the “dual” versus “single” tasks for participants who successfully performed both tasks (n = 13). F, Average RTs in the “dual” versus “cued” task for participants who successfully performed both tasks (n = 18). G, H, During dual timing, each beep train is linked to a timer that tracks the time elapsed since the last beep (G). These timer signals then feed into a decision variable, which monitors the distance of the action-relevant timer from its respective decision threshold (H). I, The readiness variable is defined as the absolute value of the decision variable.
We first examined whether participants correctly attended to the cued stimulus in the “cued” task and to both stimuli in the “dual” task. We calculated the probability of choosing left as a function of the difference between the response time measured from the last left beep (RT-Left) and the response time measured from the last right beep (RT-Right). In the “dual” task, if the participants were attending to both stimuli and choosing the first to end, then this function would be a step centered around zero, indicating that the participants chose left when the right beep was more recent and vice versa. Alternatively, if the participants were only attending to one of the sides (as they should in the “cued” task), then this function would be flat, and if they were waiting for both stimuli to end before choosing the last stimulus to end, it would resemble a negative step. We found that the majority of participants (31/39) correctly performed the “dual” task, attending to both stimuli and responding to the side where the beep train ended first (Fig. 2B).
We next tested whether the requirements to direct attention to two input streams affected performance by comparing RT distributions between tasks. RTs in the “cued” and “dual” tasks were defined as the intervals between the last beep from the chosen side and the button press. Surprisingly, we found only small differences in the participants’ timing performance between tasks, as can be in their overlapping RT distributions (Fig. 2C). However, within-participant analysis revealed that RTs were significantly shorter in the “single” compared with the “dual” task (“single” vs “dual” mean RTs: 4.13 ± 0.26 vs 4.5 ± 0.28, mean ± SEM; t(12) = −3.40, p = 0.0053, n = 13; Fig. 2E), but no significant difference was found between the “dual” and “cued” task (“cued” vs “dual” mean RTs: 4.08 ± 0.21 vs 4.25 ± 0.21, mean ± SEM; t(17) = −0.89, p = 0.38, n = 18; Fig. 2F). We also observed that in the dual task, the intervals between the last beep belonging to the unchosen side and the response were distributed very differently (Fig. 2D). Specifically, the distributions of intervals from the unchosen beeps were maximal for the shortest intervals, suggesting that participants correctly ignored irrelevant beeps while timing their responses to relevant ones.
Mechanistically, multiple timing can be described as two parallel timers tracking simultaneous beep trains, each resetting with every beep in its respective channel (Fig. 2G), similar to the single beep case (Fig. 1C). A decision variable alternates between the timers such that it represents at any moment in time that timer which is the closest to its respective threshold, and leading to a choice of “left has terminated” upon crossing the left threshold and “right has terminated” upon crossing the right threshold (Fig. 2H). Lastly, we define the readiness variable as the absolute value of the decision variable, representing the distance of the action-relevant beep from its corresponding threshold (Fig. 2I).
Electroencephalography
Next, we examined electroencephalography (EEG) signals during task performance in order to investigate the neural correlates of multiple timing behavior. Specifically, we were interested in dynamic signals that correspond to the selection of lateralized responses and to their timing.
We therefore performed an initial cluster-based analysis on data from the “single” condition, designed to differentiate and localize the two signals (i.e., the selected action's side and timing) through time and channels (see Materials and Methods). This analysis revealed a significant cluster representing the side of the response. This cluster's activity, which corresponds to the lateralized readiness potential, emerged just before the action at lateralized frontocentral sites with opposite polarities for ipsilateral and contralateral button presses (∼0.2 s before the action; Fig. 3A,B). In addition, this analysis also revealed a negative action-timing cluster that appeared in frontocentral regions and was not lateralized, already 1.3 s before the action (Fig. 3C), corresponding to the readiness potential. In the next sections, we analyze the nonlateralized frontocentral activity pattern in detail.
EEG cluster-based analysis for action time and side representations. A, Action side was represented in lateralized activity, forming two opposite-sign clusters. The four curves correspond to each cluster's averaged activity in the two control tasks (“single” and “cued”) aligned on response time, split by left (orange) and right (green) responses, and by the relation between the cluster's location and action (dark, ipsilateral; light, contralateral; n = 38). Gray rectangle indicates significant action-side responses (cluster peak significance: p < 0.005 at sensor T4 at time of action; threshold free cluster enhancement permutation test). Inset, Topography of action-side coding. Colors correspond to the relevant regression coefficient averaged in the time window between the action and 0.2 s before it. White circles mark the sensors that formed the two clusters. B, Zoom in plot showing the lateralized activity pattern just before the action. C, Action timing was represented in frontocentral activity. The curves show this cluster's averaged activity in the two control tasks aligned on action and split by left (orange) and right (green) responses (n = 38). Gray rectangle indicates significant action-time responses (cluster peak significance: p < 0.005 at sensor F4 at time of action; threshold free cluster enhancement permutation test). Inset, Topography of action-time coding. Colors correspond to the relevant regression coefficient averaged in the time window between the action and 1.3 s before it. White circles mark the sensors that formed this cluster.
Frontocentral EEG activity scales and resets dynamically with the decision variable
Numerous studies of human EEG report that when participants are required to perform a self-timed action, frontocentral activity displays a monotonically decreasing potential starting at ∼1.5 s prior to action, an activity pattern known as the readiness potential. Since RTs in our tasks exceeded 1.3 s, we reasoned that the observed RP (whose length was 1.3 s) cannot fully represent the decision variable introduced earlier (Fig. 1C). Instead, we hypothesized that the readiness potential, together with the preceding activity, corresponds to a reproducible, yet more complex neural activity pattern, with its rate of unfolding corresponding to the subjective perception of time leading to action, as represented by the readiness variable introduced earlier (Fig. 2I).
Three predictions can be made by this hypothesis: (1) frontocentral activity should scale with RTs, compressing in short RT trials and stretching in longer ones; (2) it should reset following all beeps in the “single” task, and following attended but not unattended beeps in the “cued” task; and (3) it should partially reset following a beep in the “dual” task, but only if the preceding beep was in the other stream. We now address these predictions.
In order to test the first prediction, we examined the detailed structure of frontocentral EEG activity between the last attended beep and the responses in the two control tasks (“single” and “cued”), since they provide uninterrupted stretches of time in which action timing unfolds from its initial value to action. Figure 4A shows the average frontocentral EEG signal aligned on these last beeps. Following an early stimulus locked event-related potential (ERP) that lasts ∼0.5 s, activity plateaus for ∼1 s, and then gives way to a prominent negative deflection, the RP.
Scaling of frontocentral EEG activity with action time. A, Average frontocentral EEG activity following last beeps of the chosen side in the two control tasks combined, aligned on beep onset (n = 38). B, Same as A but split according to response times. The red shading marks a period of significant negative correlations between neural activity and action timing, and the blue shading marks significantly positive correlations (p < 0.05; sliding window linear mixed-effects model, Bonferroni corrected). C, Zoom in on the first second after beep onset shown in B. D, Top, Hypothetical illustration for a scenario in which trial-by-trial response times variability corresponds to changes in the rate of (negative) ramping activity. Arrows mark response times. Bottom, When aligned to action timing, these traces are expected to exhibit different slopes, leading to a negative correlation between response times and activity. The strength of this correlation is predicted to increase over time when assessed at varying time windows relative to the response (colored rectangles). E, Same as D, but for the hypothetical scenario where response-time variability corresponds to differences in the timing of ramping onset. In this case, aligning traces to the action would result in overlapping activity patterns. F, Same as B, but aligned to the response. The gray shading marks a period where the intercept coefficient is significantly different from zero (p < 0.05; sliding window linear mixed-effects model, Bonferroni corrected). Inset, Average EEG activity in 0.5 s consecutive windows for binned response time data (3–4, 4–5 , and 5–6 s). Each curve represents EEG values within a specific time window, with colors corresponding to the time relative to the response, as indicated by the shaded rectangles in panels D and E.
Figure 4B shows the same data split according to binned RTs. In agreement with our first prediction, the EEG signal seems to scale as a function of RT, such that fast RT trials were associated with a faster initial increase to plateau and a correspondingly fast decrease toward action. To test this statistically, we used a sliding window analysis (0.5 s window size, 0.25 s step size), in which we tested for significant relations between the average EEG signal in each window and the RT using linear mixed-effects models. We observed an early negative slope (within a 0.25–0.75 s window after beep onset), corresponding to more positive responses in short RT trials (p < 0.05, linear mixed-effects model Bonferroni corrected for multiple comparisons, n = 38; Fig. 4B,C, pink), and a significant positive slope later on (between 2.25 and 5.75 s after the beep), where stronger negative responses were associated with shorter RTs (p < 0.05, linear mixed-effects model Bonferroni corrected for multiple comparisons, n = 38; Fig. 4B, light blue).
Further inspection of Figure 4B suggests that the EEG signal may not scale uniformly with RTs. Specifically, fast responses might correspond to either a steeper ramping of the readiness potential (Fig. 4D) or a shorter plateau leading to earlier onset of ramping (Fig. 4E). To distinguish between these possibilities, we noted that aligning the EEG signal to the action time would reveal different patterns: if faster readiness potential ramping occurs, there should be a negative correlation between the EEG signal and RT, which would become more pronounced further back in time from the action. In contrast, if the plateau scales with RT, the signals should overlap, showing no correlation between EEG and RT (Fig. 4D,E). To test this, we performed another sliding window analysis (range: −3 to 0 s before action; window size: 0.5 s, step size: 0.25 s) to assess the relationship between the action aligned average EEG signal within each window and RT using linear mixed-effects models. To avoid contamination from early ERP signals and overly late outlier trials, only trials with RTs between 3 and 6 s were considered. Our analysis supported the plateau scaling scenario, as EEG signals showed no significant dependence on RTs throughout the 3 s analysis window, whereas the intercept was significantly lower than zero during the 1.25 s preceding the action (p < 0.05, linear mixed-effects model, Bonferroni corrected for multiple comparisons, n = 38; Fig. 4F).
Next, we turned to test our second prediction, namely, that the frontocentral EEG signal should reset after some, but not all beeps in the “single” and “cued” control tasks. Specifically, since the DV resets after every beep in the “single” task, and after every attended beep in the “cued” task, we reasoned that frontocentral EEG activity should be similar in these two conditions. In contrast, since the DV does not reset after unattended beeps in the “cued” task, it is expected to be closer to threshold compared with after single or attended beeps (Fig. 5A). We therefore compared frontocentral EEG activity following “single”, attended, and unattended beeps and found that in accordance with an appropriate resetting strategy, mean EEG activity in a 0.5–1 s time window following each beep was similar for “single” and attended beeps, whereas both were significantly different (closer to baseline and therefore less negative) than after unattended beeps (F(2,20517) = 14.43, p = 5.44 × 10−07, one-way ANOVA followed by a post hoc Tukey-Kramer test; Fig. 5B).
Frontocentral EEG resetting. A, Readiness variable dynamics in the two control tasks. All beeps in the “single” task, and attended beeps in the “cued” task reset the readiness variable, whereas unattended beeps in the “cued” task do not. Note the readiness variable's ramping slope was reversed compared with Figure 1C to more intuitively match the overall direction of the neural activity. B, Frontocentral EEG activity following “single” (blue; n = 19), attended (black; n = 20), and unattended (red; n = 20) beeps. The gray rectangle marks the statistical test's time window. C, Readiness variable dynamics in the “dual” task. Switch beeps (black) are beeps that are different from the one before. They signify that the intended response should change and that the readiness variable should partially reset to reflect the elapsed time since the previous beep. In contrast, “stay” beeps (red) are the same as the one before, and should therefore not affect the readiness variable. D, Frontocentral EEG activity following switch (black) and stay (red) beeps (n = 31). E, Readiness variable dynamics in the “dual” task following switch beeps (black) as a function of the interval from the last beep from the other train. The longer the interval, the less reset is needed and the closer the DV is to threshold. F, Frontocentral EEG activity following switch beeps split by the interval from the last beep from the other train (longer than 1 s: light blue; shorter: dark blue; n = 31).
A more nuanced prediction concerns the effect of beeps in the “dual” task. Unlike the “cued” task, all beeps in the “dual” task are potentially relevant, with a beep's impact on action timing determined by its relation to the preceding beep, regardless of which train it belongs to. Beeps from a different train than the previous one (“switch beeps”) reset timing and shift the intended action, while beeps from the same train (“stay beeps”) do not affect either the timing or the identity of the action (Fig. 5C). We therefore compared frontocentral EEG responses with switch and stay beeps in the same 0.5–1 s time window following each beep and found that similar to the comparison between attended and unattended beeps discussed above, here too there was a significant difference between resetting (switch) and nonresetting (stay) events, with switch beeps being followed by a significant return to baseline (t(16409) = −4.59, p = 4.42 × 10−06, t test; Fig. 5D).
Importantly, compared with the full reset in the control tasks, in the “dual” task switch beeps result in only a partial reset of the readiness variable, since they do not mark the beginning of a new interval, but instead the relevance of an interval whose starting point was a previously heard beep. Therefore, after a switch, the longer the interval since the previous beep in the other train, the smaller the change in the readiness variable's value before and after the switch, resulting in a reduced reset (Fig. 5F).
To test this prediction, we compared frontocentral EEG responses with switch beeps following short (<1 s, which is the mean ISI) versus long intervals since the previously heard beeps from the other train. In agreement with our prediction, we found that in the same 0.5–1 s time window following switch beeps, EEG signals after long intervals were significantly lower (closer to threshold) compared with after short intervals (t(5591) = −2.05, p = 0.040, t test; Fig. 5F).
In summary, our analyses revealed a strong correspondence between the dynamics of frontocentral EEG signals and the readiness variable, as both exhibited temporal scaling with RTs and resetting patterns. However, some differences were identified, most notably while the readiness variable decreases monotonically toward threshold between beeps, the EEG signal is non-monotonic, showing both positive and negative peaks. Additionally, instead of ramping immediately, the EEG signal exhibits a flat plateau following each beep. These differences raise the possibility that the dynamics of the readiness variable alone may not fully explain the observed EEG differences following different types of resetting events.
To determine whether readiness variable resetting alone could account for our observations, we performed EEG simulations, in which beep trains were translated to readiness variable's progression, which was in turn translated into modeled EEG signals as a concatenation of three response elements: an early stimulus-locked ERP lasting 0.5 s, a plateau of variable duration, and a linear negative deflection (RP, or readiness potential) lasting 1.5 s (see Materials and Methods). To reflect the finding that differences in RTs were driven by variations in plateau duration rather than the ramping rate or threshold of the RP (Fig. 4), we implemented two thresholds for the readiness variable: a plateau threshold, above which simulated activity remained capped at zero (plateau phase), and below which activity ramped negatively toward the response threshold (RP phase). Variability was implemented by randomizing the readiness variable's slope between beep onset and plateau threshold, while ramping from the plateau threshold to the response threshold remained noiseless (Fig. 6A). This yielded similar response scaling as observed in the real data (Fig. 6B).
Frontocentral EEG simulations. A, EEG simulations were conducted based on readiness variable dynamics. In this model, upon hearing a beep, the readiness variable ramps linearly from baseline to a plateau threshold and then continues toward a response threshold to trigger a decision. Variability in response times is explained by fluctuations in the initial slope of the readiness variable during the transition from baseline to the plateau threshold. Three example trials are shown with arrows indicating response times. B, To translate readiness variable dynamics into simulated EEG signals, each beep initially generated a 0.5 s early ERP waveform. The simulated EEG then remained at zero until the readiness variable crossed the plateau threshold, at which point a negative readiness potential emerged. If the process unfolded without interruption, a response was triggered. However, if another beep occurred, the signal reset, initiating a new pattern at the time of the reset. This panel presents EEG simulations of the three example readiness variables depicted in A. C, Readiness variable dynamics in the “cued” task: attended beeps (black) reset the readiness variable, while unattended beeps (red) do not. D, A schematic representation of simulated EEG activity following attended (black) and unattended (red) beeps in the “cued” task, based on the readiness variable dynamics shown in C. E, Simulated EEG activity following attended (black) and unattended (red) beeps in the “cued” task, as well as following all beeps in the “single” task (blue). The gray rectangle marks the statistical test's time window. F, Readiness variable dynamics in the “dual” task following switch beeps (black) as a function of the interval from the last beep from the other train. The longer the interval, the less reset is needed and the closer the readiness variable is to threshold. G, In the dual task, the dynamics of the readiness variable following switch beeps depend on the interval since the previous beep from the other train. Longer intervals (>1 s; light blue) lead to weaker resets of the readiness variable compared with shorter intervals (<1 s; dark blue). For clarity, the early stimulus-locked ERP components associated with short and long interval beeps are omitted. H, Simulated EEG activity following switch beeps split by the interval from the last beep from the other train (longer than 1 s: light blue; shorter: dark blue).
According to this model, the simulated EEG level following beeps is determined by the readiness variable: the more likely it is to fall below the plateau threshold, the more negative the signal becomes. Based on this, we predicted that since all beeps in the “single” task and attended beeps in the “cued” task trigger full resets, the readiness variable remains above the plateau threshold, resulting in a relatively high simulated EEG signal. Conversely, unattended beeps in the “cued” task do not trigger a reset, allowing for a nonzero probability that the readiness variable falls below the plateau threshold, leading to a more negative signal (Fig. 6C,D). Statistical testing confirmed that simulated signals were similar for single and attended beeps, with both significantly higher than unattended beeps (F(2,74294) = 1,658, p < 1 × 10−20, one-way ANOVA with Tukey–Kramer post hoc test; Fig. 6E). Importantly, while simulated EEG signals appear to plateau at negative values following unattended beeps, this does not reflect a shift in the plateau value. Instead, it arises from analyzing all beeps rather than only the final beeps in a train, where most beeps are followed by multiple additional resetting and nonresetting beeps. These successive events introduce further resets, giving the appearance of a plateau and preventing continuous ramping toward the response threshold.
A similar pattern was found in the “dual” task, where simulated EEG was significantly higher following switch compared with stay beeps (t(36187) = 20.25, p < 1 × 10−20, t test). Moreover, the magnitude of simulated EEG following switch beeps depended on the time elapsed since the last beep from the other train. When this interval was short, the probability of the readiness variable being below the plateau threshold was low, resulting in less negative activity. In contrast, longer intervals increased the likelihood of the readiness variable being between thresholds, leading to more negative activity (Fig. 6F,G). This was supported by significantly higher simulated EEG following switch beeps preceded by short (<1 s) versus long intervals from the last beep from the other train (t(16456) = 13.46, p < 1 × 10−20, t test; Fig. 6H).
All in all, the simulation results support our claim that the readiness variable alone can sufficiently explain the observed EEG resetting dynamics.
Temporal scaling of early ERP components
In addition to the negative correlation between the duration of the preceding interval and the EEG signal in the 0.5–1 s window, we also observed an opposite modulation of earlier EEG components in response to the beeps (Fig. 5F). However, since these signals were aligned on switch beeps, longer intervals from the previously heard beep from the other train (termed Tother) were associated with even longer intervals from the last beep from the same train (Tsame).
To resolve which of the two intervals was causing this modulation, we analyzed the dependencies of early EEG responses (within a 0.25–0.5 s time window following each beep) on Tsame and Tother using a linear mixed-effects in the three tasks.
Figure 7 shows that early EEG responses scaled positively with Tsame. This pattern was consistent across all beep types: attended (p = 9.10 × 10−13) and unattended (p = 8.34 × 10−08) beeps in the “cued” task, as well as all beeps in the “single” (p = 3.23 × 10−12) and “dual” (p < 1 × 10−20) tasks.
Scaling of early ERP components. A, EEG activity in the “dual” task (n = 31) following all beeps and split according to the time since the same-type (left) and different-type (right) beep. Rectangles mark the statistical test's time window; red shading corresponds to significant negative correlations, blue to significant positive correlations, and gray to nonsignificant correlations. Asterisks indicate significance level, *p < 0.05, ***p < 0.005, linear mixed-effects model. B, EEG activity in the “cued” task (n = 20) following all attended beeps and split according to the time since the same-type (left) and unattended (right) beep. C, EEG activity in the “cued” task (n = 20) following all unattended beeps and split according to the time since the same-type (left) and attended (right) beep. D, EEG activity in the “single” task following all beeps, split according to the time since the previous beep from the same side (n = 19).
Conversely, no such positive scaling was observed when analyzing the dependencies of the early EEG response on Tother. Instead, we identified negative scaling, but only following unattended beeps in the “cued” task (p = 0.002) and all beeps in the “dual” task (p = 8.59 × 10−10). This negative scaling may be viewed as a generalization of the results shown in Figure 5F. Following both unattended beeps in the “cued” task and all beeps in the “dual” task, the longer the Tother is, the closer the DV is to threshold. In contrast, attended beeps in the “cued” task cause full reset, and therefore the DV's value following such beeps is independent of Tother.
In conclusion, this component of the ERP represents a combination of both positive scaling with Tsame, regardless of its current task relevance, and a negative, task-specific scaling with Tother.
Discussion
We found that people are able to successfully track two asynchronous beep trains and correctly detect the first to terminate. Although not explicitly instructed to do so, they adopted an appropriate resetting strategy, in which each beep marked the starting point of a new interval, and decisions were made when sufficient time had elapsed since the last heard beep with little effect of the overall trial duration on the timing of action.
We propose that the multiple timing observed in this task is driven by a dynamic decision variable (DV) representing the elapsed time since the last relevant beep. This DV resets upon hearing a switch-beep, which also causes a change in the intended action. However, if sufficient time elapses without a switch beep, the DV reaches a threshold and triggers the intended action. Notably, this reset is only partial: following a switch, the DV does not start from zero but instead from the elapsed time since the previously irrelevant beep.
Frontocentral EEG activity closely mirrored the key characteristics of the DV's dynamics. When aligned on the beeps, EEG signals could be divided into three distinct segments, each corresponding to a different phase of the timing computation: (1) an early ERP component, (2) a sustained plateau period, and (3) the readiness potential.
Immediately after each beep, an early evoked potential appeared for ∼0.5 s. Consistent with the existing literature regarding frequency specific adaptation (Näätänen et al., 1989; Deouell and Bentin, 1998; Pereira et al., 2014), we found that this pattern was positively correlated with the time since the last identical beep (Fig. 7). Scaling was evident for all task conditions, indicating that whether a beep was exclusively attended (“single” and attended beeps in the “cued” task), mutually attended (“dual” task), or not attended at all (unattended beeps in the “cued” task), the neural response covaried with the duration since the same beep was last heard.
After these brief evoked potentials, activity reached a sustained plateau, which reflected the DV's value after reset (Fig. 5). It was closest to baseline following “single” and attended beeps in the “cued” task, which cause full reset; slightly lower following switch-beeps in the “dual” task, which cause partial reset; and lowest following unattended beeps in the “cued” task and stay beeps in the dual task which do not reset the DV at all. Interestingly, our findings revealed that the duration of the plateau was the primary factor underlying temporal scaling of EEG signals across varying response times, whereas the durations of both the preceding evoked potential and the subsequent readiness potential remained largely independent of response time (Fig. 4). This response pattern strongly suggests that, for timing long intervals (>1 s), trial-by-trial variations in the decision variable's rate are reflected in the duration of this specific segment of frontocentral EEG activity.
Finally, the last portion of the frontocentral activity pattern consisted of a sustained negative ramp-like activity, leading from the plateau to the choice time, corresponding to the well-established readiness potential (Schurger et al., 2021). Importantly, this pattern was centrally localized and symmetric for both actions. Action-specific, lateralized activity appeared significantly later, suggesting that EEG representations remain decoupled from the action until just before execution.
Taken together, our findings support the notion that during sensorimotor interval timing, action timing is encoded by a single, global pattern of brain activity. This activity pattern can flexibly adjust to perform various computations, such as resetting, attention keeping, and multiple timing, providing a rich and flexible repertoire of timing-related behaviors.
Data Availability
All the data and original code used are available upon request from E.L. (eran.lottem{at}mail.huji.ac.il).
Footnotes
This work was supported by TIMECODE ERC 852387 starting grant awarded to A.N.L. In addition, the brain attention and time lab (PI: A.N.L) is grateful for the support of the James McDonnell Scholar Award for understanding human cognition, ISF grants 958/16 and 1899/21. A.N.L is also grateful for the support of the Einstein-Center: Chronoi, Berlin. E.L. is the incumbent of the Sachs Family Faculty Development Chair in Brain Sciences.
↵*A.N.L. and E.L. contributed equally to this work.
L.Y.D. is the cofounder and shareholder of and receives compensation for consultation from InnerEye, a startup neurotech company. The company business is not related to the current study. L.Y.D. is the coinventor of Israel patent no. 256068 (2018), US patent no. 10,948,990 (2021), and US patent no. 10,694,968 (2021). The patents are not related to the current study. The authors declare no competing financial interests.
- Correspondence should be addressed to Eran Lottem at eran.lottem{at}mail.huji.ac.il or Ayelet Landau at ayelet.landau{at}mail.huji.ac.il.
