Segmenting and Predicting Musical Phrase Structure Exploits Neural Gain Modulation and Phase Precession

Stimuli

We selected 11 music pieces from the 371 four-part chorales by Johann Sebastian Bach (Breitkopf Edition, Nr. 8610). In the eighteenth century, chorales were congregational hymns performed in religious settings. The regularity and transparency of the harmonic structures of the Bach chorales facilitated our subsequent analyses in a frequency tagging paradigm to investigate musical phrase segmentation. In the selected music pieces, musical phrases are often marked by fermatas, indicating a pause or the prolongation of a note; the predefined phrasal structures of the selected pieces are shown in Table 1.

We then processed the original musical scores so that several confounds that may appear in EEG analyses could be avoided. We first ensured that each beat was always marked by a note onset, and hence the neural tracking of beat could be measured accordingly. Concretely, the whole and half notes were substituted by four or two quarter notes and the ties across notes were deleted so that they could be repeated. The material consisted thus mainly of quarter and eighth notes and a few sixteenth notes. More importantly, as fermatas in the original music pieces provided rhythmical or acoustic cues for phrasal structures (such as pauses between phrases and lengthened notes), we removed them so that temporal cues would be drastically limited for musical phrase segmentation.

Here, we investigate how listeners rely on chord progressions and abstract musical structures to parse music streams into phrases. In the original music pieces, there are two other nonstructural factors that potentially confound with structure-based musical segmentation: (1) The chords at the beginning and end of musical phrases, namely, onset and ending beats, are salient cues of phrasal boundaries; those boundary beats appear periodically as the phrasal structures are regular in the selected chorales (Table 1). Since listeners might lock to those salient boundary beats rhythmically to parse music streams, we need a control condition to tease out the effect of those boundary beats from the musical phrase segmentation based on harmonic progression. (2) In Western tonal music, the progressions between chords follow specific rules (i.e., the circle of fifths) that typically develop into cadences and lead to a closure feeling at the end of the musical phrase. The selected chorales follow those typical rules that are well known to listeners in Germany, surrounded by a Western tonal musical system. Therefore, identifying the phrasal structures of those chorales in their original condition does not present a challenging test on musical phrase segmentation. However, we can manipulate the selected materials in a way that, while basic musical contents (e.g., notes and tempi) are kept intact, a set of new harmonic progressions are generated, which are still plausible but less typical to Western listeners. If musical phrase segmentation can still be observed in those manipulated materials, this would provide additional evidence showing that listeners can parse music streams into phrases according to harmonic progressions, even when progressions are less familiar to listeners. For those two purposes, we employed a paradigm of temporal reversal that has been used in speech and language studies (Saberi and Perrott, 1999; Baldassano et al., 2017), although our reasons for using the reversal procedure here are different from such studies.

We created two reversal conditions. We first applied a local reversal procedure to the pieces. According to the phrasal structures defined in the musical scores, we kept the onset beat and the ending beat of each phrase intact but temporally reversed the order of the beats in the middle. For example, a phrase of eight beats [1 2 3 4 5 6 7 8] becomes a new phrase [1 7 6 5 4 3 2 8] after the local reversal. A schematic illustration of this procedure on the musical scores is illustrated in Figure 1A. The rationale for local reversal was as follows: as we kept the salient onset and ending beats intact but disrupted the harmonic progression, neural indices for tracking musical phrases and structural prediction should be lower in the condition of local reversal than in the original condition, supporting that intact harmonic progressions in the original pieces are used for phrasal segmentation and prediction. Alternatively, if the salient onset and ending beats are sufficient for listeners to parse continuous music streams into musical phrases, musical phrase segmentation should not differ between locally reversed pieces and the original pieces. To implement the local reversal procedure, we first used MuseScore 3 to generate audio files of the music pieces from the processed original scores. As the beats in the generated audio files had the exact same length at a certain tempo, we cut out each beat and then reversed the order of the beats while preserving the waveform of each beat (the note order in a beat was not reversed).

Second, we applied a global reversal procedure on the music pieces. For each piece, the order of all beats was temporally reversed. Figure 1A provides a schematic illustration of the procedure. This reversing procedure was implemented on the order of beats but not on the waveforms of beats. The chord progressions in the selected chorales follow the circle of fifths, a well-established structure in Western classical music and typical for listeners enculturated by Western tonal music (e.g., the participants recruited in Germany in the current study). In the reversed pieces, basic music content (e.g., tone scales and relationships between adjacent chords) is kept plausible, but the direction of chord progression is simply reversed. The reversed chord progressions, by no means, are random or lose any musical structures. The new (reversed) chord progressions, arguably, are less expected by listeners for this music genre but can still provide cues of phrasal opening and ending. In other words, the musical phrasal structures were preserved in a sense that the harmonic progressions in the reversed pieces can still be used to parse a phrasal structure. If musical phrase segmentation is still observed for globally reversed pieces, and if the neurophysiological tracking of musical phrases is comparable to the original pieces, it can be concluded that listeners do extract harmonic progressions to parse music streams into phrases and predict phrasal boundaries, even though the chord progressions are less familiar to the participants.

We need to emphasize that the reasons for using the reversal manipulations here differ from the reasons for the reversal procedures used in studies on spoken language and movies (Saberi and Perrott, 1999; Hasson et al., 2008; Lerner et al., 2011), where the reversal procedures of different timescales were used to completely disrupt the information within corresponding timescales. In the case of music, using random sequences by scrambling the order of beats cannot be used as a control condition, since compromising musical structure entirely leads to losing the “musical” character of the material (at least from this time period) and simply prevents direct comparison between conditions. Indeed, the current study focuses on the role of the harmonic progressions in musical segmentation and prediction and is specifically testing tonal material. Therefore, both control conditions contain musical structure while contrasting with the original condition. Note that the local reversal condition was designed to preserve the boundary beats and their temporal regularity, so the observation of certain neural responses locking to the boundary beats in the local reversal condition should not be taken as a failure to disrupt all musical structures.

We generated the music audio files in MuseScore 3 for three tempi, 66, 75, and 85 bpm (beats per minute), which arguably cover a reasonable range of tempi of actual recordings of the Bach chorales (the tempo range we observed in a set of 45 interpretations by voice, organ, or piano). Choosing three tempi, instead of only one, also enabled us to investigate how musical phrase segmentation varied with tempo. Furthermore, if phrase and beat tracking are observed in the music pieces of different tempi at different frequencies of EEG signals, this aids in validating the neural measurement—neural tracking of music structures is not constrained to a specific tempo or a frequency range of EEG signals but aligns with musical structures.

In summary, we had three reversal conditions—original (without reversal), global reversal, and local reversal—and three tempi: 66, 75, and 85 bpm. This generated a 3-by-3 experimental design. We used the music piece number 0, BWV 255 (Table 1), as the training piece before the formal experiment to familiarize participants with the experimental materials and the task. The 10 remaining pieces were included in the data analyses. In total, we presented 90 music pieces (3 conditions * 3 tempi * 10 pieces). The sampling rate of audio files was 44,100 Hz and the amplitude of the audio files was normalized to 70 dB SPL by referring the music materials to a 1 min white noise piece that had the same sampling rate and was measured beforehand to be 70 dB SPL at the experimental setting.

Acoustic analyses of the music material

To characterize temporal dynamics of the music pieces, we derived the amplitude envelopes of the musical materials. We filtered the audio files through a Gammatone filterbank of 64 bands logarithmically spanning from 50 to 4,000 Hz. The amplitude envelope of each cochlear band was extracted by applying the Hilbert transformation on each band and taking the absolute values (Glasberg and Moore, 1990). The amplitude envelopes of 64 bands were then averaged and downsampled from 44,100 to 100 Hz to match the sampling rate of the following processed EEG signals for further analyses [i.e., cerebral-acoustic coherence and temporal response function (TRF); see below].

We transformed the amplitude envelopes to modulation spectra so that the canonical temporal dynamics of the music pieces can be concisely shown in the spectral domain. We first calculated the modulation spectrum of the amplitude envelope of each music piece using FFT with zero-padding of 8,000 points and took the absolute value of each frequency point. As each piece had a different length and the zero-padding caused the modulation spectra of different music pieces to have different ranges of magnitude, we normalized the modulation spectrum of each piece by dividing the norm of its raw modulation spectrum. We then averaged the normalized modulation spectra across the 10 pieces for each condition at each tempo. At each tempo, the average modulation spectra across three conditions were highly similar, so we only showed the modulation spectra averaged across the three conditions for each tempo (Fig. 2A). Nonetheless, to demonstrate that the differences of the amplitude modulation spectra between the three conditions were not noteworthy, we calculated at each tempo the standard deviation over the modulation spectra of the three conditions (Fig. 2B). The music pieces of the three conditions showed the standard deviations close to zero below the beat rate. In addition, we further tested the acoustic differences between the reversal conditions by conducting a one-way repeated-measures ANOVA at each frequency point for each tempo, with nine pieces of eight beats per phrase as “subjects” and the reversal condition as the main factor. For this analysis, we left out the piece of 12 beats per phrase, which has a different phrase rate than the other 9 pieces. Consistent with what is shown in Figure 2B, we found no significant effects at any frequency point for each tempo (p > 0.05), so the neural tracking of phrasal structures found later cannot be caused by the acoustic differences between the music stimuli of the different reversal conditions.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Acoustic analysis and MCCA. A, Acoustic analyses. We extracted the amplitude envelopes of the music pieces and calculated the modulation spectra averaged over 10 music pieces of each tempo in each reversal condition. The modulation spectra of the three reversal conditions highly overlapped and were therefore plotted together. Note that there are no salient spectral peaks around the phrase rates (0.138, 0.156, and 0.177 Hz). B, The standard deviation across the three reversal conditions at each tempo is tiny. In terms of the acoustic spectral components, the three conditions are comparable at each tempo. This shows that the acoustic properties of the music pieces do not differ and should not contribute to the difference in phrasal segmentation between the reversal conditions. C, MCCA components of the local reversal condition at the tempo of 65 bpm. This provides an example of MCCA and shows that the first component of MCCA stands out and explains a considerable amount of variance. D, Amplitude spectra of first three MCCA components of the local reversal condition at the tempo of 65 bpm. We calculated the amplitude spectrum of each MCCA component in C to examine the spectral component of each component. The spectrum of the first component shows amplitude peaks corresponding to the beat rate and the note rate (the first harmonic of the beat rate). This further supports our choice of analysis—specifically, the first MCCA component, but not other components, contained the neural signals induced by beat and note structures in the music pieces. Furthermore, this is consistent with the topography in the inset, showing that the topography reflects auditory responses to beats and notes. Similar results are shown for other tempi and other reversal conditions.

Experimental protocol and EEG recording

EEG data were recorded using an actiCAP 64-channel, active electrode set (10–20 System, BrainVision Recorder, Brain Products, brainproducts.com), at a sampling rate of 500 Hz, with a 0.1 Hz online high-pass filter (12 dB/octave roll-off). There were 62 scalp electrodes, one electrode (originally, Oz) was placed on the tip of the nose. All impedances were kept below 5 kΩ, except for the nose electrode, which was kept below ∼10 kΩ. The auditory stimuli were delivered through plastic air tubes connected to foam earpieces (E-A-RTone Gold 3A Insert Earphones, Aearo Technologies Auditory Systems).

The experiment included a training session and a testing session. In the training session, we presented the three conditions of the piece BWV 255, at the three tempi, to the participants, who were instructed to answer a question after listening to each piece: “how do you like this music piece.” The participants rated the music pieces using a 6-point scale from 1 to 6, with 6 being the most positive rating and 1 being the most negative rating. This behavioral task was designed primarily to keep the participants engaged and to make them pay attention to music pieces and is thus not reported here. In each trial, before each piece was presented, the participants were required to focus on a white cross in the center of a black screen. After 3–3.5 s of silence, a piece was presented, and the participants were instructed to keep their eyes open while listening to it. After each piece ended, the question appeared on the screen and the participants rated the piece. The next trial started right after the participants’ responses. The order of music pieces was randomized between participants. The behavioral data and EEG signals were not recorded during the training session.

The testing session followed the training session. We presented the 90 music pieces in six blocks to the participants while they were undergoing EEG recording. Fifteen pieces were presented in each block. The order of the music pieces was randomized within and across the blocks, and a different order of pieces was presented to each participant. The trial structure was the same as in the training session. After each block, the participants could choose to take a short break of ∼1–3 min or to initiate the next block. Behavioral ratings are not analyzed in this study, as our main goal here was to investigate how the brain segments high-level musical structures.

MCCA

We focused on the neurophysiological signals evoked by the musical pieces (auditory stimuli). To extract EEG signals that largely reflect auditory-related neural responses, instead of arbitrarily selecting certain electrodes (e.g., Cz or FCz), we deployed MCCA to extract shared EEG components across all the participants. The participants were listening to the same set of pieces and hence similar neural responses to the music pieces should be observed across participants, although each participant's EEG recording varied with his or her head size and EEG cap position and with different sources of noise. MCCA, briefly, first implements a principal component analysis (PCA) on each participant's EEG recordings and then pools all the PCA components across the participants to conduct another set of PCA. The components from the second PCA reflect shared components of neural responses across the participants that are invariant to EEG cap position and individual head sizes. Detailed procedures and further explanations can be found in de Cheveigne et al. (2019). This procedure of component extraction simplified further analyses and avoided biases introduced by arbitrary EEG channel selections and by differences of EEG cap positions and head sizes across participants.

We first sorted each participant’ EEG recordings according to the music piece number and concatenated EEG recordings of each condition and each tempo after epoching to form one long trial. For example, the EEG recording for the piece 1 (BWV153/1) was put in the beginning and was followed by the EEG recordings of the piece number 2, 3, 4, 5, 6, 7, 8, 9, and 10, sequentially. The 3 s pre- and poststimulus periods were included for each trial. For each condition and each tempo, we applied MCCA to derive 50 components and selected the first component, as the first component explained the most variance (Figs. 1C, 2C). We checked MCCA weights over EEG channels and plotted them as topographies to check whether the first component is related to music-related auditory responses. We showed one condition (local reversal at 66 bpm) as an example in Figure 2C,D, from which it can be seen that the topography of MCCA weights resembled typical topographies of auditory EEG responses. To further validate our component selection, we calculated the power spectra for the first three components of MCCA of all conditions and showed an example in Figure 2D (local reversal, 66 bpm). We zero-padded the time series of each component to 8,000 points (8,000 points correspond to a stimulus duration of 80 s) and calculated power spectra using FFT. We calculated the absolute value for each frequency point in the amplitude spectrum obtained from the FFT and then normalized the amplitude spectrum by its norm. Figure 2D shows that only the first component showed amplitude peaks corresponding to the frequency of the beat rate of the corresponding tempo. This demonstrated that the first component of MCCA indeed represented neural responses to music pieces. All the other conditions showed similar patterns as the example in Figure 2C,D.

After conducting MCCA for each condition and each tempo and extracting the first component, we projected the component back to each participant and derived one long “trial” including the 10 music pieces for each participant. We then cut out the neural response for each piece from the long trial. As PCA sometimes reversed polarity of EEG signals, the polarity of the derived signals was manually checked and corrected for each participant.

Note that we performed the MCCA separately on the 10 pieces in each condition at each tempo because we hypothesized that the neural responses to music could be affected by both the reversal conditions and the tempo. If we performed the MCCA only once over all the conditions (all the 90 pieces), the MCCA components would reflect a mix of weights from all conditions. In other words, the MCCA components would reflect weights from all conditions—this would mean that the information about the reversal manipulations and tempi would be spread across all conditions and comparisons between conditions would be skewed (e.g., one condition may have very high magnitude of neural responses and this condition would drive the weights of the MCCA whereas other conditions contribute less to the MCCA). Therefore, we decided instead to conduct MCCA on each condition to prevent the extracted neural responses of each condition from being contaminated by other conditions. In this way, one MCCA component was independently derived from each condition without being biased by other conditions.

Cerebral-acoustic coherence (Cacoh)

We calculated the coherence between the neural signals and the amplitude envelopes of music pieces in the spectral domain—cerebral-acoustic coherence (Cacoh)—to investigate how pieces of different conditions were tracked in the brain at different neural frequencies. The rationale is that, if the music pieces of one condition can be more reliably tracked by the auditory system than the pieces of the other conditions because of their specific musical structures or tempi, this difference can be observed in the magnitude of coherence between the neural signals and the amplitude envelopes. This measurement, Cacoh, has been used in neurophysiological studies on speech perception (Peelle et al., 2013; Doelling et al., 2014), which quantified how phase-locked neural responses correlate with acoustic signals—at what frequencies and how strong the neural signals and the acoustic signals correlate with each other. In essence, Cacoh calculates cross-spectrum coherence between neural signals and acoustic signals; here we calculated magnitude-squared coherence using the function “mscohere” in MATLAB 2016b (https://de.mathworks.com/help/signal/ref/mscohere.html). This allowed us to control the temporal window used to calculate the cross-spectrum coherence and the spectral resolution of the coherence values so that the temporal window sizes and the spectral resolution were matched across all the conditions. This helped avoid the biases of calculating spectral coherence over the entire music pieces introduced by different lengths of the music pieces. Nonetheless, we still referred to our calculation as “Cacoh” as this term has been well adopted in the literature.

We calculated Cacoh for each music piece and each participant using the neural data and the amplitude envelope derived between 1 s after stimulus onset and 1 s before offset, to minimize the influence of onset and offset neural responses of a whole musical piece. The temporal window used was a Hanning window of 1,024 points with an overlap of 512 points (50% of overlap between adjacent temporal windows). We constrained the frequency range from 0.1 to 20 Hz, with a step of 0.05 Hz. The inputs, the neural signals, and the amplitude envelope had a sampling rate of 100 Hz. We then averaged Cacoh values across the 10 pieces within a condition. Therefore, a Cacoh value was derived for each tempo and each condition at each frequency point.

Temporal response function

To investigate the temporal dynamics of the phase-locked responses to the music pieces, we calculated the reverse correlation between the neural signals and the music amplitude envelopes in the temporal domain—TRFs. TRF has been commonly used to investigate how the auditory system codes acoustic and linguistic features in neurophysiological studies on speech perception (Di Liberto et al., 2015; Brodbeck et al., 2018; Broderick et al., 2018). The rationale for this analysis is similar to Cacoh: if music pieces of a condition robustly evoke auditory responses, such robust auditory responses should be reflected by TRF in the time domain. Furthermore, we could extract temporal information from TRFs on when the auditory responses are modulated by different conditions. For example, TRFs may reflect differences between tempi within the first 100 ms after the onset of an auditory event while TRFs may differ between the conditions after 100 ms because of different manipulations on the musical structures. Early auditory responses likely reflect sensory processes whereas late responses tend to reveal high-level cognitive processes in music perception (Koelsch et al., 2002, 2005; Koelsch and Siebel, 2005).

The TRF was derived from the amplitude envelopes of stimuli (S; for details see Acoustic analyses of the music material) and their corresponding EEG signals (R; for details see above, EEG preprocessing and data analysis) through ridge regression with a parameter (lambda) to control for overfitting and with M to enforce a smoothness constraint by penalizing the difference between adjacent time points (Lalor et al., 2006; Crosse et al., 2016; superscript t indicating transpose operation):TRF=(StS+λM)−1StR

We calculated a TRF for each music piece for each participant using the neural data and the amplitude envelopes between 1 s from the stimulus onset to 1 s before the offset so that the influences of the onset and offset neural responses were avoided. The TRFs were calculated from 200 ms before onsets of auditory events and 500 ms after. As the highest tempo was 85 bpm, the shortest interbeat interval of the music pieces was ∼700 ms and the internote interval was <400 ms. In the statistical analysis, we focused on the period of TRFs from −100 to 400 ms but showed the TRFs from −200 to 500 ms. The EEG signals and the amplitude envelopes of music pieces were not filtered or decomposed into different frequency bands. We fixed the lambda at 0.1 for calculating TRFs across all the conditions and tempi, and hence the differences of TRFs across conditions should not be because of different lambda values. After calculating TRF for each music piece, we averaged TFRs over 10 pieces of one condition for each participant and conducted statistic tests on the mean TFRs across different conditions. The TRFs were calculated using the multivariate temporal response function (mTRF) toolbox (Crosse et al., 2016).

The fixed lambda value of 0.1 was decided empirically. The lambda value decides how signals are temporally smoothed during TRF calculation and makes sure that the encoding model does not overfit to noise: the larger the lambda value is, the signals are smoothed to a larger extent temporally. If zero is chosen for the lambda, the inversion of matrices cannot be implemented as sometimes the covariance of neural responses recorded here is not fully ranked. Therefore, we chose 0.1 as the smallest value of the lambda to make sure that the inversion can be implemented, and the signals were not smoothed seriously. We believe that the choice of the lambda value should not affect our results, as we compared TRFs across the conditions instead of drawing conclusions from the absolute TRF values and the choice of the lambda value exerted an equal influence on TRFs of all the conditions.

Temporal modulation of EEG power

Here we aim to investigate musical phrase segmentation beyond the level of the beat structure. The lengths of the music phrases ranged from ∼7 s (8 beats per music phrase at 65 bpm) to ∼5 s (8 beats per music phrase at 85 bpm), which corresponded to an ultralow EEG frequency range of ∼0.13 to ∼0.18 Hz. This ultralow EEG frequency range has proved to be challenging to analyze because of the 1/f shape of power spectrum of EEG recording and hence a low signal-to-noise ratio at the ultralow frequency range. Another fact is that the time-frequency analysis requires an ultra-long temporal window for such frequency ranges (i.e., two cycles for 0.13 Hz, which is ∼15 s). Previous studies have not yet shown a meaningful and robust neural signature within this range. Given the complexities mentioned earlier, it is indeed challenging to directly observe temporal dynamics of musical phrase segmentation and its corresponding frequency components in EEG signals.

To circumvent the challenge of analyzing the ultralow frequencies of EEG signals and to directly observe temporal dynamics of musical phrase segmentation, we resorted to a different strategy: The musical phrasal structures likely modulated the power of neural responses to each beat. The beat rates of the music pieces were above 1 Hz, at which neural signals can be well recorded by EEG. Therefore, we chose to measure the temporal modulation of EEG power at the beat rate to investigate musical phrasal tracking. If listeners do track musical phrases and the neural responses to beats are modulated by musical structures, we should be able to observe neural signatures reflecting musical phrase segmentation through temporal modulation of EEG power at the beat rates. This procedure is akin to recovering low-frequency amplitude envelopes from high-frequency carriers in speech signal processing (Teng et al., 2019). For instance, in speech signal processing, the amplitude envelopes of low-frequency ranges (e.g., <30 Hz) can be extracted from carrier frequencies above 100 Hz. Essentially, the temporal modulation of EEG power denotes the oscillatory pattern of EEG signal modulation. In signal processing terms, the bandpass filter operates on carrier signals (raw EEG signals), and the slow oscillatory pattern is the envelope of amplitude modulation on these carrier signals.

We first conducted a time-frequency analysis on EEG signals to derive power spectrograms of neural responses to the music pieces. The single-trial data derived from MCCA were transformed using the Morlet wavelets embedded in the FieldTrip toolbox, with a frequency range from 1 to 35 Hz in steps of 1 Hz and a temporal range from 3 s before the onset of music pieces to 3 s after the offset of music pieces in steps of 100 ms. To balance spectral and temporal resolution of the time-frequency transformation, from 1 to 35 Hz, the window length increased linearly from 3 cycles to 10 cycles. Power (squared absolute value) was extracted from the wavelet transform output at each time-frequency point. Moreover, the power values for each trial were normalized by dividing the mean power value over the baseline range (−1.5 to −0.5 s) and taking logarithms with base 10 and then was converted into values with unit of decibel by multiplying by 10.

We next quantified temporal modulation spectra of EEG power to examine at which frequency the EEG power was modulated and whether the salient modulation frequencies were related to the frequency range of musical phrasal structures. This procedure is similar to the calculations of acoustic modulation spectra of the music pieces (Fig. 2A). We calculated the modulation spectrum of EEG power at each frequency using FFT with zero-padding of 20,000 points and took the absolute value of each frequency point, so that the spectral resolution of the modulation spectra was 0.005 Hz. The high spectral resolution guaranteed to resolve different music phrase rates (8 beats per music phrase: 0.1375 Hz at 65 bpm; 0.1562 Hz at 75 bpm; 0.1771 Hz at 85 bpm). We then normalized the modulation spectra of each trial by dividing the modulation spectrum of each trial by the length of its corresponding music piece. As the music pieces of each condition had different lengths, they could not be summed together in the temporal domain, we chose to sum the modulation spectra (complex numbers) of 10 pieces within each condition in the spectral domain. This spectral summation on complex number kept phase information, and the shared frequency components with similar onset phases across the 10 pieces in a condition were emphasized. After the spectral summation, we took the absolute value of each frequency point to derive the modulation spectrum for each condition. The above procedure resulted in a two-dimensional modulation spectrum for each condition, with one dimension as the frequency of EEG power and the other dimension as the frequency of temporal modulation of EEG power. It should not matter for the following statistic tests in this case whether we summed the modulation spectra or averaged the modulation spectra over the music pieces, as both summation and averaging are linear operations.

To examine how the musical phrase segmentation varied with the tempo of music pieces, we averaged over three reversing conditions within each tempo and showed the modulation spectra of three tempi, respectively (Fig. 4A). If the neural signatures of musical phrase segmentation varied with the tempo, this helped validate the observed neural signatures. We also checked whether other frequency ranges of EEG power (i.e., the alpha band, 8–12 Hz, and the beta band, 13–30 Hz) showed signatures of musical phrase segmentation, as the previous literature reported an important role of the neural beta band in music processing (Doelling and Poeppel, 2015; Fujioka et al., 2015).

We next focused on the frequencies of EEG power at the beat rates of three tempi, as we only observed salient modulations of EEG power ∼1 Hz (Fig. 4A), which was close to the beat rates of three selected tempi. We conducted once again the time-frequency transformations on the EEG signals, but only at the beat rates, 1.1001 Hz (65 bpm), 1.25 Hz (75 bpm), and 1.4165 Hz (85 bpm). The window length in the time-frequency transformations was three cycles. Other procedures and parameters were the same as the above analyses of calculating the EEG power and the modulation spectrum of each tempo and each reversing condition. We selected the third music piece of the original condition (BWV267), which had the longest duration, and showed the temporal dynamics of EEG power at the beat rate of each tempo (Fig. 4B). This directly showed how the EEG power was modulated by the musical phrasal structures. For each participant, we summed the modulation spectra of 10 pieces within a condition in the spectral domain and conducted statistics on the summed modulation spectrum of each condition (Fig. 4C).

Surrogate test on modulation spectra of EEG power

The modulation magnitudes of neural signals of musical phrase segmentation cannot be directly compared between different tempi, because the music pieces of different tempi have varying lengths, and the tempi bias estimation of the modulation spectra. As a result, frequency components in the lower frequency range tend to have greater amplitude due to the 1/f nature of neurophysiological signals. Therefore, individual baselines need to be constructed for the modulation spectra of different tempi; then the modulation spectra can be normalized according to the baselines within each tempo. Therefore, we conducted surrogate tests in the spectral domain to derive a null distribution of the modulation spectrum for each condition and each tempo. This served two purposes—we can directly test the significant frequency ranges of musical phrase segmentation using the null distributions and then normalize the modulation spectra to do comparisons across conditions.

The null hypothesis of the surrogate test here is that the neural signals did not track or phase-lock to the musical phrasal structure, and hence the sum of modulation spectra across 10 pieces of each condition should not differ from the surrogate modulation spectra derived from jittering the temporal alignment between each of the 10 music pieces and its neural recording temporally. However, it is not appropriate to conduct this surrogate test in the time domain. As the lengths of musical phases (>5 s) were much longer than the pre- and poststimulus periods (3 s), temporal surrogation cannot efficiently disrupt the temporal correspondence between neural signals and musical phrasal structures—the jittering can only happen within a cycle of musical phrase. More importantly, the tempi here still affect the procedure of surrogation—for example, jittering within a temporal range of 6 s at 85 bpm has a different jittering effect from the same jittering range at 65 bpm because of different lengths of musical phrases at the two tempi. Therefore, we conducted the surrogate test in the spectral domain after Fourier transformation of EEG power.

After calculating the modulation spectrum of the EEG power of each music piece, we multiplied the modulation spectrum (complex numbers) with a complex number that has a norm of 1 but a randomly generated phase from 0 to 2 * pi. This multiplication reset the onset phase of the EEG power of each music piece so that the EEG power of each music piece now started with a random phase, whereas the amplitude magnitude of the modulation spectrum was kept intact. We then derived the summed modulation spectrum of the surrogated data for each tempo and each condition. By doing this, the 10 pieces within a condition all had different onset phases, and therefore the temporal components locked to the musical phrasal structures should be averaged out or severely smoothed. This surrogating procedure on phase in the spectral domain was invariant to different tempi, as the phase values were normalized to different tempi. We repeated this procedure 1,000 times and derived a null distribution of modulation spectra for each condition and each tempo and for each participant.

We averaged the modulation spectra of each surrogation across participants and derived a null distribution of the group-averaged modulation spectrum for each condition and each tempo. We chose a one-side alpha level of 0.01 as the significant threshold for each condition and each tempo (Fig. 4C, dashed lines). The frequency ranges where the empirical modulation spectra are above this threshold can be considered as the frequency ranges showing robust effects of musical phrase segmentation. To derive one frequency range across three conditions for one tempo so that the frequency range selected is unbiased to each condition for the following analysis, we chose the lower bound and the upper bound of the significant frequencies among all the three conditions and used them to define one frequency range for all the three conditions. The significant frequency ranges are as follows: at 65 bpm, 0.125–0.15 Hz; at 75 bpm, 0.14–0.175 Hz; at 85 bpm, 0.165–0.2 Hz.

Normalization of modulation spectra of EEG power

From the surrogate tests above, we obtained the null distributions for the modulation spectra of EEG power. As mentioned, using the null distributions we could normalize the raw modulation spectra that were biased because of different tempi of music pieces. Here, we calculated the mean of the null distribution for each condition and each tempo and subtracted out the mean of the null distribution from the raw modulation spectrum.

Temporal response function of EEG power envelope with phrasal boundaries as regressor

In the above spectral analysis, we characterized the frequency components reflecting musical phrase segmentation. What are the temporal trajectories of musical phrase segmentation? Is the musical phrase segmentation a result of neural signals locking to the onset beat or to the offset beat of a musical phrase? Spectral analyses cannot provide such temporal information. Additionally, the length of the chosen music pieces varies, resulting in differing precision in the estimation of fundamental frequency and harmonics in the spectrum. Normalizing the spectral magnitude between pieces of different lengths is also challenging. Conversely, TRF analysis, which is not affected by stimulus length and does not require normalization between stimuli, is more appropriate for this study. Therefore, we calculated TRFs of EEG power using the phrasal boundaries of music pieces as the regressors.

The TRFs of EEG power envelopes were calculated in a similar way as in the calculation of TRF of boundary beats of musical phrases. We marked the boundaries between two phrases in each music piece. For example, there are seven phrases in the music piece of BWV153/1 (Table 1, Piece Number 1), and hence there are six phrase boundaries; these six phrase boundaries served as a regressor for the EEG power envelope to derive a TRF for this piece. After deriving the EEG power envelopes at each tempo, we calculated TRF for each music piece from 1 s after stimulus onset to 1 s before stimulus offset. The length of TRF estimation was 6 s, with 3 s before the phrase boundary and 3 s after. We then averaged TRFs of 10 pieces within a condition of each tempo. The lambda was set as 0, which indicated that no smoothing was applied and the calculation of TRF was equal to a reverse correlation.

The TRFs of EEG power are shown in Figure 4E. It can be seen that the peak latencies for different conditions differ. To determine the latency of each condition and to find the peak time point, we fitted each group-averaged TRF using a one-term Gaussian model, whose center point determined the peak latency of each group-averaged TRF.

Phrasal phase precession and phrase segmentation at group-average level

The original music pieces are well structured, and a listener can readily extract the temporal regularity of the phrase structures and predict the unfolding of the musical pieces. In contrast, the reversed conditions may preserve the salient onset and offset beats of musical phrases or expected chord progressions but reduce the structural cues to the phrases, so listeners experience difficulties to predict the unfolding of the pieces. To test this conjecture, we tried to quantify phase precession of neural responses on the level of musical phrases—phrasal phase precession (PPP), as phase precession of neural signals has been argued to reflect the prediction of future events (Jensen and Lisman, 1996; Lisman, 2005). The rationale here is that, if listeners can predict the incoming musical contents, the neural phase should proceed faster as the music unfolds. On the other hand, if listeners cannot predict the incoming music contents but passively respond to or are solely driven by the acoustic events in music, the neural phases would proceed at a constant speed, as the speed/tempo of the acoustic events is constant in each music piece.

To quantify phrasal phase precession, we need to conduct our analysis on the neural response to each music piece at a single-trial level. However, we conducted EEG recording here, and the EEG recording at a single-trial level often has a low signal-to-noise ratio; usually at least tens to hundreds of repetitions are needed for analyzing neural responses to a single stimulus. To resolve this issue, we took another strategy for quantifying phase precession for each music piece—we first averaged each trial across 29 participants and then conducted our analyses on the group-averaged trial. Although such a strategy prevented us from probing individual differences, the process of the group averaging did aid in emphasizing the neural components induced by the music pieces while averaging out neural signals unrelated to the music listening. If a music piece is predictable on the level of musical phrase segmentation, the phrasal phase precession should be observed for all the participants to various degrees. Therefore, the phase precession quantified from the group-averaged trials could collectively indicate that the participants predict the unfolding of each music piece as well as reflect the predictability of each music piece.

The above analyses extracted the EEG power envelopes reflecting the musical phrase segmentation and determined the significant frequency ranges of musical phrase segmentation at each tempo. We averaged the power envelopes of each music piece across 29 participants and employed a two-pass second-order Butterworth filter embedded in the FieldTrip toolbox to filter the group-averaged envelopes, using the significant frequency range of each tempo as the cutoff frequencies. The phases of the averaged power envelope of each music piece were extracted by applying the Hilbert transformation on the filtered signal and taking the angle at each time point. We showed in Figure 5A (left panel) the phase series of three conditions of the music piece, BWV267, at 75 bpm, as this piece has the largest length. We also plotted the phase values at the phrase boundaries in Figure 5A (right panel) so that it can be clearly seen that the proceeding speed of the neural phases was increasing in the original and global reversal conditions as the music piece unfolded.

We constructed an index, phrasal phase precession index (PPPi), to quantify phrasal phase precession over each music piece. If there is no phase precession, the neural phase series would proceed in a step of 2 * pi over every musical phrase. This is to say that, if we fit a line between the musical phrasal boundaries and their corresponding neural phase values, the slope will be exactly 2 * pi if there is no phase precession. In contrast, if the neural phase series is accelerating as the music piece is unfolding, this line will be steeper and the slope will be larger than 2 * pi; if the phase series is slowing down, the slope will be smaller than 2 * pi. Therefore, the differences between the slope of the fitted line and 2 * pi can represent whether there is phase precession and to what extent the phase series is accelerating or slowing down. The differences here were termed as PPPi. We provide an example in Figure 5C, where we show the lines fitted between the music phrase boundaries and their corresponding neural phase values in three reversing conditions at 75 bpm.

The peak frequencies of modulation spectra of EEG power correlate with PPPi, as the phase precession can be considered as frequency modulation or phase modulation—the frequency of musical phrase segmentation is modulated along the time—and the mean frequency over the whole music piece becomes larger than the phrase rate if the phrasal phase precession happens. Therefore, here, we also quantified the peak frequencies of the modulation spectrum of EEG power within the significant phrase segmentation range for each music piece, so that the results of PPPi can be further validated by a different index. In the sense of frequency modulation, PPPi reflects linear phase changes. For example, if the phrase rate is 0.1 Hz (10 s per circle) and the neural frequency is 0.105 Hz (∼9.5 s per circle), the phase shift, measured by time, will be 0.5 s every phrase. If we use the phrase rate as the reference frequency (which was the case here), the phase shift will be 0.5/10 * 2 * pi every phrase (0.1 * pi). Admittedly, neural dynamics may not follow such a linear manner strictly. In future studies, neural recordings of high signal-to-noise ratios, such as intracranial recordings, can be used to measure the timing of each neural peak around the phrasal boundary.

PPPi, as stated above, reflects the value of a shifted phrase every phrase (phase shift per phrase). As the music is unfolding, the amount of shifted phase accumulates. The accumulated phase shift correlates with the length of the music piece, so it is not a good measurement and is affected by the music piece length, whereas PPPi can be viewed as an index normalized by the music piece length, which also directly reflects how the prediction is built up as the music is unfolding. For the longest music piece at 75 bpm, the PPPi value is ∼0.1 (Fig. 5E); the accumulated phase shift is 0.9 after nine phrases, which is ∼0.29 * pi and is close to the length of one beat in the phase domain (0.25* pi) − 2 * pi covers one phrase, which includes eight beats; a PPPi value of 0.25 * pi represents that neural signals shift forward by one beat. Asymptotically, the accumulated phase shift will amount to 2 * pi (eight beats, a whole phrase), if the wrapped phase is used and its value range is constrained to be between 0 and 2 * pi. For example, if PPPi for an extremely long music piece is 0.1, after 62 stationary phrases (∼8 min), the accumulated phase shift will reach 2 * pi. It would be interesting to test the maximum value of PPPi using longer music pieces; it is likely that PPPi values would asymptote to a few beats per phrase.