Speech Categorization Reveals the Role of Early-Stage Temporal-Coherence Processing in Auditory Scene Analysis

Stimulus generation

The stimuli used in the present study draw from and expand on the materials and methods previously described in Viswanathan et al. (2021b). Twenty consonants from the Speech Test Video (STeVi) corpus (Sensimetrics) were used. The consonants were /b/, /t∫/, /d/, /ð/, /f/, /g/, /dʒ/, /k/, /l/, /m/, /n/, /p/, /r/, /s/, /∫/, /t/, /θ/, /v/, /z/, and /ʒ/. The consonants were presented in consonant-vowel (CV) context, where the vowel was always /a/. Each consonant was spoken by two female and two male talkers (to reflect real-life talker variability). The CV utterances were embedded in the following carrier phrase: “You will mark /CV/ please” (i.e., in natural running speech).

Stimuli were created for five experimental conditions: (1) Speech in quiet (SiQuiet): Speech in quiet was used as a control condition. (2) Speech in speech-shaped stationary noise (SiSSN): Speech was added to stationary Gaussian noise at −8 dB signal-to-noise ratio (SNR). The long-term spectra of the target speech (including the carrier phrase) and that of stationary noise were adjusted to match the average (across instances) long-term spectrum of the four-talker babble. A different realization of stationary noise was used for each SiSSN stimulus. (3) Speech in babble (SiB): Speech was added to four-talker babble at −8 dB SNR. The long-term spectrum of the target speech (including the carrier phrase) was adjusted to match the average (across instances) long-term spectrum of the four-talker babble. Each SiB stimulus was created by randomly selecting a babble sample from a list of 72 different four-talker babble maskers obtained from the QuickSIN corpus (Killion et al., 2004). (4) Speech in a masker with only DC modulations (SiDCmod) (Stone et al., 2012): In line with the procedure described in Stone et al. (2012), the target speech was filtered into 28 channels between 100 and 7800 Hz, and a sinusoidal masker centered on each channel was added to the channel signal at −18 dB SNR. To minimize peripheral interactions between maskers, odd-numbered channels were presented to one ear and even to the other; this procedure effectively yields an unmodulated masker (i.e., a masker with a modulation spectrum containing only a DC component). Thus, the SiDCmod condition presented stimuli that were dichotic, unlike the other conditions, which presented diotic stimuli. The long-term spectra of the target speech (including the carrier phrase) and that of the masker were adjusted to match the average (across instances) long-term spectrum of the four-talker babble. (5) Vocoded speech in babble (Vocoded SiB): SiB at 0 dB SNR was subjected to 64-channel envelope vocoding. A randomly selected babble sample was used for each Vocoded SiB stimulus, similar to what was done for SiB. In accordance with prior work (Qin and Oxenham, 2003; Viswanathan et al., 2021b), our vocoding procedure retained the cochlear-level envelope in each of 64 contiguous frequency channels with center frequencies equally spaced on an equivalent rectangular bandwidth (ERB)-number scale (Glasberg and Moore, 1990) between 80 and 6000 Hz; however, the stimulus temporal fine structure (TFS) in each channel was replaced with a noise carrier. The envelope in each channel was extracted by half-wave rectification of the frequency content in that channel followed by low-pass filtering with a cutoff frequency of 300 Hz, or half of the channel bandwidth, whichever was lower. The envelope in each channel was then used to modulate a random Gaussian white noise carrier; the result was band-pass filtered within the channel bandwidth and scaled to match the level of the original signal. We verified that the vocoding procedure did not significantly change envelopes at the cochlear level, as described in Viswanathan et al. (2021b). Table 1 describes the rationale behind including these different stimulus conditions in our study.

The stimulus used for online volume adjustment was running speech mixed with four-talker babble. The speech and babble samples were obtained from the QuickSIN corpus (Killion et al., 2004); these were repeated over time to obtain a ∼20 s total stimulus duration to give subjects sufficient time to adjust their computer volume with the instructions described in Experimental design. The root mean square value of this stimulus corresponded to 75% of the dB difference between the softest and loudest stimuli in the consonant identification experiment, which ensured that no stimulus was too loud for subjects once they had adjusted their computer volume to a comfortable level.

Experimental design

The online consonant identification experiment was previously described in Viswanathan et al. (2021b). Subjects performed the experiment using their personal computers and headphones/earphones. Our online infrastructure included checks to prevent the use of mobile devices. The experiment consisted of the following parts: (1) headphone/earphone checks, (2) demonstration (Demo), and (3) Test. Each of these three parts had a volume-adjustment task at the beginning. In this task, subjects were asked to make sure that they were in a quiet room and wearing wired (not wireless) headphones or earphones. They were instructed not to use desktop/laptop speakers. Headphone/earphone use was checked using the procedures in Mok et al. (2021). They were then asked to set their computer volume to 10–20% of the full volume, after which they were played a speech-in-babble stimulus and asked to adjust their volume up to a comfortable but not too loud level. Once subjects had adjusted their computer volume, they were instructed not to adjust the volume during the experiment, as that could lead to sounds being too loud or soft.

The Demo stage consisted of a short training task designed to familiarize subjects with how each consonant sounds and with the consonant identification paradigm. Subjects were instructed that in each trial they would hear a voice say, “You will mark *something* please.” They were told that they would be given a set of options for *something* at the end of the trial, and that they should click on the corresponding option. After subjects had heard all consonants sequentially (i.e., the same order as the response choices) in quiet, they were tasked with identifying consonants presented in random order and spanning the same set of listening conditions as the Test stage. Subjects were instructed to ignore any background noise and only listen to the particular voice saying, “You will mark *something* please.” To ensure that all subjects understood and were able to perform the task, only those subjects who scored ≥85% in the Demo's SiQuiet control condition were selected for the Test stage.

Subjects were given instructions in the Test stage similar to those in the Demo but told to expect trials with background noise from the beginning. The Test stage presented, in random order, the 20 consonants (with one stimulus repetition per consonant) across all four talkers and all five experimental conditions. In both Demo and Test, the masking noise, when present, started 1 s before the target speech and continued for the entire duration of the trial. This was done to cue the subjects' attention to the stimulus before the target sentence was played. In both the Demo and Test parts, subjects received feedback after every trial on whether their response was correct to promote engagement with the task. However, subjects were not told what consonant was presented to avoid overtraining to the acoustics of how each consonant sounded across the different conditions; the only exception to this rule was in the first subpart of the Demo, where subjects heard all consonants in quiet in sequential order.

Separate studies were posted on Prolific.co for the different talkers. When a subject performed a particular study, they would be presented with the speech stimuli for one specific talker consistently over all trials. Thus, each subject was not just trained with one talker but also tested with that same talker to avoid training-testing disparities. To obtain results that are generalizable, we used 50 subjects per talker (subject overlap between talkers was not controlled); with four talkers, this yielded 200 subject-talker pairs or data samples. Within each talker and condition, all subjects performed the task with the same stimuli. Moreover, all condition effect contrasts were computed on a within-subject basis and averaged across subjects.

Data preprocessing

Only samples with intelligibility scores ≥85% for the SiQuiet control condition in the Test stage were included in results reported here. All conditions for the remaining samples were excluded from further analyses as a data quality control measure. This yielded a final N = 191 samples.

Quantifying confusion matrices from perceptual measurements

The 20 English consonants used in this study were assigned the phonetic features described in Table 2. The identification data collected in the Test stage were used to construct consonant confusion matrices (pooled over samples) for the different conditions; these matrices in turn were used to construct voicing, place of articulation (POA), and manner of articulation (MOA) confusion matrices by pooling over all consonants.

Given that all psychophysical data were collected online, we performed data quality checks; the analyses performed and the results are described in detail in Mok et al. (2021) and Viswanathan et al. (2021b), and are only briefly presented here. We compared consonant confusions for SiSSN, a commonly used condition in the literature, with previous lab-based findings. Phatak and Allen (2007) found that for a given overall intelligibility, recognition scores vary across consonants. They identified three groups of consonants, C1, C2, and C3 with low, high, and intermediate recognition scores, respectively, in speech-shaped noise. The SiSSN data that we collected online closely replicated that key trend for the groups they identified. Moreover, based on a graphical analysis of confusion patterns in speech-shaped noise, Phatak and Allen (2007) identified perceptual clusters (i.e., sets where one consonant is confused most with another in the same set). In the current study too, we identified perceptual clusters for SiSSN by subjecting the consonant confusion matrix to a hierarchical clustering analysis (Ward, 1963); our results closely replicated the lab-based clustering results of Phatak and Allen (2007). As previous lab-based results were not readily available for the remaining masking conditions in our study, we instead examined whether subjects randomly chose a different consonant from what was presented when they made an error, or if there was more structure in the data. The percent errors in our data fell outside the distributions expected from random confusions, suggesting that the error patterns have a nonrandom structure. Together, these results support the validity of our online-collected data.

We wished to test whether there are any significant differences in consonant confusion patterns across the different masking conditions, namely, SiSSN, SiB, SiDCmod, and Vocoded SiB. If so, these differences could then be predicted by computational modeling to test our hypothesis about the role of temporal-coherence-based across-channel masking of target speech by noise fluctuations. As the SiQuiet condition was intended to primarily be used as a control condition to ensure data quality (see Data preprocessing), SiQuiet data were not subjected to this analysis. To test whether confusion patterns differed across the masking conditions, we first normalized the overall intelligibility for these conditions to 60% by scaling the consonant confusion matrices such that the sum of the diagonal entries was the desired intelligibility (note that overall intelligibility was not normalized for the main modeling analyses of this study). By matching intelligibility in this manner, differences in confusion matrices across conditions could be attributed to changes in consonant categorization and category errors rather than differences in overall error counts because of one condition being inherently easier at a particular SNR. Because overall intelligibility was similar across the masking conditions to start with (see Fig. 5), small condition differences in intelligibility could be normalized without loss of statistical power. Confusion-matrix differences between the intelligibility-matched conditions were then compared with appropriate null distributions of zero differences (see Statistical analysis) to extract statistically significant differences (see Fig. 6).

Auditory periphery modeling

The auditory-nerve model of Bruce et al. (2018) was used to simulate processing by the auditory periphery. The parameters of this model were set as follows. Thirty cochlear filters with characteristic frequencies (CFs) equally spaced on an ERB-number scale (Glasberg and Moore, 1990) between 125 and 8000 Hz were used. Normal function was chosen for the outer and inner hair cells. The species was chosen to be human with the Shera et al. (2002) cochlear tuning at low sound levels; however, with suppression, the Glasberg and Moore (1990) tuning is effectively obtained for our broadband, moderate-level stimuli (Heinz et al., 2002; Oxenham and Shera, 2003). The noise type parameter for the inner-hair-cell synapse model was set to fixed fractional Gaussian noise to yield a constant spontaneous auditory-nerve firing rate. To avoid single-fiber saturation effects, the spontaneous rate of the auditory-nerve fiber was set to 10, corresponding to that of a medium-spontaneous-rate fiber. An approximate implementation of the power-law adaptation dynamics in the synapse was used. The absolute and relative refractory periods were set to 0.6 ms.

The periphery model was simulated with the same speech stimuli used in our psychophysical experiment (i.e., CV utterances that spanned 20 consonants, four talkers, and five conditions, and were embedded in a carrier phrase) as input. The level for the target speech was set to 60 dB SPL across all stimuli, as this produced sufficient (i.e., firing rate greater than spontaneous rate) model auditory-nerve responses for consonants in quiet and also did not saturate the response to the loudest stimulus. The periphery model was provided with just one audio channel input for all conditions except SiDCmod, as that was the only condition that was dichotic rather than diotic. Instead, for SiDCmod, the model was separately simulated for each of the two audio channels. Two hundred stimulus repetitions were used to derive peristimulus time histograms (PSTHs) from model auditory-nerve outputs. The model was simulated for the full duration of each stimulus, as opposed to just the time period when the target consonant was presented. A PSTH bin width of 1 ms (i.e., a sampling rate of 1 kHz) was used. This was done to capture fine-structure phase locking up to and including the typical frequency range of human pitch for voiced sounds. In the case of the SiDCmod condition, a separate PSTH was computed for each of the two dichotic audio channels.

Although the full speech stimuli (including the carrier phrase and CV utterances) were used as inputs to the periphery model, the responses to the target consonants were segmented out from the model PSTHs before being input into the scene analysis models. This segmentation had to be performed manually because the duration of the carrier phrase varied across consonants and talkers, and the start and end times corresponding to any given target consonant were unknown a priori. The time segment corresponding to when the target consonant was presented was calculated for each speech-in-quiet stimulus by visualizing speech spectrograms computed by gammatone filtering (Patterson et al., 1987) followed by Hilbert-envelope extraction (Hilbert, 1906). One hundred twenty-eight gammatone filters were used for this purpose, with center frequencies between 100 and 8000 Hz and equally spaced on an ERB-number scale (Glasberg and Moore, 1990). A fixed duration of 104.2 ms was used for each consonant segment. Segmentation accuracy was verified by listening to the segmented consonant utterances. The time segments thus derived were used to extract model auditory-nerve responses to the different target consonants across the different conditions and talkers. These responses were then used as inputs to the scene analysis models described below.

Scene analysis modeling to predict consonant confusions

To study the contribution of across-channel temporal-coherence processing to consonant categorization, we constructed two different scene analysis models. The first is a within-channel modulation-masking-based scene analysis model inspired by Relaño-Iborra et al. (2016), and the second is a simple across-channel temporal coherence model mirroring the physiological computations that are known to exist in the cochlear nucleus (Pressnitzer et al., 2001).

In the within-channel modulation-masking-based model, the auditory-nerve PSTHs (i.e., the outputs from the periphery model; see Auditory periphery modeling) corresponding to the different consonants, conditions, and talkers were filtered within a 1 ERB bandwidth (Glasberg and Moore, 1990) to extract band-specific envelopes. Note that the envelopes extracted from auditory-nerve outputs may contain some TFS converted to envelopes via inner-hair-cell rectification (assuming envelope and TFS are defined at the output of the cochlea), but that is the processing that is naturally performed by the auditory system as well. Pairwise dynamic time warping (Rabiner, 1993) was performed to align the results for each pair of consonants across time. Dynamic time warping can help compensate for variations in speaking rate across consonants. A modulation filterbank (Ewert and Dau, 2000; Jørgensen et al., 2013) was then used to decompose the results at each CF into different modulation frequency (MF) bands. This filterbank consists of a low-pass filter with a cutoff frequency of 1 Hz in parallel with eight bandpass filters with octave spacing, a quality factor of 1, and center frequencies ranging from 2 to 256 Hz. For each condition, talker, CF, MF, and consonant, Pearson correlation coefficients were computed between the filterbank output for that consonant in that particular condition and the output for each of all 20 consonants in quiet. Each of the individual correlations was squared to obtain the variance explained; the results were averaged across talkers, CFs, and MFs to obtain a raw neural metric ψ for each experimental condition. A separate ψ value was obtained for each condition, and every pair of consonant presented and option for consonant reported. For the dichotic SiDCmod condition, the variance explained was separately computed for the left and right ears at each CF, then the maximum across the two ears (i.e., the better-ear contribution) was used for that CF (Zurek, 1993). Finally, for each condition, the ψ values were normalized such that their sum across all options for consonants reported for a particular consonant presented was equal to one; this procedure yielded a condition-specific neural consonant confusion matrix.

We wanted to test whether across-channel temporal-coherence processing of input fluctuations could better predict consonant categorization than a purely within-channel modulation masking model. To simulate across-channel temporal-coherence processing, we modeled a physiologically plausible wideband-inhibition-based temporal-coherence processing circuit proposed by Pressnitzer et al. (2001) to account for physiological correlates of CMR in the cochlear nucleus. A schematic of this circuit is provided in Figure 1. Note that the circuit model parameter corresponding to the excitation-to-inhibition ratio cannot be readily compared to its physiological correlate because the model is rate based and lacks important membrane conductance properties that spiking models can be endowed with. The overall across-channel scene analysis model is similar to the within-channel model, except that the envelope extraction stage of the within-channel model is replaced with the CMR circuit model in the across-channel model. Thus, the across-channel model can account for both within-channel modulation masking effects as well as across-channel temporal-coherence processing. Figure 2 shows schematics of both the within- and across-channel models.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

CMR circuit based on wideband inhibition in the cochlear nucleus. This physiologically plausible circuit was proposed by Pressnitzer et al. (2001) to model CMR effects seen in the cochlear nucleus (CN). CN units at different CFs form the building blocks of this circuit. Each CN unit consists of a narrowband cell (NB) that receives narrow on-CF excitatory input from the auditory nerve (AN) and inhibitory input from a wideband inhibitor (WBI). The WBI in turn receives excitatory inputs from AN fibers tuned to CFs spanning 2 octaves below to 1 octave above the CF of the NB that it inhibits. The time constants for the excitatory and inhibitory synapses are 5 ms and 1 ms, respectively. The WBI input to the NB is delayed with respect to the AN input by 2 ms. Note that our model simulations were rate based; that is, they used AN PSTHs rather than spikes. Thus, all outputs were half-wave rectified (i.e., firing rates were positive at every stage). All synaptic filters were initially normalized to have unit gain, then the gain of the inhibitory input was allowed to vary parametrically to implement different excitation-to-inhibition (EI) ratios between 3:1 and 1:1. The EI ratio was adjusted to obtain the best consonant confusion prediction accuracy for SiSSN (i.e., the calibration condition), and the optimal ratio for the calibration condition was found to be 1.75:1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Schematic of the within- and across-channel scene analysis models. The speech stimuli were input into the Bruce et al. (2018) model, which simulated a normal auditory periphery with 30 cochlear filters having CFs equally spaced on an ERB-number scale (Glasberg and Moore, 1990) between 125 Hz and 8 kHz. PSTHs from the periphery model were processed to retain only the time segments when the target consonants were presented. For the within-channel model, these results were filtered within a 1 ERB bandwidth (Glasberg and Moore, 1990) to extract band-specific envelopes; however, for the across-channel model, the results were instead input into the CMR circuit model (Fig. 1). Pairwise dynamic time warping was performed to align the outputs from the previous step across time for each pair of consonants. A modulation filterbank (Ewert and Dau, 2000; Jørgensen et al., 2013) was then used to decompose the results at each CF into different MF bands. This filterbank consists of a low-pass filter with a 1 Hz cutoff in parallel with eight bandpass filters with octave spacing, a quality factor of 1, and center frequencies between 2 and 256 Hz. For each condition, talker, CF, MF, and consonant, Pearson correlation coefficients were computed between the filterbank output for that consonant in that particular condition and the output for each of all 20 consonants in quiet. Each of the individual correlations was squared to obtain the variance explained; the results were averaged across talkers, CFs, and MFs to obtain a raw neural metric ψ for each experimental condition. A separate ψ value was obtained for each condition, and every pair of consonant presented and option for consonant reported. The ψ values were normalized such that their sum across all options for consonants reported for a particular consonant presented was equal to one, which yielded a condition-specific neural consonant confusion matrix.

To verify that the CMR circuit model (Fig. 1) produced physiological correlates of CMR similar to those reported by Pressnitzer et al. (2001), we used the same complex stimuli that they used (Fig. 3). The stimuli consisted of a target signal in a 100% sinusoidally amplitude-modulated (SAM) tonal complex masker. There were three experimental conditions: Reference, Comodulated, and Codeviant. In the Reference condition, the masker had just one component, a SAM tone with a carrier frequency of 1.1 kHz (to allow comparison to data from Pressnitzer et al., 2001); this masking component is also referred to as the on-frequency component (OFC). The Comodulated and Codeviant conditions presented the OFC along with six flanking components that were SAM tones at the same level as the OFC. The carrier frequency separation between the different flanking components and the OFC were −800, −600, −400, 400, 600, and 800 Hz, respectively. The flanking components were modulated in phase with the OFC in the Comodulated condition, and 180° out of phase with the OFC in the Codeviant condition. A 10 Hz modulation rate was used for all SAM tones. The target signal consisted of a 50-ms-long (i.e., half of the modulation time period) tone pip at 1.1 kHz that was presented in the dips of the OFC modulation during the last 0.3 s of the stimulus period (i.e., in the last three dips) at different values of signal-to-component ratio (SCR; defined as the signal maximum amplitude over the amplitude of the OFC before modulation). These stimuli were presented to the periphery model, and the corresponding model outputs were passed into the CMR circuit model.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Stimuli used to validate the CMR circuit model. The stimuli used were from Pressnitzer et al. (2001), and consisted of a target signal in a 10 Hz 100% SAM tonal complex masker. The masker differed depending on the experimental condition. In the Reference condition, the masker was a 1.1-kHz-carrier SAM tone (referred to as the OFC). In the Comodulated and Codeviant conditions, six flanking components were presented in addition to the OFC. The flanking components were SAM tones at the same level as the OFC. The flanking components were separated from the OFC by −800, −600, −400, 400, 600, and 800 Hz, respectively. The modulation of each flanking component was in phase with the OFC modulation in the Comodulated condition, but 180° out of phase with the OFC modulation in the Codeviant condition. The target signal was a 50-ms-long 1.1 kHz tone pip that was presented in the dips of the OFC modulation during the last 0.3 s of the stimulus period (i.e., in the last 3 dips) at different values of SCR (defined as the signal maximum amplitude over the amplitude of the OFC before modulation).

The rate-level function at the output of the CMR circuit model (Fig. 4D) closely matches physiological data for chopper units in the ventral cochlear nucleus (Winter and Palmer, 1990) and was used to set the masker level for the CMR stimuli. The firing-rate threshold was 0 dB SPL for pure-tone inputs at CF; thus, a fixed level of 40 dB SPL (i.e., 40 dB SL) was used for the OFC. The PSTH outputs from the CMR circuit model (at 1.1 kHz CF) are shown in Figure 4A. The time-averaged statistics of the firing rate during the last 0.3 s of the stimulus period and in the absence of the target signal were used as the null distribution against which the neurometric sensitivity, d^′, was calculated; a separate null distribution was derived for each condition. The average firing rate during the target signal periods was compared with the corresponding null distribution to estimate a separate d^′ for each SCR and condition (Fig. 4B). The d^′ of 0.4 was used to calculate SCR thresholds and the corresponding CMR (threshold difference between the Codeviant and Comodulated conditions). Note that the absolute d^′ values cannot be interpreted in a conventional manner given that the choice of window used to estimate the null-distribution parameters introduces an arbitrary scaling; thus, our choice of the d^′ criterion to calculate CMR was instead based on avoiding floor and ceiling effects. Results indicate that the CMR circuit model shows a CMR effect consistent with actual cochlear nucleus data in that signal detectability is best in the Comodulated condition, followed by the Reference and Codeviant conditions (compare Figs. 4A and B with Figs. 2 and 6A from Pressnitzer et al., 2001). The size of the predicted CMR effect is also consistent with perceptual measurements (Mok et al., 2021). As expected, no CMR effect is seen at the level of the auditory nerve. Thus, the CMR circuit model accounts for the improved signal representation in the Comodulated condition where the masker is more easily segregable from the target signal, an advantage that derives from the fact that the different masking components are temporally coherent with one another. In addition, it also accounts for the greater cross-channel interference in the Codeviant condition, where the flanking components are temporally coherent with the target signal that is presented in the dips of the OFC. Finally, when the modulation rate of the input SAM tones was varied, CMR effects were still seen (Fig. 4C) and followed the same low-pass trend as human perceptual data (Carlyon et al., 1989).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

CMR circuit model validation. A, PSTH outputs from the CMR circuit model at 1.1 kHz CF for the stimuli in Figure 3. Results are shown separately for the Comodulated and Codeviant conditions and at different SCRs. The black horizontal bars indicate the time points corresponding to when the target signal was presented. B, Summary of the results from A showing the neurometric sensitivity, d^′, as a function of SCR for the auditory-nerve and CMR circuit model outputs (both at 1.1 kHz CF). The CMR circuit model shows a clear separation between the Comodulated and Codeviant conditions, that is, a CMR effect. This is not seen at the level of the auditory nerve. C, The variation in the CMR obtained from the circuit model as a function of modulation rate. D, The pure-tone rate-level function (i.e., mean steady-state firing rate versus input tone level) for the CMR circuit model.

Each scene analysis model was separately calibrated by fitting a logistic/sigmoid function mapping the neural consonant confusion matrix entries from that model for the SiSSN condition to corresponding perceptual measurements. The mapping derived from this calibration was used to predict perceptual consonant confusion matrices from the corresponding neural confusion matrices for unseen conditions. Voicing, POA, and MOA confusion matrices were then derived by pooling over all consonants. Finally, the Pearson correlation coefficient was used to compare model predictions to perceptual measurements across the voicing, POA, and MOA categories. The prediction accuracy for the different models is reported in Results.

Statistical analysis

Permutation testing (Nichols and Holmes, 2002) with multiple-comparisons correction at 5% false discovery rate (FDR; Benjamini and Hochberg, 1995) was used to extract significant differences in the SiSSN, SiB, SiDCmod, and Vocoded SiB consonant confusion matrices quantified earlier (see Quantifying confusion matrices from perceptual measurements). The null distributions for permutation testing were obtained using a nonparametric shuffling procedure, which ensured that the data used in the computation of the null distributions had the same statistical properties as the measured confusion data. A separate null distribution was generated for each consonant. Each realization from each null distribution was obtained by following the same computations used to obtain the actual differences in the confusion matrices across conditions but with random shuffling of condition labels corresponding to the measurements. This procedure was independently repeated with 10,000 distinct randomizations for each null distribution.

The p-values for the Pearson correlation coefficients between model predictions and perceptual measurements (Tables 3, 4) were derived using Fisher's approximation (Fisher, 1921).

To test whether the improvements in prediction accuracy (i.e., the correlation between model predictions and perceptual measurements) offered by the across-channel model compared to the within-channel model are statistically significant, a permutation procedure was used once again. Under the null hypothesis that the within- and across-channel models are equivalent in their predictive power, the individual entries of the confusion matrices predicted by the two models can be swapped without effect on the results. Thus, to generate each realization of the null distribution of the correlation improvement, a randomly chosen half of the confusion matrix entries were swapped; this permutation procedure was independently repeated 100,000 times. A separate null distribution was generated in this manner for each condition. The actual improvements in correlation were compared with the corresponding null distributions to estimate (uncorrected) p-values. To adjust for multiple testing, an FDR procedure (Benjamini and Hochberg, 1995) was used. Table 5 indicates whether each test met criteria for statistical significance under an FDR threshold of 5%.

Code Accessibility

Subjects were directed from Prolific.co to the SNAPlabonline psychoacoustics infrastructure (Bharadwaj, 2021; Mok et al., 2021) to perform the study. Offline data analyses were performed using custom software in Python (https://www.python.org) and MATLAB (MathWorks). The code for our computational models is publicly available on GitHub at https://github.com/vibhaviswana/modeling-consonant-confusions.