Distributed Neural Representations of Phonological Features during Speech Perception

Stimuli.

Stimuli consisted of the following 16 phonemes followed by the vowel /a/ as in father: /p/, /t/, /k/, /f/, /θ/ (“th” as in thumb), /s/, / ∫ / (“sh” as in shoe), /b/, /d/, /g/, /v/, /ð/ (“th” as in that), /z/, / ʒ / (“s” as in measure), /m/, and /n/. The recordings were tokens from the standardized University of California, Los Angeles, version of the Nonsense Syllable Test (Dubno and Schaefer, 1992). Each consonant–vowel syllable (CV) was produced by four different speakers (two male, two female) and each speaker produced three different recordings of each sound. Thus, 12 different versions of each CV were used, ensuring that there was some acoustic variability across tokens of the same type. The sounds were played at a comfortable volume for participants through electrodynamic MR-Confon headphones. As depicted in Table 1, the stimuli spanned the major distinctive features.

MRI set-up and data acquisition.

Participants were scanned with a 3.0 T Siemens Magnetom Trio MRI scanner using a 12-channel head coil system. High-resolution gradient-echo multislice T1-weighted scans (160 slices of 1 mm thickness, 19.2 × 25.6 cm field of view) coplanar with the echo-planar imaging scans (EPIs) as well as whole-brain magnetization prepared rapid gradient echo (MP-RAGE) 3-D T1-weighted scans were first acquired for anatomical localization, followed by T2*-weighted EPIs sensitive to BOLD contrast. Images were acquired using a two-shot gradient-echo EPI sequence (22.5 × 22.5 cm field of view with a 96 × 96 matrix size, resulting in an in-plane resolution of 2.35 × 2.35 mm for each of 26 3.5 mm axial slices with a 0.5 mm interslice gap; repetition time, 1.5 s; echo time, 27 ms; flip angle, 62°).

MRI preprocessing and whole-brain univariate regression analyses.

Functional images were converted into NIfTI-1 (Neuroimaging Informatics Technology Initiative-1) format, motion-corrected, and realigned to the first image of the first run with AFNI's (analysis of functional neuroimages; Cox, 1996) 3dvolreg program. All image volumes were then smoothed with a 4 mm FWHM Gaussian kernel. Single-subject multiple-regression modeling was performed using the AFNI program 3dDeconvolve. Each of the 16 phonemes was modeled by convolving a hemodynamic response function (statistical parametric mapping canonical function as implemented in AFNI) with a delta function formed from the vector of speech stimulus onsets. An additional set of five nuisance regressors (a constant term plus linear, quadratic, and higher-order polynomial terms) was included for each scanning run to model low-frequency noise in the time series data. Separate regression analyses were performed for each of the 10 scanning runs, yielding a set of 16 (one per CV) β regression weights for each run. The regression models were computed separately for each run so that we would have independent samples for cross-validation in the multivariate analyses.

All statistical analyses, both voxelwise and region-of-interest (ROI) based, were first conducted on the spatially smoothed and realigned functional images in the participant's native EPI space. The MP-RAGE anatomical scan was normalized to the Montréal Neurological Institute (MNI) template using nonlinear symmetric normalization implemented in Advanced Normalization Tools (ANTs; Avants et al., 2008). An equivalent transformation was then applied to maps of univariate statistical results (see Voxelwise correlation of MDS dimensions with fMRI activation) derived from functional images using ANTs to normalize these maps to MNI space for multisubject analyses. Statistical significance at the group level was determined using Monte Carlo simulations of expected cluster sizes under the null hypothesis using the AFNI program AlphaSim. For a voxelwise threshold of p < 0.005, only clusters with >13 voxels were determined to be significant at the cluster-corrected (p < 0.05) level.

Behavioral data.

Trials with reaction times of <400 ms were removed from behavioral analyses. Trials for which the participant did not respond within 1500 ms of stimulus onset were scored as incorrect. A 16 (syllable) × 4 (speaker) repeated-measures ANOVA was performed on accuracy scores. We conducted further ANOVAs to test for an effect of manner (stops, fricatives, and nasals) and place of articulation (labial, dental, alvolar, palatoalveolar, and velar), as well as a paired-sample t test to assess voicing effects (voiced vs voiceless).

Multidimensional scaling analysis.

Miller and Nicely (1955) recorded phonemic identification data (how often each CV stimulus was incorrectly identified as another CV) from participants using the 16 phonemes used in the current study to establish confusability matrices at different levels of noise. They found that while the number of confusions is affected by noise, the overall pattern is not. Thus, the fact that, for example, /m/ is more confusable with /n/ than it is with /k/ is both reliable and informative—it tells us something about the way in which our brains process speech sounds. Following Shepard (1972), nonmetric multidimensional scaling [MDS; using isoMDS from the MASS (Modern Applied Statistics with S) package in the R programming language; Venables and Ripley, 2002] was applied to the phonemic confusion data taken directly from Tables 1–6 in Miller and Nicely's (1955) report (noise levels: −18, −12, −6, 0, +6, +12 dB). The acoustic confusion data were first converted to distances using a logarithmic transformation of the normalized confusion probabilities (Shepard, 1972). Nonmetric MDS was then applied to the resultant distance matrix and a two-dimensional solution was computed, resulting in a continuous and empirically derived map reflecting the relative positions of each phoneme in coordinate space. The result of this analysis closely resembles Shepard's original solution (Fig. 3; cf. Shepard, 1972, Fig. 4).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Multidimensional scaling of 16 consonants. Two-dimensional spatial representation of 16 phonemes based on pooled data from Miller and Nicely (1955). Voicing is represented on the x-axis, with unvoiced phonemes occupying the left side and voiced phonemes occupying the right side. Manner of articulation is approximately represented on the y-axis, with nasals occupying the top half of the figure and fricatives and stops occupying the bottom. A clear division is observed between the unvoiced stops and fricatives, while overlap exists between voiced stops and fricatives. /θ/, “th” in thumb; /∫/, “sh” in shoe; /ð/, “th” in that; /ʒ/, “s” in measure.

The first dimension, plotted on the x-axis, approximately corresponds to the feature known as voicing, with unvoiced stimuli on the left side and voiced stimuli on the right. The second dimension, plotted on the y-axis, is related to manner of articulation, with nasals occupying the upper half of the graph and fricatives and stops occupying the lower half.

Voxelwise correlation of MDS dimensions with fMRI activation.

To examine the relation between the two MDS-derived dimensions based on perceptual distance (Fig. 3, x-axis and y-axis) and brain activation, we computed separate Spearman rank correlations between each of the first two dimensions and the corresponding β estimates, averaged over runs, for the 16 CVs. This produced rank correlations for each MDS dimension for all voxels in the brain and all participants in the group. To test for reliable effects at the group level, these correlation images were z-transformed, spatially normalized to MNI space, and submitted to a voxelwise one-sample t test.

Multivariate analyses.

Although perceptual confusability offers a means of investigating the representational basis of acoustic–phonetic speech perception, MDS coordinate maps derived from such data are dominated by only two of the three major distinctive features, namely, voicing and manner of articulation. However, because place of articulation (or something akin to it) must be represented at some level in the brain—otherwise sounds that share voicing and manner but differ in place (such as “tan” and “can”) would be indistinguishable from one another—perceptual confusability does not fully reflect the information contained in the underlying neural code. Moreover, from a theoretical and empirical standpoint, place of articulation has played a prominent role in studies of acoustic–phonetic speech perception (see Discussion). Furthermore, place of articulation is associated with rapid fluctuations in spectrotemporal fine structure and, unlike manner and voicing features, is not easily recognized on the basis of lower-frequency envelope information (Shannon et al., 1995). Thus, according to AST and other theories of speech perception that posit left hemisphere specialization for rapid temporal processing, place of articulation should be more robustly represented in the left auditory cortex than in the right. We therefore investigated the distribution of patterns of activity associated with the major distinctive features of voicing, manner, and place of articulation directly, rather than through the lens of perceptual confusability.

To do so, we used a class of techniques known as multivoxel pattern analysis (MVPA). The basic premise of MVPA is to identify reliable patterns of activation rather than relying on subtraction logic that is typical of univariate approaches. The main benefit of MVPA is that it takes advantage of systematic variance distributed across voxels rather than being based only on the average activity within a single voxel. Thus, we can train pattern classifiers to discriminate spatial patterns associated with one feature (i.e., voiced) from those associated with another feature (i.e., unvoiced) and then test these classifiers on independent test trials.

To identify reliable patterns of distributed brain activity associated with each feature, a series of MVPA analyses was carried out. We used Freesurfer's (Dale et al., 1999) automatic anatomical labeling (“aparc 2009”; Destrieux et al., 2010) algorithm to define a set of 148 cortical and subcortical ROIs. These ROIs were defined using each participant's high-resolution anatomical scan and therefore group analyses could be performed without applying any spatial normalization. We used anatomically defined ROIs rather than a moving “searchlight” (Kriegeskorte et al., 2006) procedure because we wished to preserve borders between spatially adjacent regions along the sylvian fissure (e.g., ventral frontal cortex and superior temporal cortex). This was especially important in light of one the major aims of this study, namely, to quantify the contribution of temporal and prefrontal regions—portions of which are adjacent in volumetric space—to the neural representations underlying phoneme perception.

For all MVPA analyses we used shrinkage discriminant analysis (SDA) as implemented in the R package “sda” (http://cran.rproject.org/web/packages/sda/). SDA is a form of linear discriminant analysis that estimates shrinkage parameters for the variance–covariance matrix of the data, making it suitable for high-dimensional classification problems. It has the advantage of these shrinkage parameters being estimated analytically from the data, obviating a doubly nested cross-validation scheme (Ahdesmäki and Strimmer, 2010). To evaluate classifier performance, we used 10-fold cross-validation where each fold of data consisted of the β regression weights of nine of the 10 runs, with one run held out for testing. Thus, during cross-validation, observations drawn from the same scanning run were never part of both the training and test datasets. MVPA analyses were performed within each anatomical ROI, or in groups of ROIs (see below), yielding regional estimates of classifier performance. The SDA classifier produces both a categorical prediction (i.e., the label of the test case) as well as a continuous probabilistic output (the posterior probability that the test case is of label x). The continuous outputs were used to compute area-under-the-curve (AUC) metrics and the categorical predictions were used to compute classifier accuracy.

To test for statistical significance at the group level, classifier performance for each ROI was evaluated with a one-sample t test where the null hypothesis assumed a theoretical chance AUC of 0.5. To validate this assumption, we performed permutation analyses in which we randomly shuffled the labels in the training set for each cross-validation fold and repeated the process 100 times for each subject, ROI, and phonological feature category. The grand mean, over subjects and ROIs, of these permutation-based AUC values was 0.50000219, which by the exact binomial test was not significantly different from the theoretical chance level of 0.5. We therefore concluded that for null hypothesis testing we could assume that an AUC of 0.5 is the expected value when the classifier is performing at chance.

Because classification analyses were conducted in anatomically defined ROIs specific to each subject, no spatial normalization was applied to the subject-specific classifier performance scores. To display statistics at the group level, the statistic of interest was projected on the parcellated (aparc 2009; Destrieux et al., 2010) cortical flat map associated with the Freesurfer average template (“fsaverage”).

Exploratory MVPA of individual feature subcategories.

To investigate the relative contributions of each brain area to the neural pattern of activity associated with distinctive features, separate classifiers were trained to detect the subcategories of voicing (voiced and voiceless), manner of articulation (nasals, stops, and fricatives), and place of articulation (labial, dental, alveolar, palatoalveolar, and velar). Because the full phonological feature matrix is not orthogonal for the set of 16 CVs—and thus feature dimensions are confounded across phonemes—we used a subsampling approach to ensure that classification for one feature category (e.g., nasal) could not be driven by a correlated feature category (e.g., voiced). To achieve this, we trained classifiers to discriminate each feature category from a subset of phonemes matched across the other feature dimensions. For example, a classifier was trained to discriminate labials from the subset of nonlabials matched on voicing and manner of articulation. Thus, nine classifiers (voiced, stop, fricative, nasal, labial, dental, alveolar, palatoalveolar, velar) were trained for each ROI. Note that because voicing consists of only two categories—voiced and voiceless—training two classifiers would be redundant as one is essentially the inverse of the other. We computed both classification and AUC as performance metrics. However, we preferred the latter metric for group statistics due to its increased sensitivity (Bradley, 1997). To test for significant group classifier performance using AUC, we used a one-sample t test where the null hypothesis (chance performance) assumed an AUC of 0.5, which is the expected value for the AUC when there is no relationship between the continuous classifier output and the category labels of the test cases. We then counted the number of significant feature subcategories (p < 0.05, uncorrected) for each ROI. Regions with ≥3 significant features (p < 0.0083, by the binomial probability distribution) were deemed significant in this exploratory analysis.

MVPA of distinctive features in speech-sensitive cortex.

The overall performance for a feature (voicing, manner, and place of articulation) was then computed as a weighted average of the individual category classifiers, where the weights were determined by the frequency of each category. For example, the performance score for manner of articulation was calculated by averaging classifier performance for stops (N = 6), fricatives (N = 8), and nasals (N = 2) with the following weights: 0.375, 0.5, and 0.125. Statistical significance at the group level was evaluated with a one-sample t test corrected for multiple comparisons using a false discovery rate of q = 0.05.

In this analysis we limited the ROI search space to regions known to be sensitive to tasks involving the production and perception of speech. We used Neurosynth (http://neurosynth.org/; Yarkoni et al., 2011) to create a meta-analytic mask using the search term “speech” (http://neurosynth.org/features/speech/). This resulted in a coordinate-based activation mask constructed from 424 studies and encompassing the language-related areas in the temporal and frontal lobes. We intersected this meta-analytic mask with the Freesurfer aparc 2009 ROI mask as defined in MNI space. If any of the intersected ROIs had ≥10 voxels, we included that ROI in our search space. To ensure hemispheric symmetry, if a left hemisphere ROI was included so too was its right hemisphere homolog. This resulted in an ROI mask consisting of 38 ROIs (19 left hemisphere and 19 right hemisphere), which are shown in Figure 4.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

ROIs in speech-relevant areas of the temporal and frontal lobes. An ROI mask was created by intersecting a Neurosynth automated meta-analysis (search term: “speech”; Yarkoni et al., 2011) and the 148 Freesurfer ROIs (aparc 2009 atlas; Destrieux et al., 2010). The final set of 38 ROIs (19 left, 19 right) spanned cortical areas in the temporal and frontal lobes important for the perception and production of speech.

Hemispheric differences in the superior temporal lobe.

To test for lateralization effects in MVPA measures within broadly defined auditory processing structures of the superior temporal lobe, paired-samples t tests were performed on a subset of left versus right auditory cortex ROIs for classification performance of voicing, manner, and place. Auditory cortex was defined as any region in the temporal lobe extending inferiorly to the middle temporal gyrus, medially to the posterior insula, anteriorly to the planum polare, and posteriorly to the lateral fissure. According to this definition, the following ROIs were included: transverse temporal gyrus [Heschl's gyrus (HG)], transverse temporal sulcus, STG, middle temporal gyrus, planum polare, planum temporale (PT), lateral fissure, insula, and superior temporal sulcus (Fig. 4, 1–9). Based on the AST hypothesis (Poeppel, 2003), our prediction was that features that are processed on faster time scales would be left lateralized while those that are processed on slower time scales would be right lateralized. As reviewed in Rosen (1992), the three main temporal cues for speech are the envelope, periodicity, and fine structure, each of which is based on varying fluctuation rates and have a differential impact on the perception of phonological features (Fig. 1). Thus, because voicing and manner of articulation are based on all three temporal cues, it was predicted that these features would be represented bilaterally in the superior temporal cortices, while place would be left lateralized due to its requirement for rapid temporal processing.

Deriving neural confusability matrices through classification of individual phonemes.

In a final analysis, we examined whether individual phonemes could be reliably classified from patterns of activity across the left and right superior temporal lobe and whether the pattern of confusions made by the classifier is correlated with the original perceptual confusability matrices reported in Tables 1–6 of Miller and Nicely's (1955) classic study. Using the combined ROI sets defining the superior temporal lobe described in the previous section, we trained three classifiers (one for each hemisphere and one for both hemispheres combined) to distinguish among each of the 16 CVs, and recorded classification accuracy (chance performance, 0.0625; or 1 of 16) and the full matrix of phoneme confusions produced by the classifier predictions. The raw confusion counts were then normalized by the row totals (i.e., the number of times the classifier predicted each phoneme), yielding conditional probabilities for each ordered phoneme pair. Confusion probabilities between the same pairs (e.g., m → n and n → m) were then averaged to generate a symmetric probability matrix. This matrix was converted to a distance matrix using Shepard's log transformation (see Multidimensional scaling analysis) and then rank correlated with the distance matrix shown in Figure 2. To evaluate statistical significance for phoneme classification across the group, we computed three (left, right, both hemispheres) one-sample t tests against a null hypothesis of chance accuracy (0.0625).