Syntactic and auditory spatial processing in the human temporal cortex: An MEG study
Research highlights
► Early syntactic processing activates the anterior superior temporal gyrus (aSTG) ► Auditory spatial processing activates the posterior superior temporal gyrus (pSTG) ► Simultaneous syntactic and perceptual processing by activating the aSTG and pSTG ► Syntactic and auditory spatial effects within the first 200 ms after the word onset
Introduction
Previous studies investigating early syntactic processes in auditory sentence comprehension reported an enhanced negativity around 100–200 ms, labeled early left anterior negativity (ELAN), that was elicited by sentences containing a syntactic word category violation (Friederici et al., 1993, Hahne and Friederici, 1999, Hahne and Friederici, 2002). It has been a point of discussion whether prosodic effects contribute to such early syntax-related responses. However, a recent finding provides contradicting evidence by showing that a change in the prosodic contour cannot account for the ELAN effect, thereby highlighting the interpretation of the ELAN as a marker for initial syntactic processes (B. Herrmann et al., 2011).
Another question that has been raised in the context of the interpretation of the ELAN is whether this component belongs to the family of the mismatch negativity (MMN), an early negativity associated with rule violations in auditory perception (Näätänen et al., 1978). The MMN is elicited by an infrequently presented auditory event among a series of frequently repeated auditory events reflecting a memory comparison process (Schröger, 2005). The MMN has its maximum at around 100–200 ms following the onset of the infrequent event and has been reported for frequency (Näätänen et al., 1978, Shalgi and Deouell, 2007), duration (Jemel et al., 2002) or spatial deviations (Schröger, 1996, Nager et al., 2003), for example.
The relation between processing syntactic and auditory perceptual information was first investigated by Hahne et al. (2002) who studied early syntactic and auditory spatial processing in combination using electroencephalography (EEG). In this study, infrequent spatial deviations within a spoken sentence elicited a MMN, whereas sentences containing a syntactic word category violation elicited an ELAN. Sentences including an infrequent spatial deviation as well as a syntactic violation led to a larger negativity around 125–175 ms than the single violations, although the amplitude was less than a complete addition of the two single violations. The results were taken as an indicator that early syntactic and physical acoustic information can be processed in parallel within the first 200 ms (Hahne et al., 2002).
However, it remains an open question what parallel processing in this context means as only slight differences in the EEG scalp distribution of the syntactic and auditory spatial violation effects were reported in the study of Hahne et al. (2002). The present magnetoencephalography (MEG) study aims to shed more light on this issue. It might be the case that both types of single deviations recruit the same brain regions and that the double deviation thus leads to stronger activation in these very same regions. On the other hand, different brain regions might be involved in processing syntactic versus spatial deviations and that these regions are activated simultaneously when a combined violation is encountered. According to dual stream auditory processing models (Rauschecker and Scott, 2009) spatial information is processed in the dorsal stream involving the posterior portion of the superior temporal cortex, while speech (intelligibility of speech) processes recruit regions anterior to Heschl's gyrus in the ventral stream. It could, therefore, be hypothesized that partly different regions are activated when speech-related and auditory spatial features are processed in parallel.
Previous studies on the localizations of the neural mechanisms that underlie early syntactic processes, have localized the sources of the magnetic ELAN (ELANm) to the superior temporal cortex (Groß et al., 1998, Knösche et al., 1999), and more specifically to the anterior parts of the superior temporal gyrus (aSTG) and the inferior frontal cortex (IFC; Friederici et al., 2000). More recently, a so called “sensory hypothesis” for early syntactic effects has been introduced in the visual domain (Dikker et al., 2009). This hypothesis is based on the observation that early sensory cortex activations were affected at around 100–200 ms when participants encountered a syntactic word category violation (Dikker et al., 2009, Dikker et al., 2010; B. Herrmann et al., 2009). It has been suggested that these early sensory effects rely on form properties associated with the syntactic category, e.g. overt category marking by an affix (Dikker et al., 2010). For the processing of syntactic violations in an auditory oddball paradigm, B. Herrmann et al. (2009) observed modulations of the primary auditory cortex (AC) and the superior temporal sulcus, thus, suggesting activations in regions different from previous localizations showing aSTG and IFC activations in processing word category violations (Friederici et al., 2000). One explanation for these diverging findings that has been proposed relates to the methodological approaches applied (Dikker et al., 2009). In the study conducted by Friederici et al. (2000), for instance, dipole modeling was constrained by functional magnetic resonance imaging (fMRI) results from a previous sentence processing study (Meyer et al., 2000). Dikker et al. (2009) argued that this constraint might not be valid, as fMRI lacks the temporal resolution to derive solid assumptions about an early stage in sentence processing. In the study by B. Herrmann et al. (2009), on the other hand, two-word utterances were presented in an auditory oddball paradigm in which syntactic processing is accompanied by an acoustic change (Shtyrov and Pulvermüller, 2007), thus, possibly biasing the source localization towards primary regions.
The underlying neural sources of the MMN have mainly been localized to auditory sensory cortex regions (Giard et al., 1990, Alho et al., 1998, Maess et al., 2007). Particularly relevant for the current study, the MMN and its magnetic counterpart (MMF, mismatch field) has been found sensitive to infrequent changes of interaural time and interaural level differences (ITD, ILD; Schröger, 1996, Schröger and Wolff, 1996, Kaiser et al., 2000, Nager et al., 2003). The ITD and ILD are two important auditory cues which allow spatial sound localization (Middlebrooks and Green, 1991). The MMN to changes in ITD/ILD has been shown to modulate brain activations in the posterior STG/AC (Kaiser et al., 2000, Tata and Ward, 2005, Sonnadara et al., 2006, Deouell et al., 2006). Sometimes, an additional neural generator localized to the right IFC has been reported to underlie the MMN mechanism (Giard et al., 1990, Jemel et al., 2002, Shalgi and Deouell, 2007).
In addition to the ELAN effect in the 100–200 ms time window, previous studies were able to disentangle the “early syntax effect” into different sub-stages, observing an additional very early syntax-related effect that modulated the M50 component (C. S. Herrmann et al., 2000; B. Herrmann et al., 2009, Herrmann et al., 2011). Furthermore, the detection of simple rule violations in an auditory oddball paradigm has not only been shown to elicit the MMN, but also to modulate the brain's activity very early, starting at around 30 ms (Boutros and Belger, 1999, Ermutlu et al., 2005, Slabu et al., 2010, Grimm et al., 2011).
In the present study, anatomically constrained magnetoencephalography (MEG) was used to further investigate the early syntactic and auditory perceptual parallel processing effect in the brain. On this account, auditory sentence materials were presented that varied in syntax (syntactically correct vs. syntactically incorrect) and auditory space (standard vs. infrequent ITD change). For the source analysis, a distributed source model was used without any priors regarding the location of the underlying cortical regions modulated by syntax and auditory space. Our main goals were to examine the neural mechanisms that underlie the processing of (1) syntactic violations, (2) auditory spatial violations (ITD change), and (3) double violations, i.e. in syntax and auditory space.
- (1)
The localization of the neural responses elicited by syntactic violations allowed us to test previous localizations of the ELANm (Friederici et al., 2000), and to examine whether auditory sensory cortices are sensitive to syntactic manipulations (Dikker et al., 2009). The former study predicts activations in the anterior STG with additional weaker frontal activations, whereas the latter view predicts the ELANm to be localized in auditory sensory cortices.
- (2)
We sought to localize the neural sources of the MMF elicited by infrequent ITD changes within naturally spoken sentences in order to have a condition which reflects auditory perceptual rule processing (Schröger, 1996, Schröger, 2005). We expected the posterior STG/AC to be sensitive to auditory spatial deviations (Kaiser et al., 2000, Deouell et al., 2006). Based on the dual pathways in the auditory system (Rauschecker and Scott, 2009), the neural sources of the ELANm and MMF were expected to differ in location.
- (3)
By localizing the neural responses to sentences including a double violation, we aimed to investigate how processing a syntactic violation and an auditory spatial violation in parallel is accomplished by different regions in the temporal cortex (Rauschecker and Scott, 2009). We expected the brain regions involved in processing the single syntactic and those involved in processing the auditory spatial violations to be activated simultaneously for sentences containing both violations.
Apart from the effects in the 100–200 ms time window (i.e., the ELANm and MMF), we intended to further elucidate on the very early syntax and simple rule violation effects that precede the ELANm and MMF, and ask whether parallel processing can be observed already at this processing stage.
Section snippets
Participants
Twenty-four healthy, native German-speaking adults (11 female, mean age = 25.3 years, standard error of the mean (SEM) = 1) participated in the MEG study. They were all right-handed as measured by the Edinburgh Handedness Inventory (Oldfield, 1971). The 20th percentile of the laterality quotient was 100 (range: 83–100). All participants gave written consent prior to testing and were paid seven Euros per hour. They had no known hearing deficit or neurological diseases in their history.
Stimulus material
The material
Behavioral task
The overall mean error rate was 2.9% (SEM = 0.5). Statistical comparisons did not reveal any differences between conditions.
Source localization—activation strength
For each time window, the grand average activations of the violation conditions are depicted in Fig. 2. The corresponding STG activation time courses as well as a summary of the statistical comparisons for the two time windows are displayed in Fig. 3.
In the very early time window from 40 to 90 ms, the rmANOVA revealed a main effect of Syntax (F(1,23) = 33.82, P < 0.001, η²G =
Discussion
The goal of the present MEG study was to investigate the underlying neural mechanisms of the parallel processing effect observed for early syntax-related and auditory perceptual-related responses, and to test previous localizations of early responses to syntactically incorrect sentences. The present data replicate earlier findings (Friederici et al., 1993, Friederici et al., 2000, Schröger, 1996, Hahne and Friederici, 1999, Hahne and Friederici, 2002, Hahne et al., 2002, Nager et al., 2003), by
Conclusions
The present MEG study revealed a number of similarities and differences in processing syntactic and auditory spatial information. Approximately 60 ms after stimulus onset, processing of syntactic and auditory spatial deviations led to modulations in the STG, showing that both deviations are processed in parallel when encountered in combination. Following these very early effects, clear evidence of a dissociation of speech-related processes and auditory spatial processes was observed in the
Acknowledgments
We thank Yvonne Wolff for carefully acquiring the data. BH was supported by the Deutsche Forschungsgemeinschaft, graduate program “Function of Attention in Cognition” at the University of Leipzig. We thank three anonymous reviewers for their very helpful comments.
References (74)
- et al.
Effects of feature-selective attention on auditory pattern and location processing
NeuroImage
(2008) - et al.
Midlatency evoked potentials attenuation and augmentation reflect different aspects of sensory gating
Biological Psychology
(1999) - et al.
A review of the evidence for P2 being an independent component process: age, sleep and modality
Clinical Neurophysiology
(2004) - et al.
Sensitivity to syntax in visual cortex
Cognition
(2009) - et al.
Effects of cold stress on early and late stimulus gating
Psychiatry Research
(2005) - et al.
Cortical surface-based analysis II: Inflation, flattening, and a surface-based coordinate system
NeuroImage
(1999) - et al.
Event-related brain potentials during natural speech processing: effects of semantic, morphological and syntactic violations
Cognitive Brain Research
(1993) - et al.
The functional anatomy of the MMN: a DCM study of the roving paradigm
NeuroImage
(2008) - et al.
Thresholding of statistical maps in functional neuroimaging using the false discovery rate
NeuroImage
(2002) - et al.
Differential task effects on semantic and syntactic processes as revealed by ERPs
Cognitive Brain Research
(2002)
Localization of the syntactic mismatch negativity in the temporal cortex: an MEG study
NeuroImage
Violation of syntax and prosody — disentangling their contributions to the early left anterior negativity (ELAN)
Neuroscience Letters
Combined mapping of human auditory EEG and MEG responses
Electroencephalography and Clinical Neurophysiology
Differentiation of syntactic processes in the left and right anterior temporal lobe: event-related brain potential evidence from lesion patients
Brain and Language
Magnetoencephalography detection of early syntactic processing in humans: comparison between L1 speakers and L2 learners of English
Neuroscience Letters
Localizing the distributed language network responsible for the N400 measured by MEG during auditory sentence processing
Brain Research
Localizing pre-attentive auditory memory-based comparison: magnetic mismatch negativity to pitch change
NeuroImage
Whole-head mapping of middle-latency auditory evoked magnetic fields
Electroencephalography and Clinical Neurophysiology
Neurocognition of auditory sentence comprehension: event related fMRI reveals sensitivity to syntactic violations and task demands
Cognitive Brain Research
Early selective-attention effect on evoked potential reinterpreted
Acta Psychologica
Tracking of multiple sound sources defined by interaural time differences: brain potential evidence in humans
Neuroscience Letters
The assessment and analysis of handedness: the Edinburgh Inventory
Neuropsychologia
Automatic processing of grammar in the human brain as revealed by the mismatch negativity
NeuroImage
Hemispheric specialization during discrimination of sound sources reflected by MMN
Neuropsychologia
Interaural time and level differences: integrated or separated processing?
Hearing Research
Direct evidence for differential roles of temporal and frontal components of auditory change detection
Neuropsychologia
Effects of spatial separation and stimulus probability on the event-related potentials elicited by occasional changes in sound location
Brain Research
Functional specialization of the human auditory cortex in processing phonetic and musical sounds: a magnetoencephalographic (MEG) study
NeuroImage
Sensitivity of MEG and EEG to source orientation
Brain Topography
Task-modulated “what” and “where” pathways in human auditory cortex
Proceedings of the National Academy of Sciences
Processing of novel sounds and frequency changes in the human auditory cortex: magnetoencephalographic recordings
Psychophysiology
Recommended effect size statistics for repeated measures designs
Behavior Research Methods
Controlling the false discovery rate: a practical and powerful approach to multiple testing
Journal of the Royal Statistical Society: Series B (Statistical Methodology)
The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages
Psychological Review
Spatial location is accurately tracked by human auditory sensory memory: evidence from the mismatch negativity
The European Journal of Neuroscience
Early occipital sensitivity to syntactic category is based on form typicality
Psychological Science
High-resolution intersubject averaging and a coordinate system for the cortical surface
Human Brain Mapping
Cited by (29)
Acupoint-brain (acubrain) mapping: Common and distinct cortical language regions activated by focused ultrasound stimulation on two language-relevant acupoints
2021, Brain and LanguageCitation Excerpt :The within-modality neuromodulatory effects of acupoints observed in our FUS study may implicate different types of language processing. Previous studies have identified specific brain regions engaged in different types of language processing, for example, bilateral superior temporal gyri during prelexical phonemic speech perception (Leff et al., 2009), BA45 and BA47 during semantic processing (Thompson-Schill, D’Esposito, Aguirre, & Farah, 1997; Wagner, Paré-Blagoev, Clark, & Poldrack, 2001), dorsal BA44 and anterior portion of superior temporal gyrus during syntactic processing (Friederici, 2018; Herrmann, Maess, Hahne, Schröger, & Friederici, 2011), and bilateral superior temporal sulci during sentence comprehension (Kroczek, Gunter, Rysop, Friederici, & Haetwigsen, 2019). The specific brain regions activated by FUS on different acupoints may be related to different types of language processing but our tasks were unable to differentiate between them.
Language and action in Broca's area: Computational differentiation and cortical segregation
2021, Brain and CognitionCitation Excerpt :The findings replicated the early error-detection component found in Maffongelli et al. (2015), and they further showed a facilitatory effect reflected in faster RTs reported for the error detection task. So far, the ELAN component for language has been localized in the temporal cortex and in the inferior frontal cortex or in the temporal cortex only, using MEG and dipole source localization (Groß et al., 1998; Herrmann, Maess, Hahne, Schröger, & Friederici, 2011; Herrmann, Maess, Hasting, & Friederici, 2009; Knösche, Maeß, & Friederici, 1999). On the other hand, the localization of the early component reported for action processing—recorded from a wide centro-frontal region of interest—still awaits further investigation.