Challenges in Studying Multidimensional Semantic Representations in the Human Brain

Concepts are a necessary component of higher-order cognition, and they facilitate the comprehension and production of language. However, because semantic representations are encoded by spatially diverse populations of neurons across frontal, parietal, and temporal lobes, localizing function is difficult (Huth et al., 2016). Adding to the challenge, the similarity of semantic stimuli can be delineated using multiple conceptual dimensions (e.g., shared features, co-occurrence, familiarity), which may be represented by different patterns of distributed neuronal response. These factors have hindered our understanding of how concepts are represented by the brain.

Research on semantic encoding suggests several hypotheses about how conceptual representations are implemented by the brain. One proposal is that concepts are encoded through distributed, anatomically connected, modality-specific regions specialized to process visual, auditory, or spatial features of stimuli (Tyler and Moss, 2001; Wang et al., 2016). Another theory proposes that the neural mechanisms of semantic processing involve a “hub and spoke” configuration, in which a “hub” region of the brain serves as a general purpose intermediary to many inputs and outputs that respond in a specialized manner to particular features of stimuli (Rice et al., 2015; Ralph et al., 2017). The key difference between these theories is whether a centralized hub region is necessary for generating coherent, integrated representations of semantic concepts, or if distributed, decentralized patterns of activation are sufficient.

The anterior temporal lobe (ATL) has been proposed as a hub integrating inputs from “spoke” regions involved in conceptual processing. Evidence for this includes findings that ATL lesions lead to increased rates of category-independent semantic errors (Schwartz et al., 2011); ATL degeneration leads to semantic dementia which is characterized by word naming errors (Ralph et al., 2017); and intracranial recordings in ATL showed modality-independent coding during proper name retrieval and picture naming (Abel et al., 2015; Shimotake et al., 2015). The ATL may also contain graded representations of more focal types of semantic relations, facilitated by cytoarchitectural gradients and anatomical connectivity between ATL and regions specialized for processing sensory features, such as visual or auditory information (Ding et al., 2009; Binney et al., 2010). For example, focal ATL damage in herpes simplex encephalitis leads to category specific deficits in naming biological entities, possibly mediated by deficits in white matter connectivity between ATL and visual regions (Gainotti, 2018).

To clarify the role of potential hub regions in semantic categorization, a recent study by Xu et al. (2018) used fMRI to investigate brain responses during categorization focusing on two dimensions of semantic similarity: taxonomic categorization (stimuli with similar features; e.g., people or objects) and thematic (stimuli that co-occur; e.g., people, places and things associated with sports or school). Participants completed a simple task in which they categorized the same set of words along either taxonomic or thematic dimensions. The authors used representational similarity analysis (RSA) to compare the brain states associated with each form of categorization. RSA uses representative geometry to estimate the information contained within voxel patterns of activation as they relate to different task conditions (Kriegeskorte and Kievet, 2013). Since fMRI has limited temporal resolution and summarizes activation indirectly over very large populations of neurons (on the order of thousands to millions), multivariate methods such as RSA can provide an intermediate descriptive level between neuronal populations and cognitive representations. Xu et al. (2018) examined taxonomic and thematic processing using RSA in two regions of interest, the left ATL and the temporoparietal junction (TPJ). They chose these areas based on results from a large study in stroke patients, which implicated lesions of ATL in increased taxonomic naming errors, and TPJ in thematic naming errors (Schwartz et al. 2011). They also conducted whole-brain searchlight analysis of taxonomic and thematic processing, RSA within two a priori defined semantic networks, and multidimensional scaling (MDS) to map the representational semantic space within ATL and TPJ.

Xu et al. (2018) identified a complex pattern of encoding in the ATL and TPJ, with both regions representing thematic and taxonomic information, but to different degrees. Using RSA, they found that both the left ATL and TPJ regions of interest responded more strongly to taxonomic categorization than to thematic, but TPJ showed a greater response to thematic categorization than ATL. They used Spearman's rank partial correlation to measure representational patterns in the brain during taxonomic categorization while controlling for effects of thematic categorization, and vice versa. Using this approach, they found that responses in ATL during thematic categorization were significant only when controlling for taxonomic categorization, whereas TPJ significantly represented both taxonomic and thematic categorization. These findings suggest that populations of neurons in ATL and TPJ represent both thematic and taxonomic relations, but thematic relations are embedded within taxonomic relations. Another interpretation is that patterns of neuronal firing in response to shared features may lead to more stable patterns for the similarity-based taxonomic category compared with the co-occurrence based thematic category (see limitations discussion below).

Whole-brain searchlight analyses revealed a similarly complex pattern, where brain response to thematic processing was stronger when controlling for taxonomic processing, and vice versa. A distributed network of regions across the parietal, frontal, and occipital cortex responded during taxonomic categorization, and this network corresponded well with foci identified in a meta-analysis of semantic processing (Binder et al., 2009). When examining taxonomic categorization and controlling for thematic, these results became stronger and more distributed. Notably, however, ATL did not show a significant response during taxonomic categorization measured using whole-brain searchlight analysis.

Based on lesion results and hub and spoke theories of semantic processing, the ATL has been proposed to function as a hub that integrates inputs from other regions to facilitate behaviors such as categorization. However Xu et al. (2018) only detected significant ATL responses during taxonomic categorization when using RSA at the ROI level, and then only with a modest Spearman's r = 0.24. ATL responses were not significant during taxonomic categorization when measured using whole-brain searchlight analysis. These findings suggest that while some neurons in ATL represent taxonomic information, this representation is not as strong as in other regions in the whole-brain analysis. This calls into question whether ATL meets the definition of a hub in this context. Thus, the results seem to better support the distributed theory of semantic network function. Still, it is possible that the hub role of the ATL would be delineated more clearly using metrics intended to capture how essential a region is to the function of the semantic network, such as connectivity analysis (Hoffman et al., 2015; Jung and Lambon Ralph, 2016; Wang et al., 2016), or with imaging sequences better optimized for signal-to-noise in this region (see meta-analysis by Visser et al., 2010). The ATL might show strong connectivity in concert with regions that encode modality-specific stimulus features to facilitate coherent conceptual representations, despite not showing a strong response on its own.

Overall, these findings suggest that in healthy young adults, taxonomic and thematic categorization are not well behaved processes that correspond neatly to distinct neural substrates, despite what lesion studies and theories of semantic encoding might suggest (Schwartz et al., 2011; Ralph et al., 2017). Semantic categorization instead appears to rely on a complex, distributed, multivariate signal where discrete regions represent multiple dimensions of categorization even while preferring certain dimensions over others. The authors interpret their findings of greater representation of taxonomic relations in ATL and TPJ relative to thematic processing to mean that thematic processing is subordinate to, or embedded within, taxonomic processing. However, there are several concerns that limit interpretation of these findings. First, there appears to be variability in the typicality of word stimuli within their taxonomic or thematic domain. For example, the place words associated with sports, such as stadium, award platform, etc. could also apply to school events. This is illustrated by the results of multidimensional scaling, which was used to measure the distance in representational space between types of stimuli; stimuli that are closer in representational space tend to recruit more similar forms of processing than stimuli that are further apart. Within ATL and TPJ, sports location stimuli grouped more closely to school and medical locations than to sports-related people or object stimuli. Similarly, the school object stimuli such as chalk and ruler can map onto other dimensions (e.g., art supplies), and group more closely to other objects rather than the other school stimuli in the MDS plots, suggesting a lack of specificity of stimulus-category mappings. If participant perception of stimulus-category ambiguity contributed to variability in strength of neuronal representations, accounting for this explicitly might reveal more stable representational patterns, such as stronger thematic categorization in TPJ, without needing to first account for taxonomic processing. This might help clarify whether thematic processing is truly embedded within taxonomic processing in the brain, and whether ATL and TPJ serve as hubs or as part of a decentralized distributed network.

The findings of Xu et al. (2018) also highlight challenges of interpreting differences in neuronal responses while comparing taxonomic and thematic categorization (Mirman et al., 2017). Thematic stimuli are linked to each other based on co-occurrence, and experiences of co-occurrence vary across individuals. This may lead to discrepancies in the stability of stimulus representations across participants and interfere with the reliability of group level inference, as has been demonstrated for representations of other concepts that vary in stability (e.g., concrete versus abstract categorization; Anderson et al., 2014). This potential confound adds ambiguity as to whether there is truly an embedded structure of thematic relations within taxonomic relations in ATL and TPJ. This could be clarified by testing whether within- versus between-subject classification accuracy varies based on taxonomic and thematic categorization. More generally, these results highlight challenges in interpreting brain response during laboratory experiments, which may not generalize to the more complex naturalistic context (Krakauer et al., 2017). There are a wide range of approaches used to study semantic categorization, ranging from the simple, hypothesis-driven experimental approaches such as Xu et al. (2018) to the data-driven analysis of hours of passive listening of Huth et al. (2016). It remains challenging to find a middle ground between experimental control in the laboratory and the ecological validity of capturing a process close to how it happens in the real world.

In sum, Xu et al. (2018) shed light on how distributed populations of neurons can flexibly represent taxonomic and thematic categorization. Their results suggest a more nuanced understanding of semantic categorization than a 1:1 mapping of function to specific brain regions and demonstrate that even potentially specialized areas of the ATL and TPJ represent categorical dimensions outside their preferred domain. More research will be needed to discover the boundaries of this flexible representation and how these functions subserve real-world behaviors.