Review article
Redefining the resolution of semantic knowledge in the brain: Advances made by the introduction of models of semantics in neuroimaging

https://doi.org/10.1016/j.neubiorev.2019.05.015Get rights and content

Highlights

  • Semantic models capture our rich conceptual knowledge.

  • These models can in part explain how our brain represents semantic similarity.

  • They proved the hypothesis of concept representation in perirhinal cortex.

  • Semantic models can be used to untangle perceptual and conceptual effects.

  • These models could also be used to evaluate semantic deficits in patients.

Abstract

The boundaries of our understanding of conceptual representation in the brain have been redrawn since the introduction of explicit models of semantics. These models are grounded in vast behavioural datasets acquired in healthy volunteers. Here, we review the most important techniques which have been applied to detect semantic information in neuroimaging data and argue why semantic models are possibly the most valuable addition to the research of semantics in recent years. Using multivariate analysis, predictions based on patient lesion data have been confirmed during semantic processing in healthy controls. Secondly, this new method has given rise to new research avenues, e.g. the detection of semantic processing outside of the temporal cortex. As a future line of work, the same research strategy could be useful to study neurological conditions such as the semantic variant of primary progressive aphasia, which is characterized by pathological semantic processing.

Introduction

The neurobiological basis of semantic knowledge is a key topic in cognitive neuroscience. On average, human adults know between 16,000 and 58,000 words (Keuleers et al., 2015) and across the lifespan we continue to expand our mental lexicon (Shafto and Tyler, 2014). A vital characteristic of the mental lexicon is that many related words exist, referring to similar but discrete concepts (e.g. cucumber versus zucchini, blackbird versus crow). However, in the last decades, a common strategy to map the semantic network is to examine response differences elicited by stimuli (pictures, words) referring to very different entities, for instance faces, buildings, animals and tools. Comparing across these broad categories then results in a rough map of representations of these concepts in the brain. This approach has resulted in several seminal insights in the neurobiology of language (for a review: Binder et al. (2009); Price (2012)). For one, it demonstrated which regions of the brain are activated by semantic processing and which are not. However, the results do not shed any light on how our brains represent meaning for all but the coarsest conceptual differences (e.g. distinguishing furniture and animal concepts). Recent advances in cognitive science, linguistics, and neuroimaging resulted in the use of fine-grained semantic models that can capture in depth the semantic similarity and differences between closely related concepts by replacing one-dimensional characterization (e.g. animate-inanimate) by a multi-dimensional approach derived from a large number of variables (e.g. concept features). Using multivariate analysis, the behavioural relationship between the concepts can be compared to response patterns in the brain elicited by the same concepts (Kriegeskorte et al., 2006). The goal of this paper is to critically review different types of semantic models and to demonstrate how they can be used to elucidate the neurobiology of language using neuroimaging. We review similarities and differences between these recent semantic models and describe how they have led to several breakthroughs in the field of semantic processing. We also propose novel approaches to implement subject-specific semantic models in order to study semantic deficits using neuroimaging. Ultimately, learning about the structure of our lexicon allows us to learn the way our brain structures information.

Section snippets

What are semantic models?

The most import aim of constructing semantic models is to provide a quantifiable version of our mental lexicon. Semantic models can be generated using very different strategies. These types of semantic models can represent semantic similarity between any pair of concepts and provide a continuous numerical estimate of their proximity in space. Such an approach contrasts with the “categorical” approach, where a binary measure is used for every concept that reflects whether or not a concept is

How can semantic models be tested on neuroimaging data?

Semantic models provide a quantitative way of describing the pairwise relationships between concepts, and we ask whether this semantic information can partially explain the variable neuronal response to these concepts. As discussed in Naselaris and Kay (2015), multivariate pattern analysis (MVPA) techniques can be used without semantics models, for instance by using semantic categories. However, the use of semantic models have several advantages. particular, they do not require a priori

Advances made by using semantic models

Following the introduction of semantic models and the most common approaches to assess whether neural activity changes reflect semantic information, we discuss a number of advances made in characterizing the neurobiology of language using models of semantics. First, we discuss how semantic models have been successfully implemented to test a priori hypotheses about semantic information processing in the anterior temporal lobe. This is a specific instance where semantic models convincingly

Future research using models of semantics

A next step would be the introduction to semantic models in neuroimaging in patients with a semantic deficit. Only a few attempts have been made to map the semantic space of patients, and most work has been performed in patients suffering from psychiatric diseases (de Boer et al., 2018) and in Alzheimer’s disease (Pakhomov et al., 2015, 2012; Pakhomov and Hemmy, 2014). A notable study in the domain of PPA analyzed unconstrained speech of patients with the semantic variant of PPA (Hoffman et

Conclusion

The introduction of semantic models to functional neuroimaging of the semantic network has been overall very successful. Various multivariate approaches can be used to test whether neural activity patterns, measured using fMRI, MEG or intracranial recording, reflect semantic information. An advantage of semantic models is that they offer a model that can be used to study the meaning of concepts derived from pictures but also across other modalities such as written and spoken words. Because of

Funding

This work was supported by Federaal Wetenschapsbeleid [Belspo 7/11]; FWO [Grant no. G0925.15] and KU Leuven [OT/12/097, C14/17/108]. R.V. is a Senior Clinical Investigator and R.B. is a postdoctoral fellow of the Research Foundation Flanders (FWO). S.D.D. was supported through ARC grant DE140101749.

Glossary

Convolutional neural network (CNN)
a computational architecture containing a series of layers likening the architecture of the visual cortex. Across the layers, the input data is transformed. Layers can be fully connected and contain an equal number of units, or during convolution, inputs across a layer can be pooled to a smaller number of units in the next layer
Word Embeddings
n-dimensional real-value vectors with n typically ranging between 1 to 1000 elements. These vectors capture the

References (108)

  • M. Feilong et al.

    Reliable individual differences in fine-grained cortical functional architecture

    NeuroImage

    (2018)
  • E. Freud et al.

    ‘What’ is happening in the dorsal visual pathway

    Trends Cogn. Sci.

    (2016)
  • A.D. Halai et al.

    Dual-echo fMRI can detect activations in inferior temporal lobe during intelligible speech comprehension

    NeuroImage

    (2015)
  • P. Hoffman et al.

    Broadly speaking: vocabulary in semantic dementia shifts towards general, semantically diverse words

    Cortex Lang. Comput. Cogn. Neurosci.

    (2014)
  • A.G. Huth et al.

    A continuous semantic space describes the representation of thousands of object and action categories across the human brain

    Neuron

    (2012)
  • N. Kriegeskorte et al.

    Interpreting encoding and decoding models

    Curr. Opin. Neurobiol.

    (2019)
  • N. Kriegeskorte et al.

    Representational geometry: integrating cognition, computation, and the brain

    Trends Cogn. Sci.

    (2013)
  • A.C.H. Lee et al.

    Perirhinal cortex activity during visual object discrimination: an event-related fMRI study

    NeuroImage

    (2006)
  • A.G. Liuzzi et al.

    Left perirhinal cortex codes for similarity in meaning between written words: comparison with auditory word input

    Neuropsychologia

    (2015)
  • A.G. Liuzzi et al.

    Cross-modal representation of spoken and written word meaning in left pars triangularis

    NeuroImage

    (2017)
  • P. Mandera et al.

    Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: a review and empirical validation

    J. Mem. Lang.

    (2017)
  • T. Naselaris et al.

    Resolving ambiguities of MVPA using explicit models of representation

    Trends Cogn. Sci.

    (2015)
  • S.V.S. Pakhomov et al.

    A computational linguistic measure of clustering behavior on semantic verbal fluency task predicts risk of future dementia in the Nun study

    Cortex

    (2014)
  • S.V.S. Pakhomov et al.

    Automated semantic indices related to cognitive function and rate of cognitive decline

    Neuropsychologia

    (2012)
  • S.V.S. Pakhomov et al.

    Language networks associated with computerized semantic indices

    NeuroImage

    (2015)
  • C.J. Price

    A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading

    NeuroImage

    (2012)
  • K. Rupp et al.

    Semantic attributes are encoded in human electrocorticographic signals during visual object recognition

    NeuroImage

    (2017)
  • M. Steyvers

    Combining feature norms and text data with topic models

    Acta Psychol. (Amst.)

    (2010)
  • G. Sudre et al.

    Tracking neural coding of perceptual and semantic features of concrete nouns

    NeuroImage

    (2012)
  • K.I. Taylor et al.

    Contrasting effects of feature-based statistics on the categorisation and basic-level identification of visual objects

    Cognition

    (2012)
  • A.J. Anderson et al.

    Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation

    Cereb. Cortex

    (2016)
  • W.A. Bainbridge et al.

    Dissociating neural markers of stimulus memorability and subjective recognition during episodic retrieval

    Sci. Rep.

    (2018)
  • L.W. Barsalou

    Grounded cognition

    Annu. Rev. Psychol.

    (2007)
  • R. Beckwith et al.

    Wordnet: a lexical database organized on psycholinguistic principles

    Lexical Aquisition: Exploiting On-line Resources to Build a Lexicon

    (1991)
  • J.R. Binder et al.

    Toward a brain-based componential semantic representation

    Cogn. Neuropsychol.

    (2016)
  • J.R. Binder et al.

    Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies

    Cereb. Cortex N. Y. NY

    (2009)
  • R. Bruffaerts et al.

    Similarity of fMRI activity patterns in left perirhinal cortex reflects semantic similarity between words

    J. Neurosci.

    (2013)
  • A.M. Chan et al.

    First-pass selectivity for semantic categories in human anteroventral temporal lobe

    J. Neurosci.

    (2011)
  • I. Charest et al.

    Unique semantic space in the brain of each beholder predicts perceived similarity

    Proc. Natl. Acad. Sci. U. S. A.

    (2014)
  • R.M. Cichy et al.

    Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

    Sci. Rep.

    (2016)
  • R.M. Cichy et al.

    Resolving human object recognition in space and time

    Nat. Neurosci.

    (2014)
  • A. Clarke et al.

    Predicting the time course of individual objects with MEG

    Cereb. Cortex

    (2015)
  • A. Clarke et al.

    Object-specific semantic coding in human perirhinal cortex

    J. Neurosci.

    (2014)
  • S.J. Crutch et al.

    Abstract conceptual feature ratings: the role of emotion, magnitude, and other cognitive domains in the organization of abstract conceptual knowledge

    Front. Hum. Neurosci.

    (2013)
  • S. De Deyne et al.

    Visual and Affective Grounding in Language and Mind

    (2019)
  • S. De Deyne et al.

    The “Small World of Words” English word association norms for over 12,000 cue words

    Behav. Res. Methods

    (2018)
  • S. De Deyne et al.

    Sources of semantic proximity

    Presented at the Proceedings of the 31th Annual Conference of the Cognitive Science Society

    (2009)
  • S. De Deyne et al.

    Word associations: norms for 1,424 Dutch words in a continuous task

    Behav. Res. Methods

    (2008)
  • S. De Deyne et al.

    Exemplar by feature applicability matrices and other Dutch normative data for semantic concepts

    Behav. Res. Methods

    (2008)
  • A.M. de Groot

    Representational aspects of word imageability and word frequency as assessed through word association

    J. Exp. Psychol. Learn. Mem. Cogn.

    (1989)
  • Cited by (0)

    View full text