Object Domain and Modality in the Ventral Visual Pathway

doi:10.1016/j.tics.2016.02.002

Trends in Cognitive Sciences

Volume 20, Issue 4, April 2016, Pages 282-290

https://doi.org/10.1016/j.tics.2016.02.002 Get rights and content

Trends

A wave of recent studies has reported similar domain preference effects in ventral occipital temporal cortex (VOTC) in sighted and congenitally blind individuals, leading to the contention that object representation in this region is multi-modal.

However, the effects of visual deprivation on object category selectivity paint an intriguing pattern of heterogeneity: selectivity to spatial navigation stimuli and manipulable artifacts found to be robustly multi-modal, whereas selectivity to animate objects reliably present only for sighted individuals when processing visual stimuli.

We formulate a novel conjecture about the nature of representations in VOTC: representation types are partly driven by the nature of the mapping between object visual properties and other object properties, which differ between animate and inanimate objects.

The nature of domain-specific organization in higher-order visual cortex (ventral occipital temporal cortex, VOTC) has been investigated both in the case of visual experience deprivation and of modality of stimulation in sighted individuals. Object domain interacts in an intriguing and revelatory way with visual experience and modality of stimulation: selectivity for artifacts and scene domains is largely immune to visual deprivation and is multi-modal, whereas selectivity for animate items in lateral posterior fusiform gyrus is present only with visual stimulation. This domain-by-modality interaction is not readily accommodated by existing theories of VOTC representation. We conjecture that these effects reflect a distinction between the visual features that characterize different object domains and their interaction with different types of downstream computational systems.

Section snippets

Ventral Visual Cortex: Visual or Multi-Modal?

A core assumption of cognitive science and cognitive neuroscience is that the brain processes information at various levels of representation, progressing from those closely tied to stimulus features to increasingly more general and abstract representations. One of the mysteries in this framework is the transition from modality specific representations – those explicable fully in the language of a given modality – to representations that capture other properties of the object – such as, for

Domain Specificity Effects across Different Modalities Are Different

When sighted individuals view pictures, various clusters in VOTC are more responsive to certain categories of objects, such as faces, bodies, tools, or places. The overall distribution of category preference follows a broad animate versus inanimate distinction, with a further differentiation within the inanimate domain between manipulable and non-manipulable objects. This results in a tripartite organization, from ventral medial regions (parahippocampal and medial fusiform) showing preference

Relationships between Visual and Other Object Properties

We formulate a novel conjecture about one of the factors that determines the nature of representations in VOTC: the difference across object domains in terms of modality effects lies in the relationship between visual shape and its functional relevance, understood as the types of computations it triggers downstream and ultimately its connectivity structure. Visual shape strongly constrains the way in which we interact physically with inanimate objects, but much less so and in a far less

Concluding Remarks

We began this review article by drawing attention to an empirical phenomenon regarding the effect of visual experience (and stimulus input) on object representations in higher-order visual cortex, showing that contrary to recent claims that this territory is multi-modal, there is a clear animate/inanimate dimension along which effects of modality differ. This empirical pattern motivates a novel conjecture about the nature of representations in VOTC: representation types are partly driven by the

Acknowledgments

This work was supported by the National Basic Research Program of China (2013CB837300) and National Natural Science Foundation of China (31221003, 31500882). A.C. was supported by the Fondazione Cassa di Risparmio di Trento e Rovereto.

Glossary

Connectional fingerprints: the unique set of anatomical or functional connections a cortical region owns, which could be measured as the vector of the cortical region's connection strengths with other cerebral regions.
Functional fingerprints: the unique response properties a cortical region exhibits, which could be measured as vector of the region's response strengths to a variety of stimuli or tasks (e.g., object categories).
Multi-modal: for the purpose of this review, multi-modal was used to mean

References (77)

U. Hasson
Eccentricity bias as an organizing principle for human high-order object areas
Neuron
(2002)
T. Konkle et al.
A real-world size organization of object responses in occipitotemporal cortex
Neuron
(2012)
B.Z. Mahon
Category-specific organization in the human brain does not require visual experience
Neuron
(2009)
T. Wolbers
Modality-independent coding of spatial layout in the human brain
Curr. Biol.
(2011)
E. Striem-Amit et al.
Visual cortex extrastriate body-selective area activation in congenitally blind people ‘Seeing’ by using sounds
Curr. Biol.
(2014)
C. He
Selectivity for large nonmanipulable objects in scene-selective visual cortex does not require visual experience
Neuroimage
(2013)
R. Adam et al.
Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex
Neuroimage
(2010)
A. Ishai
Distributed neural systems for the generation of visual images
Neuron
(2000)
N. Kriegeskorte
Matching categorical object representations in inferior temporal cortex of man and monkey
Neuron
(2008)
S.L. Fairhall et al.
Category-selective neural substrates for person- and place-related concepts
Cortex
(2013)

R. Kitada

Brain networks involved in haptic and visual identification of facial expressions of emotion: an fMRI study

Neuroimage

(2010)

A.R. Kilgour

Haptic face identification activates ventral occipital and temporal areas: an fMRI study

Brain Cogn.

(2005)

J.T. Devlin

Category-related activation for written words in the posterior fusiform is task specific

Neuropsychologia

(2005)

J.T. Devlin

Anatomic constraints on cognitive theories of category specificity

Neuroimage

(2002)

J.T. Devlin

Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI

Neuropsychologia

(2002)

S. Lambert

Blindness and brain plasticity: contribution of mental imagery? An fMRI study

Cogn. Brain Res.

(2004)

S. Lambert

Neural substrates of animal mental imagery: calcarine sulcus and dorsal pathway involvement – an fMRI study

Brain Res.

(2002)

O. Doehrmann

Probing category selectivity for environmental sounds in the human auditory brain

Neuropsychologia

(2008)

L.R. Engel

Different categories of living and non-living sound-sources activate distinct cortical networks

Neuroimage

(2009)

M. Costantini

Haptic perception and body representation in lateral and medial occipito-temporal cortices

Neuropsychologia

(2011)

G. Le Clec’H

Distinct cortical areas for names of numbers and body parts independent of language and input modality

Neuroimage

(2000)

B.Z. Mahon et al.

What drives the organization of object knowledge in the brain?

Trends Cogn. Sci.

(2011)

E. Striem-Amit

Reading with sounds: sensory substitution selectively activates the visual word form area in the blind

Neuron

(2012)

T. Hannagan

Origins of the specialization for letters and numbers in ventral occipitotemporal cortex

Trends Cogn. Sci.

(2015)

A. Caramazza et al.

The organization of conceptual knowledge: the evidence from category-specific semantic deficits

Trends Cogn. Sci.

(2003)

K.S. Weiner

The mid-fusiform sulcus: a landmark identifying both cytoarchitectonic and functional divisions of human ventral temporal cortex

Neuroimage

(2014)

D.C. Van Essen

The WU-Minn Human Connectome Project: an overview

Neuroimage

(2013)

I. Levy

Center–periphery organization of human object areas

Nat. Neurosci.

(2001)

M.J. Arcaro

Retinotopic organization of human ventral visual cortex

J. Neurosci.

(2009)

S. Nasr

Thinking outside the box: rectilinear shapes selectively activate scene-selective cortex

J. Neurosci.

(2014)

T.T. Rogers

Fusiform activation to animals is driven by the process, not the stimulus

J. Cogn. Neurosci.

(2005)

I. Gauthier

Activation of the middle fusiform ‘face area’ increases with expertise in recognizing novel objects

Nat. Neurosci.

(1999)

R. Kitada

The brain network underlying the recognition of hand gestures in the blind: the supramodal role of the extrastriate body area

J. Neurosci.

(2014)

M.V. Peelen

Tool selectivity in left occipitotemporal cortex develops without vision

J. Cogn. Neurosci.

(2013)

E. Ricciardi

Mind the blind brain to understand the sighted one! Is there a supramodal cortical functional architecture?

Neurosci. Biobehav. Rev.

(2013)

S.L. Fairhall

Person- and place-selective neural substrates for entity-specific semantic access

Cereb. Cortex

(2014)

E. Ricciardi et al.

New light from the dark: what blindness can teach us about brain function

Curr. Opin. Neurol.

(2011)

K.M. O’Craven et al.

Mental imagery of faces and places activates corresponding stimulus-specific brain regions

J. Cogn. Neurosci.

(2000)

Cited by (89)

ROSE: A neurocomputational architecture for syntax
2024, Journal of Neurolinguistics
A comprehensive neural model of language must accommodate four components: representations, operations, structures and encoding. Recent intracranial research has begun to map out the feature space associated with syntactic processes, but the field lacks a unified framework that can direct invasive neural analyses. This article proposes a neurocomputational architecture for syntax, termed ROSE (Representation, Operation, Structure, Encoding). Under ROSE, the basic data structures of syntax are atomic features, types of mental representations (R), and are coded at the single-unit and ensemble level. Operations (O) transforming these units into manipulable objects accessible to subsequent structure-building levels are coded via high frequency broadband γ activity. Low frequency synchronization and cross-frequency coupling code for recursive structural inferences (S). Distinct forms of low frequency coupling encode these structures onto distinct workspaces (E). Causally connecting R to O is spike-phase/LFP coupling; connecting O to S is phase-amplitude coupling; connecting S to E are frontotemporal traveling oscillations. ROSE is reliant on neurophysiologically plausible mechanisms and provides an anatomically precise and falsifiable grounding for natural language syntax.
Sentence-level embeddings reveal dissociable word- and sentence-level cortical representation across coarse- and fine-grained levels of meaning
2024, Brain and Language
In this large-sample (N = 64) fMRI study, sentence embeddings (text-embedding-ada-002, OpenAI) and representational similarity analysis were used to contrast sentence-level and word-level semantic representation. Overall, sentence-level information resulted in a 20–25 % increase in the model’s ability to captures neural representation when compared to word-level only information (word-order scrambled embeddings). This increase was relatively undifferentiated across the cortex. However, when coarse-grained (across thematic category) and fine-grained (within thematic category) combinatorial meaning were separately assessed, word- and sentence-level representations were seen to strongly dissociate across the cortex and to do so differently as a function of grain. Coarse-grained sentence-level representations were evident in occipitotemporal, ventral temporal and medial prefrontal cortex, while fine-grained differences were seen in lateral prefrontal and parietal cortex, middle temporal gyrus, the precuneus, and medial prefrontal cortex. This result indicates dissociable cortical substrates underly single concept versus combinatorial meaning and that different cortical regions specialise for fine- and coarse-grained meaning.
Mechanisms underlying category learning in the human ventral occipito-temporal cortex
2024, NeuroImage
The human ventral occipito-temporal cortex (VOTC) has evolved into specialized regions that process specific categories, such as words, tools, and animals. The formation of these areas is driven by bottom-up visual and top-down nonvisual experiences. However, the specific mechanisms through which top-down nonvisual experiences modulate category-specific regions in the VOTC are still unknown. To address this question, we conducted a study in which participants were trained for approximately 13 h to associate three sets of novel meaningless figures with different top-down nonvisual features: the wordlike category with word features, the non-wordlike category with nonword features, and the visual familiarity condition with no nonvisual features. Pre- and post-training functional MRI (fMRI) experiments were used to measure brain activity during stimulus presentation. Our results revealed that training induced a categorical preference for the two training categories within the VOTC. Moreover, the locations of two training category-specific regions exhibited a notable overlap. Remarkably, within the overlapping category-specific region, training resulted in a dissociation in activation intensity and pattern between the two training categories. These findings provide important insights into how different nonvisual categorical information is encoded in the human VOTC.
Vision matters for shape representation: Evidence from sculpturing and drawing in the blind
2024, Cortex
Shape is a property that could be perceived by vision and touch, and is classically considered to be supramodal. While there is mounting evidence for the shared cognitive and neural representation space between visual and tactile shape, previous research tended to rely on dissimilarity structures between objects and had not examined the detailed properties of shape representation in the absence of vision. To address this gap, we conducted three explicit object shape knowledge production experiments with congenitally blind and sighted participants, who were asked to produce verbal features, 3D clay models, and 2D drawings of familiar objects with varying levels of tactile exposure, including tools, large nonmanipulable objects, and animals. We found that the absence of visual experience (i.e., in the blind group) led to stronger differences in animals than in tools and large objects, suggesting that direct tactile experience of objects is essential for shape representation when vision is unavailable. For tools with rich tactile/manipulation experiences, the blind produced overall good shapes comparable to the sighted, yet also showed intriguing differences. The blind group had more variations and a systematic bias in the geometric property of tools (making them stubbier than the sighted), indicating that visual experience contributes to aligning internal representations and calibrating overall object configurations, at least for tools. Taken together, the object shape representation reflects the intricate orchestration of vision, touch and language.
Similar representation of names and faces in the network for person perception
2023, NeuroImage
Person-knowledge encompasses the diverse types of knowledge we have about other people. This knowledge spans the social, physical, episodic, semantic & nominal information we possess about others and is served by a distributed cortical network including core (perceptual) and extended (non-perceptual) subsystems. Our understanding of this cortical system is tightly linked to the perception of faces and the extent to which cortical knowledge-access processes are independent of perception is unclear. In this study, participants were presented with the written names of famous people and performed ten different semantic access tasks drawn from five cognitive domains (biographic, episodic, nominal, social and physical). We used representational similarity analysis, adapted to investigate network-level representations (NetRSA) to characterise the inter-regional functional coordination within the non-perceptual extended subsystem across access to varied forms of person-knowledge. Results indicate a hierarchical cognitive taxonomy consistent with that seen during face-processing and forming the same three macro-domains: socio-perceptual judgements, episodic-semantic memory and nominal knowledge. The coordination across regions was largely preserved within elements of the extended system associated with internalised cognition but differed in prefrontal regions. Results suggest the elements of the extended system work together in a consistent way to access knowledge when viewing faces and names but that coordination patterns also change as a function of input-processing demands.
Brain intrinsic connection patterns underlying tool processing in human adults are present in neonates and not in macaques
2022, NeuroImage
Tool understanding and use are supported by a dedicated left-lateralized, intrinsically connected network in the human adult brain. To examine this network's phylogenetic and ontogenetic origins, we compared resting-state functional connectivity (rsFC) among regions subserving tool processing in human adults to rsFC among homologous regions in human neonates and macaque monkeys (adolescent and mature). These homologous regions formed an intrinsic network in human neonates, but not in macaques. Network topological patterns were highly similar between human adults and neonates, and significantly less so between humans and macaques. The premotor-parietal rsFC had most significant contribution to the formation of the neonatal tool network. These results suggest that an intrinsic brain network potentially supporting tool processing exists in the human brain prior to individual tool use experiences, and that the premotor-parietal functional connection in particular offers a brain basis for complex tool behaviors specific to humans.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ReviewObject Domain and Modality in the Ventral Visual Pathway

Trends

Section snippets

Ventral Visual Cortex: Visual or Multi-Modal?

Domain Specificity Effects across Different Modalities Are Different

Relationships between Visual and Other Object Properties

Concluding Remarks

Acknowledgments

Glossary

Neuron

Neuron

Neuron

Curr. Biol.

Curr. Biol.

Neuroimage

Neuroimage

Neuron

Neuron

Cortex

Neuroimage

Brain Cogn.

Neuropsychologia

Neuroimage

Neuropsychologia

Cogn. Brain Res.

Brain Res.

Neuropsychologia

Neuroimage

Neuropsychologia

Neuroimage

Trends Cogn. Sci.

Neuron

Trends Cogn. Sci.

Trends Cogn. Sci.

Neuroimage

Neuroimage

Center–periphery organization of human object areas

Nat. Neurosci.

Retinotopic organization of human ventral visual cortex

J. Neurosci.

Thinking outside the box: rectilinear shapes selectively activate scene-selective cortex

J. Neurosci.

Fusiform activation to animals is driven by the process, not the stimulus

J. Cogn. Neurosci.

Activation of the middle fusiform ‘face area’ increases with expertise in recognizing novel objects

Nat. Neurosci.

The brain network underlying the recognition of hand gestures in the blind: the supramodal role of the extrastriate body area

J. Neurosci.

Tool selectivity in left occipitotemporal cortex develops without vision

J. Cogn. Neurosci.

Mind the blind brain to understand the sighted one! Is there a supramodal cortical functional architecture?

Neurosci. Biobehav. Rev.

Person- and place-selective neural substrates for entity-specific semantic access

Cereb. Cortex

New light from the dark: what blindness can teach us about brain function

Curr. Opin. Neurol.

Mental imagery of faces and places activates corresponding stimulus-specific brain regions

J. Cogn. Neurosci.

Review
Object Domain and Modality in the Ventral Visual Pathway