ReviewIdentification and integration of sensory modalities: Neural basis and relation to consciousness
Introduction
Over the past decades, neuroscience has made considerable progress in elucidating neural substrates underlying mental processes such as memory, attention, discrimination and evaluation of sensory inputs, and voluntary control over behavior. These processes have proven to be tractable and amenable to investigation with the methodologies of contemporary neuroscience. However, the enterprise of elucidating their neural basis hinges on the implicit agreement to refrain from probing into deeper questions related to the phenomenal content represented by neural activity. For instance, attention is commonly viewed as the selection of a body of information for intensive processing at the expense of other objects deemed less relevant; it can be modelled by neural circuits performing filtering, synchronizing, noise-suppressing and related cybernetic operations. The widespread use of the term ‘information’ in the sense of Shannon’s definition (1948) is revealing in this context, since it can be conveniently applied to both neural signals (e.g. spike trains) and mental phenomena, without exerting force to explain the relationship between these two domains. But eventually cognitive neuroscience needs to address the question of how neural activity that represents the object of attention correlates to the phenomenal content we are conscious of. The current review attempts to make a few steps in addressing this question by considering, first, how the brain may come to identify different streams of information as belonging to distinct sensory modalities. The second part of this review will be concerned with neural mechanisms to integrate these streams in producing a global, multimodal experience.
To prevent confusion about what an account of the problem presented here should or should not hold, it is mandatory to contrast two different questions. First, in a dialog on the experience of a colored object one may ask another person: “how do you see the color red?”, or: “do you see the color red in the same way I do?” This inquiry immediately runs into the difficulties in investigating other people’s private experiences (e.g. Wittgenstein, 1953), which is an area of philosophical controversy not touched upon here. Second, however, one can reasonably ask the question, based on an experience shared and reported by most people: “if I see an object with its three-dimensional shape, size, pattern of light reflections and surface texture, how is it that I also perceive this other property of the object we call ‘color’?” Similarly: “How is it that we can see a color in the first place, as distinct from other object properties?” This second question addresses how we come to experience modalities and submodalities as being phenomenally different. The definition of ‘modality’ used here relies on the commonly used distinction between the main senses (vision, touch, hearing, smell, taste, pain, thermoception, proprioception and the vestibular senses). Within a main modality one can usually distinguish submodalities (for vision: 3D-shape, movement, color, size, surface texture, etc.).
Neural substrates of consciousness have been studied almost exclusively in the visual domain, where a considerable amount of neural-substrate and neural-correlate research has been done. However, this approach bears the risk of focusing too narrowly on a single, unimodal system, without addressing commonalities and differences between vision and other modalities. Once it has been decided to focus exclusively on visual perception, the question of why there is a visual modality in the first place, as distinct from other modalities, does not have to be addressed. One can freely apply the term ‘information’ to both the tuning properties of visual cortex neurons and their perceptual counterparts without having to confront the question what makes this ‘visual’ information different from ‘auditory’ information. Shannon’s information theory (1948) offers no solution in this respect because it is concerned with the statistical relationships between sets of inputs (stimuli) and outputs (responses), not with the content or the semantics of the information under scrutiny (cf. Eggermont, 1998, Rieke et al., 1997). The existence of different sensory modalities seems so natural and basic to us that the question of how the brain produces distinct experiences of smell, taste, vision, sound and touch hardly ever comes up.
Although the subject of this paper is not to review recent philosophical debates and controversies on qualitative aspects of consciousness (or: “qualia”), it is worthwhile to briefly touch upon some relevant theoretical issues setting the background stage for the current, primarily neuroscientific discourse. For philosophical reviews on this topic, the reader is referred to Nagel, 1974, Jackson, 1986, Dennett, 1991, Searle, 1992, Searle, 2000, Chalmers, 1995, Chalmers, 1996, Churchland, 1995, Tye, 2000, Shear, 2000. For theories more strongly driven by experimental and computational neuroscience, see Churchland and Sejnowski, 1992, Crick, 1994, Tononi and Edelman, 1998, O’Brien and Opie, 1999, O’Regan and Noë, 2001, Zeki, 2001, Dehaene and Naccache, 2001, Rees et al., 2002, Taylor, 2003, Crick and Koch, 2003, Koch, 2004, Block, 2005.
This paper adopts a specific viewpoint on the general problem of qualitative experience by asking which kind of neural operations may lie at the basis of the existence of multiple sensory modalities. This question is more neuroscience-oriented yet wider in scope than the classical ‘qualia’ problem, which by tradition focuses on the ‘secondary qualities’ of objects such as color and smell, but much less on properties that can be unambiguously quantified (e.g. motion, shape and extension) and were therefore considered primary by Locke (1667). The qualia problem has not been unanimously considered a serious object of study, partly due to difficulties in maintaining a fundamental distinction between primary and secondary qualities (Boring, 1950). More recently, functionalist or physicalist accounts such as by Dennett, 1991, Churchland, 1995 have attempted to reduce the problem to an issue of resolving how qualitative experiences correspond to discriminative representational states in the brain, including dispositional properties such as those for expressing emotions and speech. Others have deemed the qualia problem scientifically unanswerable due to the subjective nature of experienced object qualities. However, the past decade has witnessed a renewed philosophical and neuroscientific interest in the relationship between sensory qualities and brain processes (O’Brien & Opie, 1999; Clark, 2000; O’Regan and Noë, 2001, Tononi, 2004). Here I take the position that qualia, when classically viewed as distinct from other sensory attributes such as body motion or depth, do present a problematic concept: classical qualitative properties such as color and smell do not stand apart from other sensory attributes, but rather form a continuum across the whole spectrum of the senses, ranging from auditory source localization to the taste of bitterness.
At the same time, however, there do not seem to be sound arguments against taking the very existence of multiple modalities as a commonly acknowledged fact of experience, and hence at least this fact demands a scientific explanation, or at least a theoretical framework in which this explanation needs to be cast. Accounts of mind-brain relationships that fail to explain why sensations in one modality are experienced as phenomenally different from those in another modality must be considered incomplete. In this paper I will first address the Modality Identification (MI) problem, which can be stated as the question: how does the brain come to identify a particular stream of information as belonging to a distinct modality in the course of experience, both during early development and after brain maturation? Thus, what is the neural basis of experiencing one type of input to the central nervous system as ‘visual’ and another type as ‘auditory’? Not only is it interesting to pursue the neural basis of MI for explaining our phenomenally differentiated experiences; the problem is also important from an ecological and evolutionary viewpoint, since it is beneficial for organisms to identify the sources of different sensory inputs as these are conveyed upon the nervous system by different physical media and transduction mechanisms. When viewing the current problem against the background of consciousness research, it appears intimately linked to the study of ‘phenomenal’ consciousness vis-à-vis ‘access’ consciousness. Briefly, these two terms have been introduced to distinguish processes related to generating the phenomenal content of our subjective experiences (of e.g. ‘red’ and ‘green’), whereas access consciousness concerns all cognitive operations operating upon and using this content, such as for verbal expression, perceptual interpretation, guidance of attention, working memory or motor decisions (cf. Baars, 2002, Block, 1990, Block, 2001, Block, 2005, Dehaene and Changeux, 2004, Dehaene et al., 1998; but see Dehaene & Naccache, 2001 for a less absolute distinction).
I will begin with a brief historical review of the MI problem and some scant empirical and theoretical evidence shedding light on it (Section 2). The classical framework for explaining the identity of sensory modalities – the ‘Labelled-Lines Hypothesis’ – will be critically reappraised, leading to the conclusion that classical neurophysiology and neural-network theory do not offer satisfactory solutions to the problem. Next I will propose a heuristic framework by which solutions to the MI problem can be pursued more concretely. This framework rests on a core hypothesis positing that a solution to the MI problem should not be sought at the level of single feature-detecting cells or unimodal sensory areas (Section 3), but at the higher aggregate level of correlations and comparisons between multimodal inputs (Section 4). A further step in elaborating neurocomputational approaches to the MI problem is to distinguish the developmental genesis of sensory modalities during early postnatal life, and the nature of neural representations of multimodal experiences in the mature brain. At this point, the relevance of unimodal and higher-order sensory systems – linked to phenomenal consciousness – will be weighed against systems for translating sensations into verbal and behavioral responses, referred to as interpreter systems and linked to access consciousness. In the second part of this article I will first address the problem how inputs from different modalities can be integrated into coherent perceptual experiences and propose to approach this question by considering a multimodal group of sensory systems as forming a dynamic associative network with globally coexisting attractor states (Section 5). A key question in this area is which neural-coding principles may be available not only to represent the detection of a sensory feature per se, but also to encode its relationships to other features. Here, the distinction between coding by firing-rate and firing phase may prove useful. In Section 6, the proposed framework will be compared with other theories of consciousness, in particular those concerning the neural basis of perceptual qualities. Finally, in Section 7 we will examine how the two hypotheses may contribute to understanding the neural basis of phenomenal consciousness, and how they may help closing the explanatory gap (Levine, 1983) between qualitative experience and brain mechanisms for multimodal perception.
Section snippets
Historical background: the labelled-lines hypothesis and neural network theory
Following Locke, it would take more than a century before sufficient empirical evidence had been gathered to formulate the initial ideas constituting the ‘labelled-lines’ hypothesis (Boring, 1950). In line with Young’s trichromatic theory of 1802, Charles Bell (1811, as cited in Bell, 1869) and Müller (1838) posited that it is the nature of the distinct sensory receptors, and the nervous pathways that relay their activation patterns to the brain, which signal the modality any sensation belongs
Can a sensory modality be identified by single neurons or unimodal maps of neurons?
Here we will briefly review why information processing by single sensory neurons may or may not contribute to elucidate the MI problem. A single-neuron (or ’cardinal-cell’) doctrine has the advantage of simplicity: the coding or representation of an object or object feature is mediated by the firing response of a single cell (or of few cells) specifically to that object or property, but not or much less so to others. Thus, a Feature Detector (FD) is defined as a neuron selectively responsive to
Comparative operations in multimodal areas and topological relationships between modalities
The above review of single, unconnected FDs and unimodal FD maps exposed their incapacity to solve the MI problem. The problem is related to, but different from the question how the brain constructs integrated neural representations of complex objects or events, often involving multiple modalities and submodalities – a question that in turn is linked to the binding problem (cf. Milner, 1974, von der Malsburg, 1981, von der Malsburg, 1995, von der Malsburg, 1999). Here I argue that comparison
Constructing integrated experiences – topographic representations and dynamic associative networks
Before discussing mechanisms of integration, we will first recall how spatial locations of stimuli or events are thought to be represented within and across brain areas, ignoring sequences of events and their representations in time for the sake of brevity. Spatial localization sets critical boundary conditions for integrating and separating object features. Many central sensory systems are marked by some type of topographic mapping, the most well known of which is the retinotopic mapping in
Comparisons to other theories
In this section a concise overview will be provided to compare previously proposed theories with the currently advocated hypotheses. Special attention will be paid to theoretical frameworks attempting to explain both the neural basis of perceptual qualities and multisensory integration in relation to consciousness.
Tononi (2004); see also Tononi & Edelman, 1998) proposed an information-theoretical account of consciousness, which states that the quantity of consciousness can be measured as the
Summary and afterthoughts on phenomenal consciousness
The proposed framework may have several implications for understanding consciousness. While aspects related to intentionality and access consciousness have not been dealt with specifically, we can ask to what extent the two proposed hypotheses may help to close the ‘explanatory gap’ in other aspects, in particular: (i) why do we perceive conscious experiences as being ‘rich’ and unitary (cf. Churchland, 1995, Llinas, 2001, Searle, 2000, Tononi and Edelman, 1998); why do we have integrated
Acknowledgments
I would like to thank Fernando Lopes da Silva and Pieter Goltstein for valuable discussions.
References (196)
- et al.
Figuring space by time
Neuron
(2001) - et al.
Representation of the visual field in striate and adjoining cortex of the owl monkey (Aotus trivirgatus)
Brain Research
(1971) The conscious access hypothesis: Origins and recent evidence
Trends in Cognitive Sciences
(2002)- et al.
Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey
Experimental Neurology
(1977) Paradox and cross purposes in recent work on consciousness
Cognition
(2001)Two neural correlates of consciousness
Trends in Cognitive Sciences
(2005)- et al.
Polymodal motion processing in posterior parietal and premotor cortex: A human fMRI study strongly implies equivalencies between humans and monkeys
Neuron
(2001) Visuospatial processing and the right-hemisphere interpreter
Brain and Cognition
(2003)- et al.
Seeing the big picture: Integration of image cues in the primate visual system
Neuron
(1999) - et al.
Reliability and state dependence of pyramidal cell-interneuron synapses in the hippocampus: An ensemble approach in the behaving rat
Neuron
(1998)
Towards a cognitive neuroscience of consciousness: Basic evidence and a workspace framework
Cognition
Segmentation, attention and phenomenal visual objects
Cognition
Is there a neural code?
Neuroscience and Biobehavioral Reviews
New concepts in generation of movement
Neuron
Is neocortex essentially multisensory?
Trends in Cognitive Sciences
Separate visual pathways for perception and action
Trends in Neurosciences
The temporal correlation hypothesis of visual feature integration: Still alive and well
Neuron
Functional implications of the anatomical organization of the callosal projections of visual areas V1 and V2 in the macaque monkey
Behavioral and Brain Sciences
A fast dynamic link matching algorithm for invariant pattern recognition
Neural Networks
A split microdrive for simultaneous multi-electrode recordings from two brain areas in awake small animals
Journal of Neuroscience Methods
The neural correlates of perceiving one’s own movements
Neuroimage
The basis of sensation
Visuo-haptic object-related activation in the ventral visual pathway
Nature Neuroscience
Shifter circuits: A computational strategy for dynamic aspects of visual processing
Proceedings of the National Academy of Sciences of the United States of America
Neural basis of embodiment: Distinct contributions of temporoparietal junction and extrastriate body area
Journal of Neuroscience
Reference frames for representing visual and tactile locations in parietal cortex
Nature Neuroscience
Persistent activity in neural networks with dynamic synapses
PLoS Computational Biology
Single units and sensation: A neuron doctrine for perceptual psychology?
Perception
Consciousness, introspection and the split brain: The two minds/one body problem
The neuronal representation of pitch in primate auditory cortex
Nature
Sound-induced illusory flash perception: Role of gamma band responses
Neuroreport
Consciousness and accessibility
Behavioral and Brain Sciences
A history of experimental psychology
Encoding of three-dimensional structure-from-motion by primate area MT neurons
Nature
Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque
Journal of Neurophysiology
Crossmodal processing in the human brain: Insights from functional neuroimaging studies
Cerebral Cortex
High gamma power is phase-locked to theta oscillations in human neocortex
Science
Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey
European Journal of Neuroscience
Facing up to the problem of consciousness
Journal of Consciousness Studies
The conscious mind
The engine of reason, the seat of the soul
The computational brain
A theory of sentience
Modulation of LIP activity by predictive auditory and visual cues
Cerebral Cortex
Reference frames for spatial cognition: Different brain areas are involved in viewer-, object-, and landmark-centered judgments about object location
Journal of Cognitive Neuroscience
Magnetically induced phosphenes in sighted, blind and blindsighted observers
Neuroreport
A neural model of figure-ground organization
Journal of Neurophysiology
The astonishing hypothesis
A framework for consciousness
Nature Neuroscience
Cited by (30)
What is neurorepresentationalism? From neural activity and predictive processing to multi-level representations and consciousness
2022, Behavioural Brain ResearchCitation Excerpt :Evidence for such cross-modal connections is mounting [21,100,109–113]. Identifying ‘the’ neural correlate of consciousness (NCC) under this conceptualization is arguably a difficult but also simplified idea, because the conscious percept of e.g. visual motion will be co-dependent on representational activity in many connected cortical areas, the neurons of which should at least be active at baseline level (for a more detailed account involving firing rate- versus firing phase in relational coding, see [1,25]). However, it may be reasonably assumed that the postulated multimodal topology is actively coded in corticothalamic systems, not by more basal, subcortical systems involved in maintaining basic vital functions such as breathing, blood pressure and osmotic control (mediated by e.g. hypothalamic and brainstem nuclei) or by the cerebellum or other structures where inflicted damage does not notably hamper consciousness (e.g. prefrontal cortex [1,114]).
Consciousness and the rabbit holes of delirium
2020, Medical HypothesesCitation Excerpt :The apotheosis of this phenomenon is the qualia which represents the ‘sensory quality of the conscious experience’ [72]. It has been proposed that the multisensory map is highly integrated and closely coupled to provide a distinct ‘corporality’ that rightly can be declared part of consciousness content [73-75]. This runs counter to the Wurzburg controversy that hypothesizes that imageless thought also forms part of the conscious experience [76,77].
The circuit architecture of cortical multisensory processing: Distinct functions jointly operating within a common anatomical network
2019, Progress in NeurobiologyCitation Excerpt :For instance, MP includes also sensory selection, a form of attention in which processing of one sensory modality is prioritized at the expense of others (Lakatos et al., 2008; Schroeder and Lakatos, 2009; Wimmer et al., 2015; Ahrens et al., 2015), but also processes such as object recognition impinging on multiple sensory modalities. Moreover, MP includes not only integration of cross-modal stimuli, but also segregation, for instance to account for the qualitatively varied nature of conscious experience (2009; Pennartz, 2015) - Box 3. Thus, not all multisensory processes can be modeled in terms of simple integration of external sensory features.
Consciousness, Representation, Action: The Importance of Being Goal-Directed
2018, Trends in Cognitive Sciences