Identification and integration of sensory modalities: Neural basis and relation to consciousness

Abstract

A key question in studying consciousness is how neural operations in the brain can identify streams of sensory input as belonging to distinct modalities, which contributes to the representation of qualitatively different experiences. The basis for identification of modalities is proposed to be constituted by self-organized comparative operations across a network of unimodal and multimodal sensory areas. However, such network interactions alone cannot answer the question how sensory feature detectors collectively account for an integrated, yet phenomenally differentiated experiential content. This problem turns out to be different from, although related to, the binding problem. It is proposed that the neural correlate of an enriched, multimodal experience is constituted by the attractor state of a dynamic associative network. Within this network, unimodal and multimodal sensory maps continuously interact to influence each other’s attractor state, so that a feature change in one modality results in a fast re-coding of feature information in another modality. In this scheme, feature detection is coded by firing-rate, whereas firing phase codes relational aspects.

Introduction

Over the past decades, neuroscience has made considerable progress in elucidating neural substrates underlying mental processes such as memory, attention, discrimination and evaluation of sensory inputs, and voluntary control over behavior. These processes have proven to be tractable and amenable to investigation with the methodologies of contemporary neuroscience. However, the enterprise of elucidating their neural basis hinges on the implicit agreement to refrain from probing into deeper questions related to the phenomenal content represented by neural activity. For instance, attention is commonly viewed as the selection of a body of information for intensive processing at the expense of other objects deemed less relevant; it can be modelled by neural circuits performing filtering, synchronizing, noise-suppressing and related cybernetic operations. The widespread use of the term ‘information’ in the sense of Shannon’s definition (1948) is revealing in this context, since it can be conveniently applied to both neural signals (e.g. spike trains) and mental phenomena, without exerting force to explain the relationship between these two domains. But eventually cognitive neuroscience needs to address the question of how neural activity that represents the object of attention correlates to the phenomenal content we are conscious of. The current review attempts to make a few steps in addressing this question by considering, first, how the brain may come to identify different streams of information as belonging to distinct sensory modalities. The second part of this review will be concerned with neural mechanisms to integrate these streams in producing a global, multimodal experience.

To prevent confusion about what an account of the problem presented here should or should not hold, it is mandatory to contrast two different questions. First, in a dialog on the experience of a colored object one may ask another person: “how do you see the color red?”, or: “do you see the color red in the same way I do?” This inquiry immediately runs into the difficulties in investigating other people’s private experiences (e.g. Wittgenstein, 1953), which is an area of philosophical controversy not touched upon here. Second, however, one can reasonably ask the question, based on an experience shared and reported by most people: “if I see an object with its three-dimensional shape, size, pattern of light reflections and surface texture, how is it that I also perceive this other property of the object we call ‘color’?” Similarly: “How is it that we can see a color in the first place, as distinct from other object properties?” This second question addresses how we come to experience modalities and submodalities as being phenomenally different. The definition of ‘modality’ used here relies on the commonly used distinction between the main senses (vision, touch, hearing, smell, taste, pain, thermoception, proprioception and the vestibular senses). Within a main modality one can usually distinguish submodalities (for vision: 3D-shape, movement, color, size, surface texture, etc.).

Neural substrates of consciousness have been studied almost exclusively in the visual domain, where a considerable amount of neural-substrate and neural-correlate research has been done. However, this approach bears the risk of focusing too narrowly on a single, unimodal system, without addressing commonalities and differences between vision and other modalities. Once it has been decided to focus exclusively on visual perception, the question of why there is a visual modality in the first place, as distinct from other modalities, does not have to be addressed. One can freely apply the term ‘information’ to both the tuning properties of visual cortex neurons and their perceptual counterparts without having to confront the question what makes this ‘visual’ information different from ‘auditory’ information. Shannon’s information theory (1948) offers no solution in this respect because it is concerned with the statistical relationships between sets of inputs (stimuli) and outputs (responses), not with the content or the semantics of the information under scrutiny (cf. Eggermont, 1998, Rieke et al., 1997). The existence of different sensory modalities seems so natural and basic to us that the question of how the brain produces distinct experiences of smell, taste, vision, sound and touch hardly ever comes up.

Although the subject of this paper is not to review recent philosophical debates and controversies on qualitative aspects of consciousness (or: “qualia”), it is worthwhile to briefly touch upon some relevant theoretical issues setting the background stage for the current, primarily neuroscientific discourse. For philosophical reviews on this topic, the reader is referred to Nagel, 1974, Jackson, 1986, Dennett, 1991, Searle, 1992, Searle, 2000, Chalmers, 1995, Chalmers, 1996, Churchland, 1995, Tye, 2000, Shear, 2000. For theories more strongly driven by experimental and computational neuroscience, see Churchland and Sejnowski, 1992, Crick, 1994, Tononi and Edelman, 1998, O’Brien and Opie, 1999, O’Regan and Noë, 2001, Zeki, 2001, Dehaene and Naccache, 2001, Rees et al., 2002, Taylor, 2003, Crick and Koch, 2003, Koch, 2004, Block, 2005.

This paper adopts a specific viewpoint on the general problem of qualitative experience by asking which kind of neural operations may lie at the basis of the existence of multiple sensory modalities. This question is more neuroscience-oriented yet wider in scope than the classical ‘qualia’ problem, which by tradition focuses on the ‘secondary qualities’ of objects such as color and smell, but much less on properties that can be unambiguously quantified (e.g. motion, shape and extension) and were therefore considered primary by Locke (1667). The qualia problem has not been unanimously considered a serious object of study, partly due to difficulties in maintaining a fundamental distinction between primary and secondary qualities (Boring, 1950). More recently, functionalist or physicalist accounts such as by Dennett, 1991, Churchland, 1995 have attempted to reduce the problem to an issue of resolving how qualitative experiences correspond to discriminative representational states in the brain, including dispositional properties such as those for expressing emotions and speech. Others have deemed the qualia problem scientifically unanswerable due to the subjective nature of experienced object qualities. However, the past decade has witnessed a renewed philosophical and neuroscientific interest in the relationship between sensory qualities and brain processes (O’Brien & Opie, 1999; Clark, 2000; O’Regan and Noë, 2001, Tononi, 2004). Here I take the position that qualia, when classically viewed as distinct from other sensory attributes such as body motion or depth, do present a problematic concept: classical qualitative properties such as color and smell do not stand apart from other sensory attributes, but rather form a continuum across the whole spectrum of the senses, ranging from auditory source localization to the taste of bitterness.

At the same time, however, there do not seem to be sound arguments against taking the very existence of multiple modalities as a commonly acknowledged fact of experience, and hence at least this fact demands a scientific explanation, or at least a theoretical framework in which this explanation needs to be cast. Accounts of mind-brain relationships that fail to explain why sensations in one modality are experienced as phenomenally different from those in another modality must be considered incomplete. In this paper I will first address the Modality Identification (MI) problem, which can be stated as the question: how does the brain come to identify a particular stream of information as belonging to a distinct modality in the course of experience, both during early development and after brain maturation? Thus, what is the neural basis of experiencing one type of input to the central nervous system as ‘visual’ and another type as ‘auditory’? Not only is it interesting to pursue the neural basis of MI for explaining our phenomenally differentiated experiences; the problem is also important from an ecological and evolutionary viewpoint, since it is beneficial for organisms to identify the sources of different sensory inputs as these are conveyed upon the nervous system by different physical media and transduction mechanisms. When viewing the current problem against the background of consciousness research, it appears intimately linked to the study of ‘phenomenal’ consciousness vis-à-vis ‘access’ consciousness. Briefly, these two terms have been introduced to distinguish processes related to generating the phenomenal content of our subjective experiences (of e.g. ‘red’ and ‘green’), whereas access consciousness concerns all cognitive operations operating upon and using this content, such as for verbal expression, perceptual interpretation, guidance of attention, working memory or motor decisions (cf. Baars, 2002, Block, 1990, Block, 2001, Block, 2005, Dehaene and Changeux, 2004, Dehaene et al., 1998; but see Dehaene & Naccache, 2001 for a less absolute distinction).

I will begin with a brief historical review of the MI problem and some scant empirical and theoretical evidence shedding light on it (Section 2). The classical framework for explaining the identity of sensory modalities – the ‘Labelled-Lines Hypothesis’ – will be critically reappraised, leading to the conclusion that classical neurophysiology and neural-network theory do not offer satisfactory solutions to the problem. Next I will propose a heuristic framework by which solutions to the MI problem can be pursued more concretely. This framework rests on a core hypothesis positing that a solution to the MI problem should not be sought at the level of single feature-detecting cells or unimodal sensory areas (Section 3), but at the higher aggregate level of correlations and comparisons between multimodal inputs (Section 4). A further step in elaborating neurocomputational approaches to the MI problem is to distinguish the developmental genesis of sensory modalities during early postnatal life, and the nature of neural representations of multimodal experiences in the mature brain. At this point, the relevance of unimodal and higher-order sensory systems – linked to phenomenal consciousness – will be weighed against systems for translating sensations into verbal and behavioral responses, referred to as interpreter systems and linked to access consciousness. In the second part of this article I will first address the problem how inputs from different modalities can be integrated into coherent perceptual experiences and propose to approach this question by considering a multimodal group of sensory systems as forming a dynamic associative network with globally coexisting attractor states (Section 5). A key question in this area is which neural-coding principles may be available not only to represent the detection of a sensory feature per se, but also to encode its relationships to other features. Here, the distinction between coding by firing-rate and firing phase may prove useful. In Section 6, the proposed framework will be compared with other theories of consciousness, in particular those concerning the neural basis of perceptual qualities. Finally, in Section 7 we will examine how the two hypotheses may contribute to understanding the neural basis of phenomenal consciousness, and how they may help closing the explanatory gap (Levine, 1983) between qualitative experience and brain mechanisms for multimodal perception.