Trends in Cognitive Sciences
Sensory substitution and the human–machine interface
Section snippets
Brain plasticity
Brain plasticity can be defined as ‘the adaptive capacities of the central nervous system – its ability to modify its own structural organization and functioning’ [17]. It permits an adaptive (or a maladaptive) response to functional demand. Mechanisms of brain plasticity include neurochemical, synaptic, receptor, and neuronal structural changes 18, 19, 20, 21.
Plastic changes in functional representation (usually occurring in response to a combination of a need and training) do not appear to
Sensory substitution
Sensory substitution can occur across sensory systems, such as touch-to-sight, or within a sensory system such as touch-to-touch. In one experiment, the touch sensory information via a glove containing artificial contact sensors was coupled to skin sensory receptors on the forehead of a person who had lost peripheral sensation from leprosy. After becoming accustomed to the device, the patient experienced the data generated in the glove as if they were originating in the fingertips, ignoring the
Auditory-vision sensory substitution: seeing via the ears
Substitutive sensory devices for blindness rehabilitation through audition have been studied by DeVolder et al. [24]. Spatial localization and object recognition has been demonstrated by Capelle et al. [25]. The system couples a rough model of the human retina with an inverse model of the cochlea, using a pixel-frequency relationship. A head-mounted TV camera allows on-line translation of visual patterns into sounds. With head or joystick movement, visual frames are grabbed at high-frequency
Tactile-vision substitution: seeing via skin receptors
Tactile-vision sensory substitution (TVSS) studies have been carried out by numerous research groups for over a century. Many electro- and vibrotactile sensory substitution HMIs have been developed which have been applied to various surface areas 26, 27, 28, 29, 30, 31. In particular, the tongue provides a practical HMI [32]. It is very sensitive and highly mobile. Because it is in the protected environment of the mouth, the sensory receptors are close to the surface. The presence of an
Tactile-vestibular sensory substitution: balancing via skin receptors
Persons with bilateral vestibular damage (BVD) experience functional difficulties that include postural ‘wobbling’, unstable gait and oscillopsia. This condition presents the unique opportunity to: (i) study a model of an open-loop human control system, and (ii) to re-establish head-postural control by means of vestibular substitution using a head-mounted accelerometer and an electrotactile HMI through the tongue sensory receptors. The use of vestibular sensory substitution produces a strong
Implanted human–machine technologies
Human–machine interfaces have provided the major challenges to practical sensory substitution. Although this review has emphasized tactile sensory substitution, other techniques are being studied. Mussa-Ivaldi and Miller have reviewed several HMIs including EEG and intra-cortical techniques [36]. They emphasize that feedback is needed for learning and control, requiring the establishment of a ‘closed loop’. Comparably, Suaning and Lovell have developed a 100-channel implanted neurostimulation
Neural correlates of sensory substitution
In addition to the subjective reports and observed performance of the subjects experiencing sensory substitution, the phenomenon has been investigated with functional neuroimaging. PET studies of sensory substitution mainly rely on cerebral perfusion mapping to reflect brain activity because changes in synaptic activity in the brain result in changes of local perfusion. Several PET studies demonstrated that in cognitive tasks, such as Braille reading and tactile discrimination tasks 40, 41,
Conclusions
Sensory substitution studies have demonstrated the capacity of the brain to adapt to information relayed from an artificial receptor via an auditory or tactile HMI. With training and with motor control of the input by the subject, percepts are accurately identified and spatially located. Thus, blind persons obtain visual information resulting in visual percepts (e.g. of a ball rolling across a table) and can produce appropriate motor responses (e.g. catching the ball) with a vision substitution
Acknowledgements
This research was partially supported in part by the National Science Foundation (Grant no. IIS-0083347). In addition to their university positions, the authors are associated with Wicab, Inc, a company to which the University of Wisconsin has licensed the patented technology for a tongue HMI, which was developed in Bach-y-Rita's university laboratory under an NIH, NEI RO1 grant. The authors thank Dr Yuri Danilov for his assistance in preparing this article.
References (52)
Brain plasticity: “visual” acuity of blind persons via the tongue
Brain Res.
(2001)- et al.
The self-composing brain: towards a glial–neuronal brain theory
Brain Cogn.
(2003) Auditory triggered mental imagery of shape involves visual association areas in early blind humans
Neuroimage
(2001)- et al.
Transmission of pictorial information through the skin
Adv. Biol. Med. Phys.
(1973) - et al.
Brain–machine interfaces: computational demands and clinical needs meet basic neuroscience
Trends Neurosci.
(2003) Occipital activation by pattern recognition in the early blind using auditory substitution of vision
Neuroimage
(2001)Changes in occipital cortex activity in early blind humans using a sensory substitution device
Brain Res.
(1999)Brain Mechanisms in Sensory Substitution
(1972)Nonsynaptic Diffusion Neurotransmission and Late Brain Reorganization
(1995)Seeing with the skin
Percept. Psychophys.
(1970)
Perception and Cognition
Molyneux's Question
A sensorimotor account of vision and visual consciousness
Behav. Brain Sci.
Vision substitution by tactile image projection
Nature
Binaural sonar electronic travel aid provides vibrotactile cues for landmark, reflector motion and surface texture classification
IEEE Trans. Biomed. Eng.
Basic eperimental research on electrotactile physiology for deaf auditory substitution
Acta Acad. Med. Shandong
Application software for electrotactile substitution
Shandong J. Biomed. Eng.
An experimental system for auditory image representations
IEEE Trans. Biomed. Eng.
Representation of locomotor space by the blind
Percept. Psychophys.
Auditory coding of visual patterns for the blind
Perception
Neural plasticity and consciousness
Biol. Philos.
Bizarre hierarchy of brain function
Softer than soft computing
Cited by (618)
Conversion of a medical implant into a versatile computer-brain interface
2024, Brain StimulationSensory substitution can improve decision-making
2023, Computers in Human BehaviorDigital Knowledge: A Philosophical Investigation
2024, Digital Knowledge: A Philosophical InvestigationThe Role of Haptics in Training and Games for Hearing-Impaired Individuals: A Systematic Review
2024, Multimodal Technologies and Interaction