Chapter 11 - Multisensory object representation: Insights from studies of vision and touch
Introduction
Object recognition research has typically focused on the visual modality but, following pioneering work by Klatzky, Lederman, and their colleagues (e.g., Klatzky and Lederman, 1995, Klatzky et al., 1985, Klatzky et al., 1987, Lederman and Klatzky, 1987), there is now a substantial literature on haptic object recognition. The representations underlying visual and haptic within-modal object recognition are strikingly similar: each, for example, is sensitive to changes in orientation, size, and surface properties. However, these similarities should not be taken to mean that vision and haptics necessarily access only a single, shared representation: the basis for similar representational characteristics differs between modalities. It is a first theme of this chapter that, though functionally similar, within-modal visual and haptic recognition are supported at a basic level by separate, unisensory representations. Vision and haptics do, however, converge on a shared representation in the service of higher-order recognition. We will review studies indicating that a single, shared representation supports within-modal, view-independent recognition in both vision and touch, and also visuo-haptic, cross-modal recognition. A shared representation suggests a shared neural substrate between vision and touch; this and the implications for the nature of the underlying representation constitute a second theme of this chapter. The final part of the chapter links these earlier themes by outlining a model of multisensory object recognition in which visuo-haptic access to multisensory representations is modulated by object familiarity and individual differences on the object–spatial dimension of mental imagery.
Section snippets
View-dependence
A major challenge for object recognition is achieving perceptual constancy, which insulates it from potentially disruptive transformations in the sensory input caused by changes in orientation, size, etc. Visual object recognition is view-dependent under certain circumstances, since rotating an object away from its original orientation impairs subsequent recognition (see Peissig and Tarr, 2007, for a review). Although the hands can explore an object from different sides simultaneously and
Size-dependence
In addition to achieving object constancy across orientation changes, the visual system also has to recognize the physical size of an object across variations in the size of the retinal image that arise from changing object–observer distance: the same object can produce retinal images that vary in size depending on whether it is near to or far from the observer. Presumably, this is compensated by cues arising from depth or motion perception, accounting for the fact that a change in size does
Integration of structural and surface properties
Although visual shape, color, and texture are processed in different cortical areas (Cant et al., 2009, Cant and Goodale, 2007), behavioral evidence suggests that visual object representations integrate structural and surface properties because changing the color of an object or its part-color combinations between study and test resulted in longer response times for a shape recognition task (Nicholson and Humphrey, 2003). Since altering the background color against which objects were presented
Multisensory cortical processing
Many studies have shown that visual cortical areas are functionally involved during haptic tasks (reviewed in Amedi et al., 2005, Sathian and Lacey, 2007). The lateral occipital complex (LOC) in the ventral visual pathway responds selectively to objects (Malach et al., 1995) and a subregion responds selectively to objects in both vision and touch (Amedi et al., 2001, Amedi et al., 2002, Stilla and Sathian, 2008). Tactile responsiveness in the LOC has been found for both 3D (Amedi et al., 2001,
What role does visual imagery play?
Haptic activation of visual cortex might arise either because haptic exploration of an object evokes visual imagery (Sathian et al., 1997), presumably processed in the LOC, or because the LOC can be directly activated by somatosensory input. A recent electrophysiological study of tactile discrimination of simple geometric shapes applied to the fingerpad shows that activity propagates from somatosensory cortex to LOC as early as 150 ms after stimulus onset, a time frame which is consistent with
A model of multisensory object representation
In this section, we draw together some of the threads reviewed earlier by outlining, and reviewing the evidence for, a preliminary conceptual model of visuo-haptic multisensory object representation that we detailed previously (Lacey et al., 2009b). In this model, the LOC contains a representation of object form that can be flexibly accessed either bottom-up or top-down, independently of the input modality, but with the choice of the bottom-up versus top-down route depending on object
Summary
In this chapter, we have shown that visual and haptic within-modal object recognition initially rely on separate representations that are nonetheless functionally similar in being view- and size-dependent. Further work is required to investigate the different mechanisms by which these similarities arise in each modality. These unisensory representations feed forward into a multisensory, view-independent representation that supports cross-modal object recognition. Here, further work is needed to
Acknowledgments
Support to K. S. from the National Eye Institute at the NIH, the National Science Foundation, and the Veterans Administration is gratefully acknowledged.
References (86)
- et al.
Direct tactile stimulation of dorsal occipito-temporal cortex in a visual agnosic
Current Biology
(2009) - et al.
Occipital activation by pattern recognition in the early blind using auditory substitution for vision
Neuroimage
(2001) The proactive brain: Using analogies and associations to generate predictions
Trends in Cognitive Sciences
(2007)- et al.
The role of familiarity in the recognition of static and dynamic objects
Progress in Brain Research
(2006) - et al.
Auditory triggered mental imagery of shape involves visual association areas in early blind humans
Neuroimage
(2001) - et al.
Object familiarity modulates effective connectivity during haptic shape perception
Neuroimage
(2010) - et al.
Effective connectivity during haptic perception: A study using Granger causality analysis of functional magnetic resonance imaging data
Neuroimage
(2008) - et al.
fMR-adaptation reveals a distributed representation of inanimate objects and places in human visual cortex
Neuroimage
(2005) - et al.
BOLD activity during mental rotation and view-dependent object recognition
Neuron
(2002) - et al.
Separate visual pathways for perception and action
Trends in Neuroscience
(1992)