Chapter 11 - Multisensory object representation: Insights from studies of vision and touch

https://doi.org/10.1016/B978-0-444-53752-2.00006-0Get rights and content

Abstract

Behavioral studies show that the unisensory representations underlying within-modal visual and haptic object recognition are strikingly similar in terms of view- and size-sensitivity, and integration of structural and surface properties. However, the basis for these attributes differs in each modality, indicating that while these representations are functionally similar, they are not identical. Imaging studies reveal bisensory, visuo-haptic object selectivity, notably in the lateral occipital complex and the intraparietal sulcus, that suggests a shared representation of objects. Such a multisensory representation could underlie visuo-haptic cross-modal object recognition. In this chapter, we compare visual and haptic within-modal object recognition and trace a progression from functionally similar but separate unisensory representations to a shared multisensory representation underlying cross-modal object recognition as well as view-independence, regardless of modality. We outline, and provide evidence for, a model of multisensory object recognition in which representations are flexibly accessible via top-down or bottom-up processing, the choice of route being influenced by object familiarity and individual preference along the object–spatial continuum of mental imagery.

Introduction

Object recognition research has typically focused on the visual modality but, following pioneering work by Klatzky, Lederman, and their colleagues (e.g., Klatzky and Lederman, 1995, Klatzky et al., 1985, Klatzky et al., 1987, Lederman and Klatzky, 1987), there is now a substantial literature on haptic object recognition. The representations underlying visual and haptic within-modal object recognition are strikingly similar: each, for example, is sensitive to changes in orientation, size, and surface properties. However, these similarities should not be taken to mean that vision and haptics necessarily access only a single, shared representation: the basis for similar representational characteristics differs between modalities. It is a first theme of this chapter that, though functionally similar, within-modal visual and haptic recognition are supported at a basic level by separate, unisensory representations. Vision and haptics do, however, converge on a shared representation in the service of higher-order recognition. We will review studies indicating that a single, shared representation supports within-modal, view-independent recognition in both vision and touch, and also visuo-haptic, cross-modal recognition. A shared representation suggests a shared neural substrate between vision and touch; this and the implications for the nature of the underlying representation constitute a second theme of this chapter. The final part of the chapter links these earlier themes by outlining a model of multisensory object recognition in which visuo-haptic access to multisensory representations is modulated by object familiarity and individual differences on the object–spatial dimension of mental imagery.

Section snippets

View-dependence

A major challenge for object recognition is achieving perceptual constancy, which insulates it from potentially disruptive transformations in the sensory input caused by changes in orientation, size, etc. Visual object recognition is view-dependent under certain circumstances, since rotating an object away from its original orientation impairs subsequent recognition (see Peissig and Tarr, 2007, for a review). Although the hands can explore an object from different sides simultaneously and

Size-dependence

In addition to achieving object constancy across orientation changes, the visual system also has to recognize the physical size of an object across variations in the size of the retinal image that arise from changing object–observer distance: the same object can produce retinal images that vary in size depending on whether it is near to or far from the observer. Presumably, this is compensated by cues arising from depth or motion perception, accounting for the fact that a change in size does

Integration of structural and surface properties

Although visual shape, color, and texture are processed in different cortical areas (Cant et al., 2009, Cant and Goodale, 2007), behavioral evidence suggests that visual object representations integrate structural and surface properties because changing the color of an object or its part-color combinations between study and test resulted in longer response times for a shape recognition task (Nicholson and Humphrey, 2003). Since altering the background color against which objects were presented

Multisensory cortical processing

Many studies have shown that visual cortical areas are functionally involved during haptic tasks (reviewed in Amedi et al., 2005, Sathian and Lacey, 2007). The lateral occipital complex (LOC) in the ventral visual pathway responds selectively to objects (Malach et al., 1995) and a subregion responds selectively to objects in both vision and touch (Amedi et al., 2001, Amedi et al., 2002, Stilla and Sathian, 2008). Tactile responsiveness in the LOC has been found for both 3D (Amedi et al., 2001,

What role does visual imagery play?

Haptic activation of visual cortex might arise either because haptic exploration of an object evokes visual imagery (Sathian et al., 1997), presumably processed in the LOC, or because the LOC can be directly activated by somatosensory input. A recent electrophysiological study of tactile discrimination of simple geometric shapes applied to the fingerpad shows that activity propagates from somatosensory cortex to LOC as early as 150 ms after stimulus onset, a time frame which is consistent with

A model of multisensory object representation

In this section, we draw together some of the threads reviewed earlier by outlining, and reviewing the evidence for, a preliminary conceptual model of visuo-haptic multisensory object representation that we detailed previously (Lacey et al., 2009b). In this model, the LOC contains a representation of object form that can be flexibly accessed either bottom-up or top-down, independently of the input modality, but with the choice of the bottom-up versus top-down route depending on object

Summary

In this chapter, we have shown that visual and haptic within-modal object recognition initially rely on separate representations that are nonetheless functionally similar in being view- and size-dependent. Further work is required to investigate the different mechanisms by which these similarities arise in each modality. These unisensory representations feed forward into a multisensory, view-independent representation that supports cross-modal object recognition. Here, further work is needed to

Acknowledgments

Support to K. S. from the National Eye Institute at the NIH, the National Science Foundation, and the Veterans Administration is gratefully acknowledged.

References (86)

  • C. Grefkes et al.

    Human somatosensory area 2: Observer-independent cytoarchitectonic mapping, interindividual variability, and population map

    Neuroimage

    (2001)
  • K. Grill-Spector et al.

    Repetition and the brain: Neural models of stimulus-specific effects

    Trends in Cognitive Sciences

    (2006)
  • K. Grill-Spector et al.

    Differential processing of objects under various viewing conditions in the human lateral occipital complex

    Neuron

    (1999)
  • A. Iriki et al.

    Attention-induced neuronal activity in the monkey somatosensory cortex revealed by pupillometrics

    Neuroscience Research

    (1996)
  • T.W. James et al.

    Differential effects of view on object-driven activation in dorsal and ventral streams

    Neuron

    (2002)
  • T.W. James et al.

    Haptic study of three-dimensional objects activates extrastriate visual areas

    Neuropsychologia

    (2002)
  • B. Krekelberg et al.

    Adaptation: From single cells to BOLD signals

    Trends in Neurosciences

    (2006)
  • S. Lacey et al.

    Object familiarity modulates the relationship between visual object imagery and haptic shape perception

    Neuroimage

    (2010)
  • S.J. Lederman et al.

    Hand movements: A window into haptic object recognition

    Cognitive Psychology

    (1987)
  • S.D. Newman et al.

    Imagining material versus geometric properties of objects: An fMRI study

    Cognitive Brain Research

    (2005)
  • A. Pasqualotto et al.

    The role of visual experience on the representation and updating of novel haptic scenes

    Brain and Cognition

    (2007)
  • S. Peltier et al.

    Activity and effective connectivity of parietal and occipital cortical regions during haptic shape perception

    Neuropsychologia

    (2007)
  • S.C. Prather et al.

    Task-specific recruitment of dorsal and ventral visual areas during tactile perception

    Neuropsychologia

    (2004)
  • L. Renier et al.

    Cross modal activation of visual cortex during depth perception using auditory substitution of vision

    Neuroimage

    (2005)
  • N.J. Rice et al.

    Orientation sensitivity to graspable objects: An fMRI adaptation study

    Neuroimage

    (2007)
  • M. Stoesz et al.

    Neural networks active during tactile form perception: Common and differential activity during macrospatial and microspatial tasks

    International Journal of Psychophysiology

    (2003)
  • M.J. Tarr et al.

    Mental rotation and orientation dependence in shape recognition

    Cognitive Psychology

    (1989)
  • K.F. Valyear et al.

    A double dissociation between sensitivity to changes in object identity and object orientation in the ventral and dorsal streams: A human fMRI study

    Neuropsychologia

    (2006)
  • A. Amedi et al.

    Convergence of visual and tactile shape processing in the human lateral occipital complex

    Cerebral Cortex

    (2002)
  • A. Amedi et al.

    Visuo-haptic object-related activation in the ventral visual pathway

    Nature Neuroscience

    (2001)
  • A. Amedi et al.

    Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex

    Nature Neuroscience

    (2007)
  • A. Amedi et al.

    Functional imaging of human crossmodal identification and object recognition

    Experimental Brain Research

    (2005)
  • L.J. Berryman et al.

    Representation of object size in the somatosensory system

    Journal of Neurophysiology

    (2006)
  • I. Biederman et al.

    Size invariance in visual object priming

    Journal of Experimental Psychology: Human Perception and Performance

    (1992)
  • O. Blajenkova et al.

    Object-spatial imagery: A new self-report imagery questionnaire

    Applied Cognitive Psychology

    (2006)
  • J.S. Cant et al.

    fMR-adaptation reveals separate processing regions for the perception of form and texture in the human ventral stream

    Experimental Brain Research

    (2009)
  • J.S. Cant et al.

    Attention to form or surface properties modulates different regions of human occipitotemporal cortex

    Cerebral Cortex

    (2007)
  • E. Combe et al.

    Observer movement and size constancy

    Psychological Science

    (2010)
  • M. Craddock et al.

    Repetition priming and the haptic recognition of familiar and unfamiliar objects

    Perception and Psychophysics

    (2008)
  • M. Craddock et al.

    Do left and right matter for haptic recognition of familiar objects?

    Perception

    (2009)
  • M. Craddock et al.

    The effect of size changes on haptic object recognition

    Attention, Perception and Psychophysics

    (2009)
  • M. Craddock et al.

    Size-sensitive perceptual representations underlie visual and haptic object recognition

    PLoS ONE

    (2009)
  • M. Craddock et al.

    The effects of temporal delay and orientation on haptic object recognition

    Attention, Perception and Psychophysics

    (2010)
  • Cited by (0)

    View full text