Elsevier

Brain Research

Volume 1329, 6 May 2010, Pages 124-141
Brain Research

Research Report
Object-sensitive activity reflects earlier perceptual and later cognitive processing of visual objects between 95 and 500 ms

https://doi.org/10.1016/j.brainres.2010.01.062Get rights and content

Abstract

Object-sensitive areas have been defined using functional magnetic resonance imaging (fMRI), but the time course of this cortical activation is largely unknown. In a high-density, event-related brain potential (ERP) version of a prior fMRI study of object-sensitive areas, people categorized intact grayscale pictures of known objects and discriminated them from scrambled versions created by randomizing the phases of the spatial frequency spectrum; no object or parts can be discerned in scrambled versions. Both indirect functional and direct anatomical approaches were taken to integrate ERP and fMRI results. The two-state interactive account of visual object cognition predicts object-sensitivity (intact vs. scrambled) both before and after 200 ms during early and later ERPs that index processes in posterior cortex critical to visual object perception and cognition. As predicted, findings from 57 participants revealed early object-sensitive activation during the P100 (95–140 ms) and VPP/N170 (145–175 ms), reflecting figure-ground segregation. After 200 ms, activity was object-sensitive during the frontocentral N3 complex (200–500 ms) implicated in cognitive decisions about visual objects, as well as a right occipitotemporal P200 (200–300 ms) implicated in perceptual grouping. All effects localized to extrastriate occipitotemporal cortex. Altogether, the findings demonstrate the time course of object-sensitive activity, which is in cortical regions defined using fMRI, and indicate that processes of figure-ground segregation (95–175 ms or longer), perceptual grouping (200–300 ms), and object model selection for a cognitive decision (200–530 ms) are achieved more for intact known objects than uncategorizable, phase scrambled versions.

Introduction

Theories of visual object perception, cognition, and memory posit a specific sequence of object processing (Farah, 1990, Humphreys et al., 1997, Kosslyn et al., 1994, Murray et al., 2007, Robertson, 2003, Treisman, 2006, Warrington, 1982); note, an ‘object’ can be a three-dimensional (3D) structure in the real world or a two-dimensional (2D) figure or shape distinct from its background. Functional magnetic resonance imaging (fMRI) studies of these abilities focus on regions of interest in extrastriate posterior cortex defined as ‘object-sensitive’ because they respond more to an object than a scrambled version of it (Grill-Spector and Malach, 2004). The gold standard method to define object-sensitive areas compares intact pictures of objects, usually known categories (e.g., dog), with versions of them created by randomizing the phases of the spatial frequencies, yielding texture-like images in which no object figure can be discerned. This object-sensitive activation is widely assumed to reflect predominantly initial, feedforward activation along the visual hierarchy from posterior to anterior areas (Grill-Spector and Malach, 2004), whereas a two-state interactive theory posits that such activation also reflects later recurrent and feedback processing (Schendan and Kutas, 2007a). The temporal smoothing of fMRI limits timing conclusions so, to define the brain dynamics of object-sensitive cortical activity, this study combines event-related brain potentials (ERPs), which have millisecond (ms) precision, with the gold standard method to define object-sensitive areas that was used in a prior fMRI study (Schendan and Stern, 2007).

This project uses the gold standard method to define object-sensitive cortical activity so that the intact and scrambled stimuli differ physically only in the phases of the spatial frequency spectrum (i.e., they do not differ in terms of the spatial frequency power spectrum, brightness, contrast, orientation, etc.). This is important because the location information carried by the phases specifies object shape in terms of the spatial locations of features (Bennett and Banks, 1987, Rousselet et al., 2007), and this feature location information is crucial for categorizing objects (Cave and Kosslyn, 1993). However, other high-order perceptual analyses computed using phase information can also differ between intact and scrambled images, such as: i) figure-ground segregation, as an object figure is obvious in intact images but cannot be discerned in scrambled images (Peterson et al., 2000), and ii) perceptual grouping, as local shapes can be grouped into a more global object in intact more than scrambled images (Koffka, 1935). Further, higher-order object cognition processes also differ, such as object individuation and category decisions (Schendan and Stern, 2008, Treisman, 2006).

The fMRI version (Schendan and Stern, 2007) successfully activated the 3 known object-sensitive areas (Grill-Spector et al., 2000, Hasson et al., 2003). (1) The occipitoparietal area includes the transverse occipital (TOS; areas V3AB) and ventrocaudal intraparietal sulci (vcIPS) implicated in analyzing the spatial configuration of parts, mental rotation, and object individuation (Schendan and Stern, 2007, Schendan and Stern, 2008, Xu, 2009). (2) The dorsal occipitotemporal (DOT) area implicated in category learning and decisions includes lateral occipital (LOS) and inferotemporal (ITS) sulci (Jiang et al., 2007, Op de Beeck et al., 2006, Schendan and Stern, 2008). (3) The ventral occipitotemporal (VOT) area implicated in abstract (conceptual) object knowledge (Martin, 2007) includes posterior fusiform gyrus (pFG) and a collateral sulcus region (CoS).

To infer the human brain dynamics, both indirect and direct approaches integrate ERP timing and fMRI location information (Luck, 1999). The direct approach requires localizing both results to a common spatial coordinate system, which here is the Montreal Neurological Institute (MNI) brain atlas. The indirect approach relates time and space findings based on similar functional patterns between them. Accordingly, ERPs were acquired using the activation paradigm from an fMRI version so that object-sensitive ERPs can be inferred to reflect activity in object-sensitive areas, as defined using fMRI. Additional functional evidence from other ERP and fMRI studies supplements this functional mapping and has been integrated in the two-state interactive account of the brain dynamics for visual object cognition. This account posits that object-sensitive areas are activated in two functionally-distinct states (i.e., processing times) (Schendan and Kutas, 2007a, Schendan and Stern, 2008, Schendan and Maher, 2009). Representations of visual objects and/or their parts are activated in each state, but the representations can differ in type and in their contributions to cognition.

In the first ‘initial classification’ state before 200 ms, initial feedforward activation of object-sensitive areas supports feature detection, structural encoding, and perceptual categorization, such as detecting an animal target in a scene (Bruce and Young, 1986, Damasio et al., 1982, Perrett et al., 1987, Thorpe et al., 1996), and can include automatic feedback loops along the visual hierarchy triggered by initial bottom-up activity (Rennie et al., 2002). Most visual object ERP studies focused on the first 200 ms, and mainly on category-specificity. Findings revealed an occipitotemporal N170 and its polarity-inverted, centrofrontal ‘vertex positive peak’ (VPP) counterpart from 120 to 170 ms that are larger to faces and letterstrings than other object categories (i.e., mainly cars, hands, or houses) and localize to object-sensitive DOT and VOT areas, or adjacent areas for special categories (e.g., faces, body parts, and buildings) (e.g., Allison et al., 1994, Allison et al., 1999, Bentin et al., 1996, Bötzel and Grüsser, 1989, Halgren et al., 1995, Horovitz et al., 2004, Itier and Taylor, 2002, Itier and Taylor, 2004a, Itier and Taylor, 2004b, Jeffreys, 1996, McCarthy et al., 1999, Miki et al., 2004, Onitsuka et al., 2006, Puce et al., 1999, Rossion et al., 1999, Rossion et al., 2000, Rossion et al., 2003, Rousselet et al., 2005, Schendan et al., 1998, Schweinberger et al., 2002, Thierry et al., 2006, Watanabe et al., 2003). Notably, VPP/N170-like ERPs recorded from human occipitotemporal cortex are larger to intact known objects (e.g., cars, butterflies) than to phase scrambled versions (created using the same software as herein) and seem to reflect initial, bottom-up activity (Allison et al., 1999, Allison et al., 2002). However, an earlier P100 may be the earliest ERP to index face-specificity (Thierry et al., 2007). This is consistent with evidence that the early phase of the P100 indexes activity in the TOS part of the occipitoparietal area and the middle occipital gyrus (MOG) of the DOT area, and the late phase of the P100 indexes area V4v in a far posterior part of fusiform gyrus, though the latter area is less object-sensitive and more sensitive to physical differences than farther along the ventral pathway in DOT and VOT areas (Avidan et al., 2002, Denys et al., 2004, Di Russo et al., 2001, Grill-Spector et al., 2000, Lerner et al., 2001, Schendan and Stern, 2007). The VPP/N170 and perhaps also the P100 thus index state 1, consistent with intracranial ERPs recorded from human occipitotemporal cortex showing category-selective P100- and VPP/N170-like waves (Liu et al., 2009).

In the second state after 200 ms, more sustained feedback, recurrent, and bottom-up interactions among object-sensitive areas and other brain structures support object model selection, selecting the best match to the percept from memory to make a cognitive decision (Schendan and Kutas, 2007a). This second state is indexed by a frontocentral N3 complex from 200 to 500 ms or longer that is the first ERP modulated with visual object cognition factors and shows similar effects as object-sensitive areas in fMRI studies (e.g., Henson et al., 2004, Horner and Henson, 2008, Schendan and Kutas, 2002, Schendan and Kutas, 2003, Schendan and Stern, 2007, Schendan and Stern, 2008, Schendan and Lucia, 2009, Schendan and Maher, 2009). Crucially, both the VPP/N170 in state 1 and the N3 in state 2 invert polarity between frontal and occipitotemporal sites and localize to occipitotemporal sources (Allison et al., 1999, Ganis and Schendan, 2008, Philiastides and Sajda, 2007, Schendan and Maher, 2009, Sehatpour et al., 2006). Further, occipitotemporal cortex, where the intracranial N200 is recorded, also shows later P290 and N700 waves that may reflect re-entrant input to the same cells (Allison et al., 1999, Allison et al., 2002, Puce et al., 1999), consistent with an interactive state 2 during the N3.

This study differs in important ways from prior ERP studies of both object cognition or perception of objects vs. scrambled versions. (a) The gold standard method of scrambling the phases of the spatial frequencies to create scrambled versions of intact objects was used, which has the advantage of equating low-level physical properties between stimulus types, and such phase scrambling has not been used in prior ERP studies that assess the success of generic object categorization performance (for review see Schendan and Maher, 2009). (b) Here, the same stimuli and durations were included in an fMRI version (Schendan and Stern, 2007), and both indirect and direct approaches integrate the ERP and fMRI information (Luck, 1999) to enable more valid inferences about the brain dynamics and validate the assumption that human posterior object-sensitive areas are activated in this ERP study. (c) Stimuli in the fMRI and this ERP study did not include the special categories (i.e., faces, buildings, body parts) that were the focus of prior studies and have neural substrates and processes differing from other categories (Downing et al., 2006, Epstein et al., 2006, Hasson et al., 2003, Spiridon et al., 2006, Thierry et al., 2006). (d) Prior ERP (and magnetoencephalography) comparisons between objects vs. scrambled versions and most object cognition studies focused on initial bottom-up processing before 200 or occasionally the first 300 ms, whereas this study assesses the entire time course until response. The full timing of object-sensitivity with diverse, generic nonface categories that commonly occupy the visual environment has thus remained largely unknown but was defined herein.

The two-state interactive account predicts that both category-specific ERPs during state 1 before ∼ 200 ms and ERPs modulated with object cognition success during state 2 after 200 ms will be object-sensitive. Specifically, the VPP/N170 and to a lesser extent perhaps also the P100 will be larger for intact than scrambled images, and the frontocentral N3 complex will be more negative for scrambled versions, which are uncategorizable, than intact objects, which will be categorized. Further, all these effects will localize to their corresponding areas of occipitotemporal cortex. Such findings would provide key support for the two-state interactive account, demonstrating that object-sensitivity – a defining property of extrastriate posterior cortex – occurs both during early and later processing of visual objects. By contrast, feedforward models of visual object cognition (Serre et al., 2007) predict object-sensitivity only during initial bottom-up activation, as later processes are undefined, and fMRI studies typically assume that object-sensitivity and object cognition reflect predominantly feedforward hierarchical processing (Grill-Spector and Malach, 2004). Object-sensitivity on only the P100 and/or VPP/N170 would support these feedforward hypotheses, and refute the two-state interactive account.

For completeness, earlier and later ERPs were also assessed. At the earliest time, the C1 (and concurrent anterior N100) is the first ERP evoked to visual stimulation and localizes to parvocellular processes in area V1/V2 (Di Russo et al., 2001, Foxe et al., 2008) that would not show object-sensitivity (e.g., Brewer et al., 2005, Grill-Spector et al., 1998, Grill-Spector and Malach, 2004, Larsson and Heeger, 2006). Later, from 500 to 900 ms, a centroparietal late positive complex (LPC) is more positive for evaluating category decisions as successful than not so, predicting more positivity for known objects, which will be categorized, than scrambled versions, which will not be (Schendan and Maher, 2009).

Section snippets

Performance

Accuracy was nearly perfect for Intact and Scrambled (both 97.9%; Object, F[1,55] = 0.03, p = 0.86). Mean response times (RTs) on correct trials excluded RTs over 2.5 SD from the mean per subject and were slightly faster to Scrambled (589 ms, SD = 162) than Intact (615 ms, SD = 149; Object, F[1,55] = 13.77, p = 0.0005).

ERPs

ERPs were analyzed on: (a) correct trials, (b) trials eliciting a response, and (c) all trials, which yielded similar results because participants were nearly perfectly accurate, so results

Discussion

People are just as accurate but ∼ 30 ms slower to decide that intact images depict known objects as opposed to nonsense in phase scrambled versions in which an object figure cannot be discerned. Together, the ERP and corresponding fMRI findings (Schendan and Stern, 2007) demonstrate that, from the early P100 onwards (up to 900 ms), human cortical processing differs between intact objects and phase scrambled versions in fMRI-defined, object-sensitive areas. Posterior effects (P100, N170, and P3

Conclusions

Indirect and direct integration of parallel ERP and fMRI (Schendan and Stern, 2007) findings support the two-state interactive idea that object-sensitive cortex is activated initially bottom-up before 200 ms and then after 200 ms during a second state of interactive activity between bottom-up, recurrent, and feedback inputs to accomplish higher-order cognitive functions, like activating visual object knowledge for a category decision. Findings indicate that, while people make simple category

Design, materials, and procedure

The stimulus conditions were (a) Intact grayscale photographs of known objects and (b) phase Scrambled versions of them (Fig. 6A). The Intact condition had 48 stimuli chosen pseudo-randomly from the fMRI ones (Schendan and Stern, 2007) to keep the category proportions identical to the fMRI version, resulting in 8 fruit or vegetables, 8 animals, 8 furniture, 8 vehicles, and 16 tools (including 4 musical instruments). Since the goodness of a view may affect results (Schendan and Kutas, 2003), 5

Acknowledgments

This study was conducted in the Vision & Memory Neuroimaging Lab in the Department of Psychology at Tufts University. Research was supported by Tufts University Faculty start-up funds to HES who designed and set-up the experiment, directed the plan for analyses, ran the sLORETA analyses, and wrote the manuscript. L.C.L. collected and analyzed the data and assisted with the manuscript preparation. The authors are grateful to Emily A. Leung and Jessica Bernard for assisting with some data

References (125)

  • K. Grill-Spector et al.

    Differential processing of objects under various viewing conditions in the human lateral occipital complex

    Neuron

    (1999)
  • T. Gruber et al.

    Oscillatory brain activity in the human EEG during indirect and direct memory tasks

    Brain Res.

    (2006)
  • T. Gruber et al.

    Brain electrical tomography (BET) analysis of induced gamma band responses during a simple object recognition task

    Neuroimage

    (2006)
  • E. Halgren et al.

    Intracerebral potentials to rare target and distractor auditory and visual stimuli. I. Superior temporal plane and parietal lobe

    Electroencephalogr. Clin. Neurophysiol.

    (1995)
  • E. Halgren et al.

    Cortical activation to illusory shapes as measured with magnetoencephalography

    Neuroimage

    (2003)
  • U. Hasson et al.

    Large-scale mirror-symmetry organization of human occipito-temporal object areas

    Neuron

    (2003)
  • R.N. Henson et al.

    The effect of repetition lag on electrophysiological and haemodynamic correlates of visual object priming

    Neuroimage

    (2004)
  • P.J. Holcomb et al.

    Event-related brain potentials reflect semantic priming in an object decision task

    Brain Cogn.

    (1994)
  • A.J. Horner et al.

    Priming, response learning and repetition suppression

    Neuropsychologia

    (2008)
  • S.G. Horovitz et al.

    Parametric design and correlational analyses help integrating fMRI and electrophysiological data during face processing

    Neuroimage

    (2004)
  • R.J. Itier et al.

    Inversion and contrast polarity reversal affect both encoding and recognition processes of unfamiliar faces: a repetition study using ERPs

    Neuroimage

    (2002)
  • X. Jiang et al.

    Categorization training results in shape- and category-selective human neural plasticity

    Neuron

    (2007)
  • W. Khoe et al.

    Interactions between attention and perceptual grouping in human visual cortex

    Brain Res.

    (2006)
  • V.A. Lamme et al.

    The role of primary visual cortex (V1) in visual awareness

    Vision Res.

    (2000)
  • H. Liu et al.

    Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex

    Neuron

    (2009)
  • K. Miki et al.

    Magnetoencephalographic study of occipitotemporal activity elicited by viewing mouth movements

    Clin. Neurophysiol.

    (2004)
  • R. Oldfield

    The assessment and analysis of handedness: the Edinburgh inventory

    Neuropsychologia

    (1971)
  • R.D. Pascual-Marqui et al.

    Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain

    Int. J. Psychophysiol.

    (1994)
  • D. Perrett et al.

    Visual neurones responsive to faces

    Trends Neurosci.

    (1987)
  • M.A. Peterson et al.

    Object memory effects on figure assignment: conscious object recognition is not necessary or sufficient

    Vis. Res.

    (2000)
  • B. Rossion et al.

    Task modulation of brain activity related to familiar and unfamiliar face processing: an ERP study

    Clin. Neurophysiol.

    (1999)
  • B. Rossion et al.

    Early lateralization and orientation tuning for face, word, and object processing in the visual cortex

    Neuroimage

    (2003)
  • G.A. Rousselet et al.

    Single-trial EEG dynamics of object and face visual processing

    Neuroimage

    (2007)
  • H.E. Schendan et al.

    Neurophysiological evidence for two processing times for visual object identification

    Neuropsychologia

    (2002)
  • T. Allison et al.

    Human extrastriate visual cortex and the perception of faces, words, numbers, and colors

    Cereb. Cortex

    (1994)
  • T. Allison et al.

    Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli

    Cereb. Cortex

    (1999)
  • T. Allison et al.

    Category-sensitive excitatory and inhibitory processes in human extrastriate cortex

    J. Neurophysiol.

    (2002)
  • G. Avidan et al.

    Contrast sensitivity in human visual areas and its relationship to object recognition

    J. Neurophysiol.

    (2002)
  • P.J. Bennett et al.

    Sensitivity loss in odd-symmetric mechanisms and phase anomalies in peripheral vision

    Nature

    (1987)
  • S. Bentin et al.

    Electrophysiological studies of face perception in humans

    J. Cogn. Neurosci.

    (1996)
  • K. Bötzel et al.

    Electric brain potentials evoked by pictures of faces and non-faces — a search for face-specific EEG-potentials

    Exp. Brain Res.

    (1989)
  • A.A. Brewer et al.

    Visual field maps and stimulus selectivity in human ventral occipital cortex

    Nat. Neurosci.

    (2005)
  • V. Bruce et al.

    Understanding face recognition

    Br. J. Psychol.

    (1986)
  • C.B. Cave et al.

    The role of parts and spatial relations in object identification

    Perception

    (1993)
  • K.R. Daffner et al.

    An electrophysiological index of stimulus unfamiliarity

    Psychophysiology

    (2000)
  • A.R. Damasio et al.

    Prosopagnosia: anatomic basis and behavioral mechanisms

    Neurology

    (1982)
  • K. Denys et al.

    The processing of visual shape in the cerebral cortex of human and nonhuman primates: a functional magnetic resonance imaging study

    J. Neurosci.

    (2004)
  • F. Di Russo et al.

    Cortical sources of the early components of the visual evoked potential

    Hum. Brain Mapp.

    (2001)
  • F. Di Russo et al.

    Source analysis of event-related cortical activity during visuo-spatial attention

    Cereb. Cortex

    (2003)
  • G.M. Doniger et al.

    Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes

    J. Cogn. Neurosci.

    (2000)
  • Cited by (44)

    • Probing the neural signature of mind wandering with simultaneous fMRI-EEG and pupillometry

      2021, NeuroImage
      Citation Excerpt :

      Furthermore, we were interested in differences in amplitudes of event-related EEG signals across midline occipital (MidOcc), occipitotemporal (OccTem), midline parietal (MidPar), and midline frontal (MidFro) channel clusters, roughly corresponding to the scalp distributions of P1, N1, P300, and associated frontal ERPs, respectively (Supplementary Figure A.1B). Where the posterior P1 and N1 are believed to signal early perceptual processes in the visual domain, the later P300 component is thought to index working memory and related cognitive processes (Shendan and Lucia, 2010). We used an offset of 8 ms to correct for the delay from the anti-aliasing filter of the Net Amps 300 amplifier.

    • Memory influences visual cognition across multiple functional states of interactive cortical dynamics

      2019, Psychology of Learning and Motivation - Advances in Research and Theory
      Citation Excerpt :

      Human occipitoparietal cortex along the dorsal visual pathway and occipitotemporal cortex along the ventral visual pathway have been shown to be object-sensitive using standard methods in fMRI, for example, comparing images of real objects and completely phase scrambled versions of them (Schendan & Stern, 2007). ERPs using this fMRI paradigm reveals that the P1(00) (in a large group of people, N = 57) first begins to differ between the real objects and phase scrambled versions (i.e., shows object-sensitivity) around 95 ms, reflecting processes to achieve figure-ground segregation (Schendan & Lucia, 2010). This object-sensitive neurophysiological activity localizes especially to the right hemisphere in extrastriate occipital cortex in Brodmanns area (BA 18) during the early P1 (95–115 ms) and inferior occipital gyrus in BA 18 during the late P1 (120–140 ms) and lasts until around 155 ms. Furthermore, these areas show reactivation after 200 ms, consistent with the MUSI account of multiple states of activation of the same region.

    View all citing articles on Scopus
    View full text