To what extent are a combination of properties that facilitate a single goal (e.g., edges for scene recognition), generalizable to other tasks
Trends in Cognitive Sciences
ReviewMaking Sense of Real-World Scenes
Section snippets
Interacting with Real-World Scenes
Making a cup of tea is an easy task that requires minimal concentration, yet the composition of behaviors involved is deceptively complex: recognizing the room next door as a kitchen, navigating to it while maneuvering around obstacles, locating and handling objects (e.g., teabag, kettle), and manipulating those objects until they are in a correct state (e.g., filling the kettle). In addition, it requires knowledge of relative locations and future destinations within the environment (e.g., take
Toward a Comprehensive Framework for Scene Understanding
There are two major challenges to producing a comprehensive theoretical framework that would outline the cognitive and neural mechanisms of scene understanding. The first is that, while the physical characteristics of our surrounding environment are generally stable, our immediate goals are not. At any given moment, different visual aspects of an environment will be prioritized based on our current goal (Figure 1A, Key Figure; Box 2). However, the dynamic nature of scene understanding is often
Mapping Properties to Goals
Based on the discussion of these four main goals of scene understanding, it should be clear that the goals themselves are not mutually exclusive. For example, recognition facilitates search and navigation processes; navigation sometimes requires searching for specific information (e.g., objects, boundaries); scene affordance must consider navigability within a space, and so forth. Similarly, informative properties overlap various goals: spatial layout facilitates the early stages of recognition
The Neural Mechanisms of Scene Understanding
In general, visual scene processing in humans has been characterized by a trio of scene-selective regions: the occipital place area (OPA), parahippocampal place area (PPA), and retrosplenial complex (RSC), on the lateral occipital, ventral temporal, and medial parietal cortical surfaces, respectively [82]. Studies in non-human primates have also reported scene-selective regions 83, 84, 85, as well as regions responsive to spatial landscapes [86]. Much of the research on humans has focused on
Concluding Remarks and Future Directions
Scene understanding entails representing information about the properties and arrangement of the world to facilitate the ongoing needs of the viewer. By focusing on four major goals of scene understanding – recognizing the environment, searching for information within the environment, moving through the environment, and determining what actions can be performed – we have demonstrated how different goals use similar properties and, conversely, how many properties can be used for different goals.
Acknowledgments
G.L.M., I.I.A.G., and C.I.B. are supported by the intramural research program of NIMH (ZIAMH002909). I.I.A.G. is supported by a Rubicon Fellowship from The Netherlands Organization of Scientific Research (NWO). We thank Wilma Bainbridge, Michelle Greene, Assaf Harel, Antje Nuthmann, and Edward Silson for commenting on earlier versions of this manuscript.
Glossary
- Convolutional neural network
- a computer vision model with a multi-layer hierarchical architecture that can be trained to perform classification of visual images.
- Diagnosticity
- the relative usefulness of a specific subset of perceptual information in facilitating an observer's goal. For example, an oven is highly diagnostic in helping to categorize a scene as a kitchen, while an apple is less so.
- Environmental space
- a physical space that is too large to be appreciated without locomotion, requiring
References (147)
- et al.
Categories of environmental scenes
Cogn. Psychol.
(1983) - et al.
Recognition of natural scenes from global properties: seeing the forest without representing the trees
Cogn. Psychol.
(2009) - et al.
Diagnostic colors mediate scene recognition
Cogn. Psychol.
(2000) Processing scene context: fast categorization and object interference
Vision Res.
(2007)The long and the short of it: spatial statistics at fixation vary with saccade amplitude and task
Vision Res.
(2006)Semantic guidance of eye movements in real-world scenes
Vision Res.
(2011)The roles of scene gist and spatial dependency among objects in the semantic guidance of attention in real-world scenes
Vision Res.
(2014)- et al.
The interplay of episodic and semantic memory in guiding repeated search in scenes
Cognition
(2013) - et al.
Modeling the influence of task on attention
Vision Res.
(2005) Gist in time: scene semantics and structure enhance recall of searched objects
Acta Psychol.
(2016)
Modeling task control of eye movements
Curr. Biol.
The where, what and when of gaze allocation in the lab and the natural environment
Vision Res.
Gaze fixation patterns for negotiating complex ground terrain
Neuroscience
The occipital place area is causally involved in representing environmental boundaries during navigation
Curr. Biol.
Interaction envelope: local spatial representations of objects at all scales in scene-selective regions
Neuroimage
The role of context in object recognition
Trends Cogn. Sci.
Conflict in the kitchen: contextual modulation of responsiveness to affordances
Conscious. Cogn.
A network for scene processing in the macaque temporal lobe
Neuron
A channel for 3D environmental shape in anterior inferotemporal cortex
Neuron
Patterns of response to visual scenes are linked to the low-level properties of the image
Neuroimage
Patterns of neural response in scene-selective regions of the human brain are affected by low-level manipulations of spatial frequency
Neuroimage
Spatial frequency processing in scene-selective cortical regions
Neuroimage
Contour junctions underlie neural representations of scene categories in high-level human visual cortex
Neuroimage
A real-world size organization of object responses in occipitotemporal cortex
Neuron
Natural scene statistics account for the representation of scene categories in human visual cortex
Neuron
Statistics of natural image categories
Netw. Comput. Neural Syst.
Beyond gist: strategic and incremental information accumulation for scene categorization
Psychol. Sci.
What you see is what you expect: rapid scene understanding benefits from prior experience
Atten. Percept. Psychophys.
Four common conceptual fallacies in mapping the time course of recognition
Front. Psychol.
Scene perception
Detecting meaning in RSVP at 13 ms per picture
Atten. Percept. Psychophys.
Failure to detect meaning in RSVP at 27 ms per picture
Attention, Perception, Psychophys.
Modeling the shape of the scene: a holistic representation of the spatial envelope
Int. J. Comput. Vis.
Brain responses strongly correlate with Weibull image statistics when processing natural images
J. Vis.
The briefest of glances: the time course of natural scene understanding
Psychol. Sci.
From image statistics to scene gist: evoked neural activity reveals transition from low-level natural image structure to scene category
J. Neurosci.
Nonaccidental properties underlie human categorization of complex natural scenes
Psychol. Sci.
Diagnostic colours contribute to the early stages of scene categorization: behavioural and neurophysiological evidence
Vis. Cogn.
The influence of color on the perception of scene gist
J. Exp. Psychol. Hum. Percept. Perform.
Scene consistency in object and background perception
Psychol. Sci.
Do simultaneously viewed objects influence scene recognition individually or as groups? Two perceptual studies
PLoS One
Visual noise from natural scene statistics reveals human scene category representations
Rapid apprehension of the coherence of action scenes
Psychon. Bull. Rev.
A perceptual paradigm and psychophysical evidence for hierarchy in scene gist processing
J. Vis.
What do we perceive in a glance of a real-world scene?
J. Vis.
Masking of foveal and parafoveal vision during eye fixations in reading
J. Exp. Psychol. Hum. Percept. Perform.
Looking at domestic textiles: an eye-tracking experiment analysing influences on viewing behaviour at Owlpen Manor
Text. Hist.
How do the regions of the visual field contribute to object search in real-world scenes? Evidence from eye movements
J. Exp. Psychol. Hum. Percept. Perform.
Viewing task influences on eye movements during scene perception
J. Vis.
The prominence of behavioural biases in eye guidance
Vis. Cogn.
Cited by (107)
Two ‘what’ pathways for action and object recognition
2022, Trends in Cognitive Sciences