Finding decodable information that can be read out in behaviour
Introduction
Multivariate pattern analysis (MVPA), also called brain decoding, is a powerful tool to establish statistical dependencies between experimental conditions and brain activation patterns (Carlson et al., 2003; Cox and Savoy, 2003; Haxby et al., 2001; Haynes, 2015; Kamitani and Tong, 2005; Kriegeskorte et al., 2006). In these analyses, an implicit assumption often made by experimenters is that if information can be decoded, then this information is used by the brain in behaviour (de-Wit et al., 2016; Ritchie et al., 2017). However, the decoded information could be different (e.g., epiphenomenal) from the signal that is relevant for the brain to use in behaviour (de-Wit et al., 2016; Williams et al., 2007), highlighting the need to relate decoded information to behaviour. Importantly, this implicit assumption of decoding models leads to testable predictions about task performance (Naselaris et al., 2011). Previous work has for example correlated decoding performances to behavioural accuracies (Bouton et al., 2018; Freud et al., 2017; Raizada et al., 2010; van Bergen et al., 2015; Walther et al., 2009; Williams et al., 2007). However, this does not model how individual experimental conditions relate to behaviour. Another approach has been to compare neural and behavioural similarity structures (Bracci and Op de Beeck, 2016; Cichy et al., 2017; Cohen et al., 2016; Grootswagers et al., 2017a; Haushofer et al., 2008; Mur et al., 2013; Proklova et al., 2016; Wardle et al., 2016). While this approach allows to link behaviour and brain patterns at the level of single experimental conditions, it is unclear how this link carries over to decision making behaviour such as categorisation (but see Cichy et al. (2017) for recent developments).
Recently, a novel methodological approach, called the distance-to-bound approach (Ritchie and Carlson, 2016), has been proposed to connect brain activity directly to perceptual decision-making behaviour at the level of individual experimental conditions. The rationale behind this approach (Bouton et al., 2018; Carlson et al., 2014; Kiani et al., 2014; Philiastides and Sajda, 2006; Ritchie and Carlson, 2016) is that for decision-making tasks, the brain applies a decision boundary to a neural activation space (DiCarlo and Cox, 2007). Similarly, MVPA classifiers fit multi-dimensional hyperplanes to separate a neural activation space. In classic signal-detection theory (Green and Swets, 1966) and evidence-accumulation models of choice behaviour (Brown and Heathcote, 2008; Gold and Shadlen, 2007; Ratcliff and Rouder, 1998; Smith and Ratcliff, 2004), the distance of the input to a decision boundary reflects the ambiguity of the evidence for the decision (Green and Swets, 1966). Decision evidence, in turn, predicts choice behaviour (e.g., Ashby, 2000; Ashby and Maddox, 1994; Britten et al., 1996; Gold and Shadlen, 2007; Shadlen and Kiani, 2013) which also has clear neural correlates (e.g., Britten et al., 1996; Ratcliff et al., 2009; Roitman and Shadlen, 2002). If for a decision task (e.g., categorisation), the brain uses the same information as the MVPA classifier, then the classifier's hyperplane reflects the brain's decision boundary. This in turn predicts that distance to the classifier's hyperplane negatively correlates with reaction times for the decision task. In the distance-to-bound approach, finding such a negative distance-RT-correlation shows that the information is then suitably formatted to guide behaviour. “Suitably formatted to guide behaviour” here means that the information is structured in such a way that the brain can apply a linear read out process to this representation to make a decision (importantly, this does not imply a causal link with behaviour). Carlson et al. (2014) demonstrated the promise of the distance-to-bound approach in a region of interest based analysis using fMRI. Here we go beyond this work by using the distance-to-bound method and a spatially unbiased fMRI-searchlight approach to create maps of where in the brain information can be used to guide behaviour.
Section snippets
Materials and methods
In this study, we separately localised information that is decodable, and information that is suitably formatted to guide behaviour in the context of decodable information about visual objects and object categorisation behaviour. To ensure robustness and generality of our results, we analysed in parallel two independent fMRI datasets (Cichy et al., 2014, 2016), with different stimulus sets, and in relation to partly overlapping categorisation behaviours. Overall, this allowed us to investigate
Results
We examined the relationship between decodable information and information that is suitably formatted for read-out by the brain in the context of decodable information about visual objects and object categorisation behaviour. We determined the relationship between decodable information and behaviour separately. First, we determined where information about objects is present in brain patterns using decoding in a standard fMRI searchlight decoding analysis (Haynes et al., 2007; Kriegeskorte
Dissociating between decodable information and information that is used in behaviour
The aim of this study was to examine where in the brain decodable information is suitably formatted for read-out by the brain in behaviour. We found that only a subset of information that is decodable could be related to behaviour using the distance-to-bound approach, which argues for a partial dissociation between decodable information and information that is relevant for behaviour. This speaks to a current challenge in neuroimaging, which is to show that information visible to the
Conclusion
In this study, we combined the distance-to-bound approach (Ritchie and Carlson, 2016) with a searchlight decoding analysis to find brain areas with decodable information that is suitable for read-out in behaviour. Our results showed that decodable information is not always equally suitable for read-out by the brain in behaviour. This speaks to the current debate in neuroimaging research about whether the information that we can decode is the same information that is used by the brain in
Acknowledgements
This research was supported by an Australian Research Council Future Fellowship (FT120100816) and an Australian Research Council Discovery project (DP160101300) awarded to T.A.C., and an Emmy Noether grant by the German Research Foundation grant (CI-241/1-1) awarded to R.M.C. The authors acknowledge the University of Sydney HPC service for providing High Performance Computing resources. The authors declare no competing financial interests.
References (94)
A stochastic version of general recognition theory
J. Math. Psychol.
(2000)- et al.
A response time theory of separability and integrality in speeded classification
J. Math. Psychol.
(1994) - et al.
The simplest complete model of choice response time: linear ballistic accumulation
Cognit. Psychol.
(2008) - et al.
Task alters category representations in prefrontal but not high-level visual cortex
Neuroimage
(2017) - et al.
What is the bandwidth of perceptual experience?
Trends Cognit. Sci.
(2016) - et al.
Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex
Neuroimage
(2003) - et al.
Untangling invariant object recognition
Trends Cognit. Sci.
(2007) - et al.
How does the brain solve visual object recognition?
Neuron
(2012) - et al.
Body selectivity in occipitotemporal cortex: causal evidence
Neuropsychologia
(2016) A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives
Neuron
(2015)
Reading hidden intentions in the human brain
Curr. Biol.
Typicality sharpens category representations in object-selective cortex
Neuroimage
Dynamics of neural population responses in prefrontal cortex indicate changes of mind on single trials
Curr. Biol.
Ultra-rapid object detection with saccadic eye movements: visual processing speed revisited
Vis. Res.
Neuroscience needs behavior: correcting a reductionist bias
Neuron
Matching categorical object representations in inferior temporal cortex of man and monkey
Neuron
Encoding and decoding in fMRI
Neuroimage
Machine learning classifiers and fMRI: a tutorial overview
Neuroimage
Natural categories
Cognit. Psychol.
Family resemblances: studies in the internal structure of categories
Cognit. Psychol.
Decision making as a window on cognition
Neuron
Topographic maps in human frontal and parietal cortex
Trends Cognit. Sci.
Psychology and neurobiology of simple decisions
Trends Neurosci.
Perceptual similarity of visual patterns predicts dynamic neural activation patterns measured with MEG
Neuroimage
Coping with confounds in multivoxel pattern analysis: what should we do about reaction time differences? A comment on Todd, Nystrom & Cohen 2013
Neuroimage
Microstimulation of inferotemporal cortex influences face categorization
Nature
Controlling the false discovery rate: a practical and powerful approach to multiple testing
J. Roy. Stat. Soc. B
Focal versus distributed temporal cortex activity for speech sound category assignment
Proc. Natl. Acad. Sci. Unit. States Am.
Task context overrules object- and category-related representational content in the human parietal cortex
Cerebr. Cortex
Dissociations and associations between shape and category representations in the two visual pathways
J. Neurosci.
A relationship between behavioral choice and the visual responses of neurons in macaque MT
Vis. Neurosci.
Domain-specific knowledge systems in the brain: the animate-inanimate distinction
J. Cognit. Neurosci.
Reaction time for object categorization is predicted by representational distance
J. Cognit. Neurosci.
Patterns of activity in the categorical representations of objects
J. Cognit. Neurosci.
Representational dynamics of object vision: the first 1000 ms
J. Vis.
Neural dynamics of real-world object vision that guide behaviour
BioRxiv
Resolving human object recognition in space and time
Nat. Neurosci.
Similarity-based fusion of MEG and fMRI reveals spatio-temporal dynamics in human cortex during visual object recognition
Cerebr. Cortex
Probing principles of large-scale object representation: category preference and location encoding
Hum. Brain Mapp.
Fast saccades toward faces: face detection in just 100 ms
J. Vis.
Low-level cues and ultra-fast face detection
Front. Psychol.
Is neuroimaging measuring information in the brain?
Psychonomic Bull. Rev.
Domain specificity in visual cortex
Cerebr. Cortex
A cortical area selective for visual processing of the human body
Science
An adaptive coding model of neural function in prefrontal cortex
Nat. Rev. Neurosci.
Experience-dependent representation of visual categories in parietal cortex
Nature
Neuronal mechanisms of visual categorization: an abstract view on decision making
Annu. Rev. Neurosci.
Cited by (38)
Are you for real? Decoding realistic AI-generated faces from neural activity
2022, Vision ResearchCitation Excerpt :We observed evidence for a positive brain-behaviour correlation around 170 ms for the upright and inverted unrealistic faces, which is consistent with time points of above-chance decoding (Fig. 4A). This result suggests that, at least for the unrealistic faces, the signal that is used by the classifier for real/fake distinction could be used by the brain to make the real/fake decision (Grootswagers et al., 2018; Ritchie et al., 2019). There is growing concern that realism is advancing at such a rate that humans will have difficulty discerning between what is real and what is fake (Fletcher, 2018; Khodabakhsh, Ramachandra, & Busch, 2019; Nightingale et al., 2017; Shen et al., 2019).
Reaction times predict dynamic brain representations measured with MEG for only some object categorisation tasks
2021, NeuropsychologiaCitation Excerpt :They found a negative correlation between representational distance from the object classifier boundary and behavioural RTs: participants had faster RTs for objects that were further from the decision boundary, with slower RTs for less distinctly represented objects that were closer to the classifier boundary. Previous studies have shown a relationship between categorisation RT and distance to boundary for animacy using fMRI or MEG (Carlson et al., 2014; Grootswagers et al., 2017a; 2018; Ritchie et al., 2015; Ritchie and de Beeck, 2019). The categorical distinction of animate versus inanimate objects has been proposed as one of the fundamental organisational principles in the neural representation of complex objects (Kriegeskorte et al., 2008b).
The speed of human social interaction perception
2020, NeuroImage