Action Organization in Lateral Occipitotemporal Cortex

Being able to recognize what other people are doing is crucial in complex social interactions. How this is achieved quasi-automatically by our brains has been a longstanding issue in cognitive neuroscience. Experiments using fMRI show that perceiving another person's action activates, to a large extent, the same regions as does action performance. This includes frontal areas (the inferior frontal gyrus and premotor cortex) and the posterior parietal cortex, in addition to regions in the lateral occipitotemporal cortex (LOTC) and posterior superior temporal sulcus (Chong et al., 2008; Kilner et al., 2009; Oosterhof et al., 2010). Because frontal and posterior parietal regions store sensorimotor representations while occipitotemporal and superior temporal regions belong to the human visual cortex, these findings led researchers to propose that understanding of actions involves mapping visual inputs onto one's own motor representations. Mirror neurons discovered in premotor cortex and inferior parietal lobule of the macaque that discharge during both observed and performed actions may be implicated in the neural basis of this process (Rizzolatti et al., 2001; Caramazza et al., 2014). Understanding others' actions may alternatively be understood in a predictive coding framework, where the different aspects of the observed action (e.g., intention, goal, kinematics) are processed through backward and forward connections between frontal, parietal, and occipitotemporal areas of the network (Kilner et al., 2007).

Regardless of the neural mechanisms, a common proposal about action representations in the brain is that lower-level aspects of an action (e.g., kinematics) are coded in high-level visual areas, in particular LOTC, whereas representations of higher-level aspects (e.g., goal, intention) are hosted in either inferior frontal/premotor or posterior parietal areas (Kilner, 2011). Recent developments have shown that abstract action concepts might also be encoded in the LOTC (Wurm et al., 2016). However, LOTC organization in relation to action representation remains unclear.

A recent study published in The Journal of Neuroscience contributes significantly to this important issue. Wurm et al. (2017) predicted a segregation of the LOTC along sociality and transitivity dimensions, the dorsal LOTC and ventral LOTC hosting representations of social and transitive actions, respectively. This hypothesis stems from previous findings demonstrating that animate objects and social cues are encoded in the dorsal LOTC, whereas knowledge about tool motion and inanimate objects preferentially activates the ventral LOTC (Allison et al., 2000; Beauchamp et al., 2002; Downing et al., 2006).

To test this hypothesis, Wurm et al. (2017) scanned 28 young adults with fMRI while the participants watched 2 s videos of hand actions varying along the dimensions sociality and transitivity. The factorial design led to four categories of actions: social/transitive (give, take an object to/from someone), social/intransitive (agree or disagree, symbolic gesture), nonsocial/transitive (open, close an object), and nonsocial/intransitive (stroke, scratch one's own arm). To make conclusions of the study as generalizable as possible, each of the two actions per category were filmed from different perspectives and in different contexts, with different actors and objects. The authors also asked participants to rate how much the actions in the videos were either social or transitive, thereby ensuring that social/nonsocial and transitive/intransitive dimensions were categorized correctly. These behavioral data were then used to build “representational dissimilarity matrices” that represented the extent to which videos differed with respect to sociality and transitivity. The brain responses to sociality and transitivity were afterward examined using multivariate analysis of fMRI time series. In essence, these techniques looked for statistical dependency (i.e., mutual information) between the video stimuli and multivoxel regional activity patterns, to evaluate whether social and transitive actions were associated with distinct patterns of regional brain activity, especially within the LOTC.

Multivariate analyses are generally more powerful than conventional univariate “activation-based” techniques to capture information about experimental conditions (e.g., Kriegeskorte and Bandettini, 2007; Jimura and Poldrack, 2012). Therefore, Wurm et al. (2017) combined searchlight-based (Kriegeskorte et al., 2006) and ROI-based multivariate pattern analysis (MVPA) using linear discriminant function and cross-validation along with representational similarity analysis (RSA) (Kriegeskorte et al., 2008). Whereas MVPA tested whether brain response patterns could reliably distinguish different categories of stimuli (i.e., social vs unsocial and transitive vs intransitive), RSA enabled the comparison of activity patterns with behavioral representational dissimilarity matrices for sociality and transitivity (for a review on the two methods, see Haxby et al., 2014). This provided greater sensitivity to decode differences between sociality and transitivity. Univariate contrast maps (action categories vs baseline) served as inputs to multivariate analyses and were used to provide complementary information on action-category organization of the brain.

Multiclass searchlight MVPA using all the actions in the four categories confirmed that action features were encoded in the LOTC, as well as in other brain sites spanning from posterior parietal cortex (intraparietal sulcus; superior parietal lobule) to premotor and prefrontal cortex. Across-category MVPA and RSA using both searchlight and ROI mapping demonstrated that social action features were mostly represented in the dorsal LOTC, whereas transitive action features were mostly encoded in ventral LOTC. Hence, these analyses provided a clear evidence that the LOTC is topographically organized along the dimensions of sociality and transitivity. It has to be noted, however, that the sociality dimension was also represented in the posterior parietal cortex and transitivity features in both posterior parietal and premotor cortices.

Univariate contrasts for sociality and transitivity further revealed significant clusters of activation located in the dorsal end of LOTC (posterior superior temporal sulcus) and ventral end of LOTC (parahippocampal cortex), respectively. This result provides support to the idea that segregation between social and transitive actions within the LOTC emerged from opposing constraints exerted by social- and object-related information. Finally, additional MVPA revealed that activity in regions more posterior to those encoding sociality and transitivity features of actions served to represent specific actions of the same category (i.e., within category decoding). Accordingly, the LOTC is not only organized along a dorsal-ventral gradient from sociality to transitivity but also along a posterior-anterior gradient from concrete to more abstract action features.

The presence of neural representations of actions associated with sociality and transitivity in the LOTC demonstrates that the ventral “perceptual” visual stream (to which the LOTC belongs) is critical to action understanding. This outcome departs from the classical view that transformation of visual inputs into action representations relies only on the dorsal “action” visual stream (Goodale and Milner, 1992; Rizzolatti and Sinigaglia, 2010). Instead, data of Wurm et al. (2017) support the two-stream model of action understanding (Kilner, 2011).

The fact that social and transitive action features could be predicted from activity patterns in posterior parietal and frontal regions in the same manner as in the LOTC is of particular importance. This suggests that all these regions are functionally connected, as previously observed in intrinsic functional connectivity (Peelen et al., 2013). This finding also raises a question about whether patterns of activity in the LOTC elicited by viewing actions were a byproduct of the connectivity with higher-level parietal and frontal regions.

Recent studies on the action observation network support a predictive coding model of the network (Gardner et al., 2015; Maffei et al., 2015), where evoked activity in the LOTC corresponds to a mismatch (i.e., prediction error) between prediction about the incoming stimuli conveyed by backward connections from other areas of the cortical hierarchy and the observed stimuli. To further elucidate how action features are represented in the human brain, it is now necessary to examine effective connectivity by modeling explicitly the coupling between brain areas and how it is influenced by experimental manipulations (e.g., dynamic causal modeling) (Friston et al., 2003). Likewise, analyzing connectivity would directly assess whether connectivity-based constraints subtend action representation in the LOTC. In particular, is social-related information in posterior superior temporal sulcus more strongly connected to social action representation in dorsal LOTC, and is object-related information in parahippocampal cortex more strongly connected to transitive actions in ventral LOTC?

In conclusion, Wurm et al. (2017) revealed a topological organization of the LOTC along a dorsoventral axis for sociality and transitivity features, and along a posterior-anterior axis for concrete to abstract action representations. Furthermore, their study suggests that action topography along the dorsoventral axis is shaped by opposing constraints between social-related and object-related cues, thus providing a powerful framework to explain the representational organization of action in LOTC. Additional studies investigating functional/effective connectivity with the LOTC will give us better insight into the likelihood of this framework. More broadly, they will shed new lights about the extent to which activity patterns in the LOTC (encoding social and transitive action dimensions) are under the influence of, or even originate from, top-down connections.