Toward an Interpersonal Neuroscience in Technologically Assisted (Virtual) Interactions

Social interaction is a fundamental aspect of our daily lives across the life span. In recent times, there has been an effort by scientists to shift from single-brain research to multibrain or interpersonal neuroscience (Pan et al., 2022), with the aim of advancing our understanding of neural mechanisms underpinning live social interaction. A typical interpersonal neuroscience approach involves the simultaneous collection of data from multiple brains during a naturalistic social encounter (Cui et al., 2012). By computing interbrain synchrony or coherence, researchers can investigate the interdependencies between brain signals. Studies have shown that increases in interbrain coherence are strongly linked with cooperation (Cui et al., 2012), learning (Pan et al., 2023), and communication (Jiang et al., 2015). It is expected that the coherence value will rise in the presence of (behavioral, cognitive, or affective) interactions between individuals (Pan et al., 2022).

Nowadays, a growing number of social interactions are taking place virtually on technology-supported platforms, such as video conferencing (Brucks and Levav, 2022). This trend began as a result of the need to overcome geographical barriers and enhance cross-regional collaboration. To develop virtual technology for better communications, it is imperative to deepen our understanding of the neural and behavioral mechanisms underlying the effects of virtual technology on human interactions. Recently, using interpersonal neuroscience paradigms, researchers have delved into the interbrain neural processes involved in virtual interactions (Lu et al., 2020; Schwartz et al., 2022). It appears that virtual technology has the potential to influence interpersonal communication; for instance, video conferencing narrows communicators' cognitive attention to the screen, leading to a more focused interaction (as opposed to in-person interactions) (Brucks and Levav, 2022). As a result, the patterns of interbrain coherence were anticipated to differ between virtual and in-person interactions.

In a recent article published in The Journal of Neuroscience, Balters et al. (2023) investigated the impact of video conferencing on social interactions by examining intrabrain activity and interbrain coherence using functional near-infrared spectroscopy (fNIRS) hyperscanning. fNIRS measures regional cerebral blood flow through detecting changes in oxyhemoglobin concentration. In this study, fNIRS channels covered the entire cortex for each participant, thereby enabling the examination of intrabrain activity and both static and dynamic interbrain coherence throughout the cortex. During the experiment, participants were instructed to complete three naturalistic tasks (i.e., problem-solving, creative-innovation, socio-emotional task) either in person or through a virtual platform (Zoom). The participants' behaviors were evaluated using video/audio recordings and questionnaire responses. Balters et al. (2023) found that video conferencing led to a decrease in conversational turn-taking behavior. Turn-taking behavior was positively correlated with subjective cooperation and task performance. Intrabrain activation did not differ between the in-person and virtual conditions. Moreover, video conferencing was found to generate higher levels of interbrain coherence in the left dorsal frontopolar regions during the problem-solving task and in bilateral dorsal frontopolar regions during the creative-innovation task compared with the in-person condition; however, in the case of the socio-emotional task, it resulted in decreased interbrain coherence in the right dorsal frontopolar regions. Four distinct interbrain coherence states (i.e., organized patterns of coherence) were revealed through dynamic analysis: State 1 was characterized by low interbrain coherence across the cerebral cortex; State 3 exhibited a comparable coherence pattern with stronger coherence than State 1; State 2 was distinguished by heightened interbrain coherence in the right frontopolar areas; and State 4 demonstrated increased coherence between the bilateral frontopolar areas. Results further showed interbrain coherence. State 1 was more prevalent in the virtual condition and negatively correlated with conversational turn-taking behaviors during the problem-solving task. These findings suggest that video conferencing may promote interbrain coherence patterns that are negatively correlated with conversational behaviors conducive to successful social interactions.

Balters et al. (2023) used fNIRS-based hyperscanning to unveil the neural substrates underlying technologically assisted social interactions in this study. fNIRS has several advantages in exploring social interactions. Compared with EEG, fNIRS provides higher spatial resolution; and compared with fMRI, it offers higher temporal resolution. Additionally, fNIRS has greater ecological validity compared with both EEG and fMRI. For instance, fNIRS enables face-to-face dialogue paradigms, which is challenging (and perhaps costly) by using fMRI, and is less susceptible to motor artifacts, such as vocalization and gestures, which often affect EEG recordings. Hyperscanning, a developing multibrain imaging technique, allows for the computations of interbrain synchrony or coherence, providing a more comprehensive view of the interaction between brains compared with the traditional single-brain approach (Cui et al., 2012). Using fNIRS-based hyperscanning, Lu et al. (2020) observed decreased interbrain coherence at the right angular gyrus in computer-assisted communication compared with face-to-face dialogue. Likewise, Schwartz et al. (2022) reported that technology attenuated interbrain synchrony that was associated with socio-affective signals during live social moments. Given the benefits of fNIRS compared with EEG or fMRI and the advantages of hyperscanning over a single-brain approach, fNIRS-based hyperscanning is considered as a preferable choice for exploring social brains.

Balters et al. (2023) took a comprehensive approach in their analysis of social brains by incorporating both single-brain activation and interbrain coherence across different regions. Unlike previous studies that mostly focused on specific ROIs because of limited number of fNIRS optodes, Balters et al. (2023) analyzed the entire cortex for each participant, providing a more complete picture of the neural responses in the scenario of technologically assisted communication. By combining intrabrain (neural activation and connection) and interbrain (coherence between same or cross regions of paired participants) analyses, their study offers a more nuanced understanding of the complex neural system of social brains.

The dynamic analysis of interbrain coherence provided a more dynamic understanding of the neural mechanisms underlying social brain during technologically assisted communication. Balters et al. (2023) identified four dynamic interbrain coherence states: in the virtual condition compared with the in-person condition, the occurrence of dynamic interbrain coherence State 1 was higher while the occurrence of States 2 and 3 was lower; in addition, the occurrence rates of States 1-3 correlated with conversational turn-taking behaviors. This suggests that technology has a complex and dynamic impact on human communication, as it triggers diverse patterns of coherence between brains overtime, rather than a simple and static effect. Li et al. (2021) developed such a dynamic functional connectivity approach in a previous hyperscanning study. They identified several interbrain coherence states during dyadic problem-solving tasks and analyzed these states based on fractional windows, transitions between states, and global state efficiency. Despite the difficulty in determining the precise meaning of these states and the connections between them, this dynamic approach will enhance our comprehension of human communications regardless of whether technology is involved or not.

Future research may benefit from using permutation tests to compare real intrabrain and interbrain coherence with those obtained from the surrogate data (generated through techniques, e.g., phase randomization or re-pairing of participants from different groups). The permutation test enables testing the possibility that intrabrain coherence simply arises from long-range temporal autocorrelation in the blood oxygen level-dependent signal, and examining whether significant interbrain coherence is specific to interacting individuals (Pan et al., 2023). Furthermore, the exact functional relevance of interbrain coherence observed in various social interactions requires further investigations. Along this line, Balters et al. (2023) analyzed the correlations between interbrain coherence and ultimate behavioral performance, such as the total number of turn-taking instances. Other researchers also attempted to address this by linking interbrain coherence to different types of behaviors (e.g., vocal and nonvocal interactions) for each time point (Jiang et al., 2015). Through the behavioral coding approach, one can obtain time-stamped behavioral markers that can be further analyzed along with the brain data. The measurement of concurrent social behaviors of interacting individuals holds strong relevance for interpreting the functional meanings of interbrain coherence.

Technologically assisted communication has become an integral part of our daily life, allowing for remote connections, but also leading to issues, such as misinterpretation of information and “Zoom fatigue.” Balters et al. (2023) provided evidence suggesting that technologically assisted interactions may shape patterns of interbrain coherence that constrain conversational turn-taking. These insights can guide the design and development of improved video conferencing technology in the future, supporting effective communication in both general population and people with social deficits (e.g., autism).