Decoding human brain activity during real-world experiences

The human brain evolved to function and survive in a highly stimulating, complex and fast-changing world. Attempting to ascertain the neural substrates of operating in naturalistic contexts represents a huge challenge. Recently, however, researchers have begun to use several innovative analysis methods to interrogate functional magnetic resonance imaging (fMRI) data collected during dynamic naturalistic tasks. Central to these new developments is the inventive approach taken to segregating neural activity linked to specific events within the overall continuous stream of complex stimulation. In this review, we discuss the recent literature, detailing the key studies and their methods. These analytical techniques can be applied in a wide range of cognitive domains and, thus, offer exciting new opportunities for gaining insights into the brain bases of thoughts and behaviours in the real-world setting where they normally occur.

Introduction

Understanding how our brain makes sense of continuous and complex inputs from the external world represents a central challenge in cognitive neuroscience. Functional neuroimaging offers a means to address this key issue, but, because many studies use simplified static stimuli, surprisingly little is known about how the human brain operates during real-world experiences. Exploring brain function with dynamic naturalistic stimuli is important for several reasons. First, it is vital to verify whether results obtained in experiments that used simplified stimuli actually hold true under natural conditions, particularly because findings are often assumed to generalize. For example, do brain regions such as the fusiform face area also show selectivity when faces appear in the real world? Second, some research questions can only be addressed with naturalistic tasks where there is little temporal regularity. To understand properly the neural substrates of driving a vehicle or navigating in a city, for instance, it is sub-optimal to use static regularized stimuli.

The need to complement highly controlled experimental manipulations has been acknowledged in the field of functional neuroimaging; consequently, the use of dynamic stimuli such as movies and virtual reality environments is increasing 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18. However, most extant studies involving naturalistic stimuli have designs where activity associated with each task (e.g. navigation or a control task) was averaged across blocks of typically 30–60 s duration (Figure 1a). This approach not only lacks fine-grained temporal resolution but also reduces the correspondence to the real world, which is rarely organized in a ‘blocked’ and orderly manner. For a true appreciation of brain function during real-world experiences, one key element that it is vital to understand is how to segregate neural activity during specific events from the continuous stream of complex stimulation of which they are a part. However, this presents a significant challenge because the lack of discrete stimuli means that standard experimental designs and analyses cannot be used (Figure 1c). Dynamic, continuous stimuli can evoke both transient and sustained responses within the same brain region or in different brain regions simultaneously, making data interpretation difficult. The unconstrained nature of eye-movements, multiple features in any given scene, and the lack of a specific task can give rise to ambiguity about what subjects were attending to or thinking about during the experience.

Recently, however, researchers have adopted a range of innovative approaches to analysing fMRI data collected during naturalistic tasks. The analytical methods tend to fall within three broad categories: (i) those based on subjects’ classification of events; (ii) those based on subjects’ behaviour and verbal reports; and (3) stimulus ‘blind’ analyses, which in general do not require information about the stimuli to be known before the analysis. Several recent reviews have already discussed one of the analysis methods from the third category, namely multi-voxel pattern analysis (see Refs 19, 20). By contrast our focus here is different. First, we widen the scope considerably to explore work from the first two categories, as well as additional stimulus ‘blind’ approaches. Second, our interest is in how these innovations have been applied specifically to the analysis of fMRI data acquired during dynamic naturalistic tasks.

We will consider each category of analytical method in turn, reviewing the key studies. Although a diversity of cognitive domains and scientific agendas are represented, two common themes are evident. Initial studies have focussed on investigating whether patterns of brain activity observed previously using experimental stimuli are mirrored during naturalistic tasks. However, it is also apparent that experience with the techniques is growing, and the ability to combine these methods with naturalistic stimuli is beginning to permit novel insights that would be difficult to gain using more traditional approaches. Future developments, therefore, might lead to genuine conceptual advances. Given this potential, it is timely to consider the different techniques used, and their advantages and disadvantages, in order that researchers might identify new opportunities for experimental investigations involving naturalistic contexts that could add a new dimension to their work.