Elsevier

NeuroImage

Volume 60, Issue 2, 2 April 2012, Pages 1171-1185
NeuroImage

The “why” and “how” of JointICA: Results from a visual detection task

https://doi.org/10.1016/j.neuroimage.2012.01.063Get rights and content

Abstract

Since several years, neuroscience research started to focus on multimodal approaches. One such multimodal approach is the combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). However, no standard integration procedure has been established so far. One promising data-driven approach consists of a joint decomposition of event-related potentials (ERPs) and fMRI maps derived from the response to a particular stimulus. Such an algorithm (joint independent component analysis or JointICA) has recently been proposed by Calhoun et al. (2006). This method provides sources with both a fine spatial and temporal resolution, and has shown to provide meaningful results. However, the algorithm's performance has not been fully characterized yet, and no procedure has been proposed to assess the quality of the decomposition. In this paper, we therefore try to answer why and how JointICA works. We show the performance of the algorithm on data obtained in a visual detection task, and compare the performance for EEG recorded simultaneously with fMRI data and for EEG recorded in a separate session (outside the scanner room). We perform several analyses in order to set the necessary conditions that lead to a sound decomposition, and to give additional insights for exploration in future studies. In that respect, we show how the algorithm behaves when different EEG electrodes are used and we test the robustness with respect to the number of subjects in the study. The performance of the algorithm in all the experiments is validated based on results from previous studies.

Introduction

The activation of a particular brain area consists of the synchronized firing of a subpopulation of neurons in that area, involved in processing information or executing a particular task. This synchronized relevant neural firing can be measured with the electroencephalogram (EEG) as event-related potentials (ERP). The neural activity is additionally accompanied by a regional increase in cerebral blood flow. These regional cerebral blood flow changes can be measured directly as the blood oxygenation level dependent (BOLD) signal with functional magnetic resonance imaging (fMRI).

Moreover, EEG and BOLD changes can be measured simultaneously to benefit from their complementary properties. EEG measures electrical responses with a millisecond precision, but does not provide a precise spatial localization of the underlying cortical activity, since the electrode position is limited to the scalp surface. fMRI, on the other hand, measures local changes in brain hemodynamics with a very good spatial precision. However, the hemodynamic response is a slow signal and one echo-planar image (EPI) is only acquired every few seconds. This is far below the brain reaction time to externally applied stimuli, thus making the analysis of detailed temporal brain changes during the reaction on external stimuli very difficult. Moreover, an additional advantage of simultaneous recordings is the full reproducibility of the recording environment for both modalities. This is especially important in cognitive studies in which habituation, learning processes, arousal state or attention mechanisms play a role.

For this reason, the simultaneous acquisition of both EEG and fMRI is getting more and more popular, as their complementarity can provide deeper insight into function and dysfunction of brain dynamics (Debener and De Vos, 2011, Mulert and Lemieux, 2010, Ullsperger and Debener, 2010). This advantage has already been exploited in numerous applications. For instance, the combination of EEG and fMRI allows localizing epileptic activity based on spike-triggered fMRI (Bénar et al., 2006, Krakow et al., 2001, Lemieux et al., 2001, Seeck et al., 1998). Other possible applications are the study of ongoing brain rhythms (Goldman et al., 2000, Goldman et al., 2002, Laufs et al., 2003, Moosmann et al., 2003) and cerebral activations during sleep (Czisch et al., 2002, Liebenthal et al., 2003, Schabus et al., 2007). Also the analysis of event-related brain responses based on multimodal information (e.g. Calhoun et al., 2006, Debener et al., 2005, Debener et al., 2006, Eichele et al., 2008, Moosmann et al., 2008, Mulert et al., 2004) becomes more and more popular.

In recent years, several integration approaches have also been proposed. The earliest proposed methods were EEG-informed fMRI analysis (such as Debener et al., 2005, Mulert et al., 2008, Novitskiy et al., 2011) and fMRI-informed EEG analysis (Yang et al., 2010, De Martino et al., 2011, Lei et al., 2011). These approaches are asymmetric, meaning that one of the modalities is considered to be prior knowledge to improve the results in the other modality. In fMRI-informed EEG, the fMRI sources are used to improve the localization of the ERP generators. In EEG-informed fMRI, the ERP information is used to model the strength of the hemodynamic response based on single-trial ERP amplitude modulations, with the aim of localizing the modulation-related fMRI sources.

Data-driven signal processing techniques are well-established for processing EEG data (e.g. De Vos et al., 2007, De Vos et al., 2010, Porcaro et al., 2010, Viola et al., 2009), and are getting more and more popular for analyzing fMRI data (in the temporal domain: McKeown and Sejnowski, 1998, McKeown et al., 1998, McKeown, 2000; and in the spatial domain: Calhoun et al., 2001, Calhoun and Pearlson, 2004). Although no standard methods have been established, they are increasingly used for exploiting advantages of combined EEG–fMRI measurements. Attractive methods for this purpose are blind source separation (BSS) algorithms, such as independent component analysis (ICA), and canonical correlation analysis (CCA) (Mulert and Lemieux, 2010, Ullsperger and Debener, 2010). The best-known ICA-based algorithms for integrated EEG–fMRI analysis are Parallel ICA (Eichele et al., 2008, Lei et al., 2010, Liu and Calhoun, 2007), and JointICA (Calhoun et al., 2006, Calhoun et al., 2009), which will now be explained in more detail. CCA-based methods are not the focus of this paper; we refer the reader to Correa et al., 2008, Correa et al., 2010, and for the generalizations of correlation-based methods, see Martinez-Montes et al. (2004).

The Parallel ICA algorithm identifies components in both modalities separately, applying spatial ICA to fMRI data, and temporal ICA to single trial EEG data (Eichele et al., 2008). After extracting the independent components from both modalities, the corresponding components are identified based on correlations between trial-to-trial fluctuations in the temporal domain. Although the Parallel ICA algorithm shows a nice performance in data-fusion applications, the algorithm does not provide full data-fusion, since the modalities are not allowed to influence each other at the first stage of the decomposition.

A similar approach is followed by Lei et al. (2010) with the difference that, in their study, the components are linked in both the spatial and temporal domain using variational Bayesian techniques. In that way, the information of one modality can be used as prior for the analysis of the other one, thereby enhancing its spatial (for EEG) or temporal (for fMRI) resolution. In both these approaches, the data were processed separately first, and connections were only identified afterwards.

A parallel approach which imposes constraints during the parallel decomposition is proposed by Liu and Calhoun (2007). Although this approach also implies separate decompositions for the two modalities, a relationship can be defined between the two mixing matrices (belonging to the two modalities). As an example, in that paper the correlation between the mixing vectors of the two matrices is maximized.

JointICA, on the other hand, identifies the independent components for both modalities simultaneously. To do this, it starts from ERP epochs in the temporal domain and fMRI activation maps in the spatial domain. In the original paper on JointICA (Calhoun et al., 2006), the method was applied to averaged ERP data from different participants. Although another possible application has been shown on single-trial simulation data by Moosmann et al. (2008), in this paper we will focus on the original algorithm on averaged data. The method assumes that the different wave components (peaks) of the ERP and the spatial components in a statistical brain activation map (activation sites) of the same stimulus co-vary, either because they are generated in the same brain region (Logothetis et al. (2001) showed that the BOLD response is highly correlated with the group neuronal activity of the same brain region), or because the BOLD active areas had participatory roles in ERP activity, without necessary being the source of a particular ERP wave. Hence, it is believed that ICA will be able to disentangle these components and connect the electrical activations (ERP peaks) to their corresponding chemical (BOLD) brain activation sites (Calhoun et al., 2006).

The hypothesis about this one-to-one relation between the physiological origin of both data sets and their statistical dependence, however, is not necessarily true, since there is no particular mathematical reason to ensure capturing different ERP peaks always together with their corresponding fMRI activation sites. First of all, spatial statistical independence among fMRI components alone cannot be justified in the sense that there is no physical reason for the spatial samples to correspond to different activity patterns with independent distributions (see Daubechies et al., 2009). Furthermore, the two signals (ERP and fMRI) are very different in nature from a signal processing point of view. As used in the JointICA algorithm in this work, ERP is a temporal signal (we use information only from one electrode, which will be explained further in the manuscript) and fMRI a spatial map (obtained from a General Linear Model — GLM analysis) without any temporal information and their probabilistic distribution functions are different. The reader should note here that generally it is also possible to use concatenated ERPs from multiple electrodes as an input to JointICA, as done in the simulation study by Moosmann et al. (2008), therefore obtaining spatio-temporal ERP information. However, in this work, only ERPs from a single electrode are used.

Although JointICA was shown to provide meaningful results with real data when introduced as a tool for integrated ERP and fMRI data analysis (Calhoun et al., 2006, Calhoun et al., 2009), the underlying ICA assumption that the ERP and fMRI information extracted in a single component are mutually dependent, and at the same time statistically independent among different components was not questioned. Also, the behavior and robustness of the algorithm were not investigated. For this reason, and because of a growing interest in the method for the integration of different multimodal datasets, also besides EEG–fMRI (Calhoun et al., 2010, Doñamayor et al., 2011, Franco et al., 2008, Xu et al., 2009), some more methodological concerns need to be explored.

In this paper, we therefore try to explore and validate the performance of JointICA and its central assumption that physiological linking between ERP and BOLD amplitudes drives the extraction of multimodal components despite their intrinsically different temporal and spatial nature. We also show that the ICA algorithm used in the JointICA method (Infomax — Bell and Sejnowski, 1995) plays a crucial role in this multimodal separation. To support this claim, we investigate the performance of JointICA when other well-known ICA algorithms are used (JADE — Cardoso and Souloumiac, 1993; and FastICA — Hyvärinen, 1998), and show the disadvantages of these algorithms compared to the performance of Infomax.

To validate the meaningfulness of the JointICA results, we used a simple and well-established visual detection task with known ERP components and fMRI activation sites. This task is well-suited for validation purposes, as consistent results have been repeatedly found in several earlier studies (Di Russo et al., 2003, Di Russo et al., 2005, Novitskiy et al., 2011). In the mentioned work, the dipole pair accounting for the C1 ERP wave was found to lie in the calcarine cortex. The early P1 wave dipole was situated in the dorsal extrastriate cortex of the middle occipital gyrus, and the late P1 dipole was found in the ventral fusiform gyrus. The dipole that accounted for the early N1 ERP wave, was localized in the parietal lobe near the intraparietal sulcus.

It is hypothesized that a large part of brain activity is reflected both in EEG and fMRI modalities, and the link between them can be established. This we refer to as the central linking hypothesis. If we can find components that reflect both EEG and fMRI activity, it provides evidence to support the CL hypothesis. The contribution of the central linking hypothesis is evaluated first by randomly reassigning ERPs and fMRI activation maps over participants, thus destroying the amplitude link between the two modalities, and second by a comparison with an individual ICA analysis of both ERPs and fMRI maps, thus excluding any intermodal interaction. Furthermore, we investigate the effect of the ERP quality for JointICA by comparing recordings inside and outside of the scanner and the dependence of the JointICA results on the sample size.

Based on the results obtained with these different analyses we will discuss why and when (under which conditions) JointICA works well, hopefully providing a better understanding for potential future users.

Section snippets

Subjects

Twenty-six healthy subjects (11 female and 15 male, aged 18–44) with no history of neurological or cardiological disorders participated in this study. Written informed consent was obtained in accordance with the local ethical committee guidelines. During the simultaneous measurements subjects were lying supine in the scanner on a cushion that ameliorated the pressure from the EEG electrodes on the head, and with soft cushions to the side to restrict head movement in the coil. Subjects were

Application of JointICA — simultaneously recorded data

In this section, we present the results of the analyses on data for the simultaneous measurements in two ways. In Fig. 2, the JointICA decomposition is shown for the down-left visual field stimuli. The ERP data were derived from the electrode PO8, on which the P1, N1, and P2 ERP peaks are clearly visible, and the corresponding fMRI IC activations obtained from JointICA are shown.

Additionally, since the C1 ERP peak reflecting the initial bottom–up neural response of primary visual neurons

Discussion

In this study, we evaluated and presented the performance of JointICA, a recently proposed method for a symmetric integration of EEG and fMRI. To this end, we recorded EEG and fMRI data during a simple visual detection task. As this particular task is very well described in the literature (Di Russo et al., 2002, Di Russo et al., 2003, Di Russo et al., 2005, Martinez et al., 2001), we use this information to assess the relevance of the results in our experiments. For this reason, we did not

Conclusion

In this study, we investigated the performance of JointICA for EEG–fMRI data on a visual detection task. We showed and compared the performance when the EEG was recorded both simultaneously with fMRI data and in a separate session (outside the scanner room). The importance of the ICA algorithm, which is used for the decomposition, is also analyzed and discussed. Although it was previously suggested to use Infomax, it was better explained in the Discussion section of this manuscript why exactly

Acknowledgments

This research is supported by

  • Research Council KUL: GOA MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC), PFV/10/002 (OPTEC), IDO 05/010 EEG–fMRI, IDO 08/013 Autism;

  • Flemish Government:

    • o

      FWO: PhD/postdoc grants, projects: FWO G.0302.07 (SVM), G.0341.07 (Data fusion), G.0427.10N (Integrated EEG–fMRI) research communities (ICCoS, ANMMM);

    • o

      TBM080658-MRI (EEG–fMRI);

    • o

      IBBT

  • Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, ‘Dynamical systems, control and optimization’, 2007–2011); ESA PRODEX

References (72)

  • S. Debener et al.

    Single-trial EEG–fMRI reveals the dynamics of cognitive function

    Trends Cogn. Sci.

    (2006)
  • A. Delorme et al.

    EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis

    J. Neurosci. Methods

    (2004)
  • F. Di Russo et al.

    Identification of the neural sources of the pattern-reversal VEP

    Neuroimage

    (2005)
  • T. Eichele et al.

    Unmixing concurrent EEG–fMRI with parallel independent component analysis

    Int. J. Psychophysiol.

    (2008)
  • R.I. Goldman et al.

    Acquiring simultaneous EEG and functional MRI

    J. Clin. Neurophysiol.

    (2000)
  • J. Himberg et al.

    Validating the independent components of neuroimaging time-series via clustering and visualization

    Neuroimage

    (2004)
  • H. Laufs et al.

    EEG-correlated fMRI of human alpha activity

    Neuroimage

    (2003)
  • X. Lei et al.

    A parallel framework for simultaneous EEG/fMRI analysis: methodology and simulation

    Neuroimage

    (2010)
  • L. Lemieux et al.

    Event-related fMRI with simultaneous and continuous EEG: description of the method and initial case report

    Neuroimage

    (2001)
  • E. Liebenthal et al.

    Simultaneous ERP and fMRI of the auditory cortex in a passive oddball paradigm

    Neuroimage

    (2003)
  • A. Martinez et al.

    Putting spatioal attention on the map: timing and localization of stimulus selection process in striate and extrastriate visual areas

    Vision Res.

    (2001)
  • E. Martinez-Montes et al.

    Concurrent EEG/fMRI analysis by multiway partial least squares

    Neuroimage

    (2004)
  • S.D. Mayhew et al.

    EEG signatures of auditory activity correlate with simultaneously recorded fMRI responses in humans

    Neuroimage

    (2010)
  • M.J. McKeown

    Detection of consistently task-related activations in fMRI data with hybrid independent component analysis

    Neuroimage

    (2000)
  • M. Moosmann et al.

    Correlates of alpha rhythm in functional magnetic resonance imaging and near infrared spectroscopy

    Neuroimage

    (2003)
  • M. Moosmann et al.

    Joint independent component analysis for simultaneous EEG–fMRI: principle and simulation

    Int. J. Psychophysiol.

    (2008)
  • M. Moosmann et al.

    Realignment parameter-informed artefact correction for simultaneous EEG–fMRI recordings

    Neuroimage

    (2009)
  • C. Mulert et al.

    Integration of fMRI and simultaneous EEG: towards a comprehensive understanding of localization and time-course of brain activity in target detection

    Neuroimage

    (2004)
  • C. Mulert et al.

    Single-trial coupling of EEG and fMRI reveals the involvement of early anterior cingulate cortex activation in effortful decision making

    Neuroimage

    (2008)
  • R.K. Niazy et al.

    Removal of FMRI environment artifacts from EEG data using optimal basis sets

    Neuroimage

    (2005)
  • N. Novitskiy et al.

    The BOLD correlates of the visual P1 and N1 in single-trial analysis of simultaneous EEG–fMRI recordings during a spatial detection task

    Neuroimage

    (2011)
  • C. Porcaro et al.

    Functional source separation improves the quality of single trial visual evoked potentials recorded during concurrent EEG–fMRI

    Neuroimage

    (2010)
  • M. Seeck et al.

    Non-invasive epileptic focus localization using EEG-triggered functional MRI and electromagnetic tomography

    Electroencephalogr. Clin. Neurophysiol.

    (1998)
  • K. Vanderperren et al.

    Removal of BCG artifacts from EEG recordings inside the MR scanner: a comparison of methodological and validation-related aspects

    Neuroimage

    (2010)
  • T.D. Wager et al.

    Optimization of experimental design in fMRI: a general framework using a genetic algorithm

    Neuroimage

    (2003)
  • L. Xu et al.

    Joint source based morphometry identifies linked gray and white matter group differences

    Neuroimage

    (2009)
  • Cited by (45)

    • Coupled tensor decompositions for data fusion

      2021, Tensors for Data Processing: Theory, Methods, and Applications
    • Real-time EEG feedback during simultaneous EEG-fMRI identifies the cortical signature of motor imagery

      2015, NeuroImage
      Citation Excerpt :

      We consider this a feasible scenario for successful neurorehabilitation, and this scenario could not be achieved with EEG or fMRI in isolation. Associations between electrophysiological and hemodynamic signatures at between- and within-subject levels have previously been confirmed with concurrent EEG–fMRI (Debener et al., 2005; Mijović et al., 2012b). For MI, a negative correlation between contralateral EEG sensorimotor rhythm amplitude and fMRI hemodynamic signals in sensorimotor areas has also previously been observed (Formaggio et al., 2010; Yuan et al., 2010).

    View all citing articles on Scopus
    View full text