The “why” and “how” of JointICA: Results from a visual detection task
Introduction
The activation of a particular brain area consists of the synchronized firing of a subpopulation of neurons in that area, involved in processing information or executing a particular task. This synchronized relevant neural firing can be measured with the electroencephalogram (EEG) as event-related potentials (ERP). The neural activity is additionally accompanied by a regional increase in cerebral blood flow. These regional cerebral blood flow changes can be measured directly as the blood oxygenation level dependent (BOLD) signal with functional magnetic resonance imaging (fMRI).
Moreover, EEG and BOLD changes can be measured simultaneously to benefit from their complementary properties. EEG measures electrical responses with a millisecond precision, but does not provide a precise spatial localization of the underlying cortical activity, since the electrode position is limited to the scalp surface. fMRI, on the other hand, measures local changes in brain hemodynamics with a very good spatial precision. However, the hemodynamic response is a slow signal and one echo-planar image (EPI) is only acquired every few seconds. This is far below the brain reaction time to externally applied stimuli, thus making the analysis of detailed temporal brain changes during the reaction on external stimuli very difficult. Moreover, an additional advantage of simultaneous recordings is the full reproducibility of the recording environment for both modalities. This is especially important in cognitive studies in which habituation, learning processes, arousal state or attention mechanisms play a role.
For this reason, the simultaneous acquisition of both EEG and fMRI is getting more and more popular, as their complementarity can provide deeper insight into function and dysfunction of brain dynamics (Debener and De Vos, 2011, Mulert and Lemieux, 2010, Ullsperger and Debener, 2010). This advantage has already been exploited in numerous applications. For instance, the combination of EEG and fMRI allows localizing epileptic activity based on spike-triggered fMRI (Bénar et al., 2006, Krakow et al., 2001, Lemieux et al., 2001, Seeck et al., 1998). Other possible applications are the study of ongoing brain rhythms (Goldman et al., 2000, Goldman et al., 2002, Laufs et al., 2003, Moosmann et al., 2003) and cerebral activations during sleep (Czisch et al., 2002, Liebenthal et al., 2003, Schabus et al., 2007). Also the analysis of event-related brain responses based on multimodal information (e.g. Calhoun et al., 2006, Debener et al., 2005, Debener et al., 2006, Eichele et al., 2008, Moosmann et al., 2008, Mulert et al., 2004) becomes more and more popular.
In recent years, several integration approaches have also been proposed. The earliest proposed methods were EEG-informed fMRI analysis (such as Debener et al., 2005, Mulert et al., 2008, Novitskiy et al., 2011) and fMRI-informed EEG analysis (Yang et al., 2010, De Martino et al., 2011, Lei et al., 2011). These approaches are asymmetric, meaning that one of the modalities is considered to be prior knowledge to improve the results in the other modality. In fMRI-informed EEG, the fMRI sources are used to improve the localization of the ERP generators. In EEG-informed fMRI, the ERP information is used to model the strength of the hemodynamic response based on single-trial ERP amplitude modulations, with the aim of localizing the modulation-related fMRI sources.
Data-driven signal processing techniques are well-established for processing EEG data (e.g. De Vos et al., 2007, De Vos et al., 2010, Porcaro et al., 2010, Viola et al., 2009), and are getting more and more popular for analyzing fMRI data (in the temporal domain: McKeown and Sejnowski, 1998, McKeown et al., 1998, McKeown, 2000; and in the spatial domain: Calhoun et al., 2001, Calhoun and Pearlson, 2004). Although no standard methods have been established, they are increasingly used for exploiting advantages of combined EEG–fMRI measurements. Attractive methods for this purpose are blind source separation (BSS) algorithms, such as independent component analysis (ICA), and canonical correlation analysis (CCA) (Mulert and Lemieux, 2010, Ullsperger and Debener, 2010). The best-known ICA-based algorithms for integrated EEG–fMRI analysis are Parallel ICA (Eichele et al., 2008, Lei et al., 2010, Liu and Calhoun, 2007), and JointICA (Calhoun et al., 2006, Calhoun et al., 2009), which will now be explained in more detail. CCA-based methods are not the focus of this paper; we refer the reader to Correa et al., 2008, Correa et al., 2010, and for the generalizations of correlation-based methods, see Martinez-Montes et al. (2004).
The Parallel ICA algorithm identifies components in both modalities separately, applying spatial ICA to fMRI data, and temporal ICA to single trial EEG data (Eichele et al., 2008). After extracting the independent components from both modalities, the corresponding components are identified based on correlations between trial-to-trial fluctuations in the temporal domain. Although the Parallel ICA algorithm shows a nice performance in data-fusion applications, the algorithm does not provide full data-fusion, since the modalities are not allowed to influence each other at the first stage of the decomposition.
A similar approach is followed by Lei et al. (2010) with the difference that, in their study, the components are linked in both the spatial and temporal domain using variational Bayesian techniques. In that way, the information of one modality can be used as prior for the analysis of the other one, thereby enhancing its spatial (for EEG) or temporal (for fMRI) resolution. In both these approaches, the data were processed separately first, and connections were only identified afterwards.
A parallel approach which imposes constraints during the parallel decomposition is proposed by Liu and Calhoun (2007). Although this approach also implies separate decompositions for the two modalities, a relationship can be defined between the two mixing matrices (belonging to the two modalities). As an example, in that paper the correlation between the mixing vectors of the two matrices is maximized.
JointICA, on the other hand, identifies the independent components for both modalities simultaneously. To do this, it starts from ERP epochs in the temporal domain and fMRI activation maps in the spatial domain. In the original paper on JointICA (Calhoun et al., 2006), the method was applied to averaged ERP data from different participants. Although another possible application has been shown on single-trial simulation data by Moosmann et al. (2008), in this paper we will focus on the original algorithm on averaged data. The method assumes that the different wave components (peaks) of the ERP and the spatial components in a statistical brain activation map (activation sites) of the same stimulus co-vary, either because they are generated in the same brain region (Logothetis et al. (2001) showed that the BOLD response is highly correlated with the group neuronal activity of the same brain region), or because the BOLD active areas had participatory roles in ERP activity, without necessary being the source of a particular ERP wave. Hence, it is believed that ICA will be able to disentangle these components and connect the electrical activations (ERP peaks) to their corresponding chemical (BOLD) brain activation sites (Calhoun et al., 2006).
The hypothesis about this one-to-one relation between the physiological origin of both data sets and their statistical dependence, however, is not necessarily true, since there is no particular mathematical reason to ensure capturing different ERP peaks always together with their corresponding fMRI activation sites. First of all, spatial statistical independence among fMRI components alone cannot be justified in the sense that there is no physical reason for the spatial samples to correspond to different activity patterns with independent distributions (see Daubechies et al., 2009). Furthermore, the two signals (ERP and fMRI) are very different in nature from a signal processing point of view. As used in the JointICA algorithm in this work, ERP is a temporal signal (we use information only from one electrode, which will be explained further in the manuscript) and fMRI a spatial map (obtained from a General Linear Model — GLM analysis) without any temporal information and their probabilistic distribution functions are different. The reader should note here that generally it is also possible to use concatenated ERPs from multiple electrodes as an input to JointICA, as done in the simulation study by Moosmann et al. (2008), therefore obtaining spatio-temporal ERP information. However, in this work, only ERPs from a single electrode are used.
Although JointICA was shown to provide meaningful results with real data when introduced as a tool for integrated ERP and fMRI data analysis (Calhoun et al., 2006, Calhoun et al., 2009), the underlying ICA assumption that the ERP and fMRI information extracted in a single component are mutually dependent, and at the same time statistically independent among different components was not questioned. Also, the behavior and robustness of the algorithm were not investigated. For this reason, and because of a growing interest in the method for the integration of different multimodal datasets, also besides EEG–fMRI (Calhoun et al., 2010, Doñamayor et al., 2011, Franco et al., 2008, Xu et al., 2009), some more methodological concerns need to be explored.
In this paper, we therefore try to explore and validate the performance of JointICA and its central assumption that physiological linking between ERP and BOLD amplitudes drives the extraction of multimodal components despite their intrinsically different temporal and spatial nature. We also show that the ICA algorithm used in the JointICA method (Infomax — Bell and Sejnowski, 1995) plays a crucial role in this multimodal separation. To support this claim, we investigate the performance of JointICA when other well-known ICA algorithms are used (JADE — Cardoso and Souloumiac, 1993; and FastICA — Hyvärinen, 1998), and show the disadvantages of these algorithms compared to the performance of Infomax.
To validate the meaningfulness of the JointICA results, we used a simple and well-established visual detection task with known ERP components and fMRI activation sites. This task is well-suited for validation purposes, as consistent results have been repeatedly found in several earlier studies (Di Russo et al., 2003, Di Russo et al., 2005, Novitskiy et al., 2011). In the mentioned work, the dipole pair accounting for the C1 ERP wave was found to lie in the calcarine cortex. The early P1 wave dipole was situated in the dorsal extrastriate cortex of the middle occipital gyrus, and the late P1 dipole was found in the ventral fusiform gyrus. The dipole that accounted for the early N1 ERP wave, was localized in the parietal lobe near the intraparietal sulcus.
It is hypothesized that a large part of brain activity is reflected both in EEG and fMRI modalities, and the link between them can be established. This we refer to as the central linking hypothesis. If we can find components that reflect both EEG and fMRI activity, it provides evidence to support the CL hypothesis. The contribution of the central linking hypothesis is evaluated first by randomly reassigning ERPs and fMRI activation maps over participants, thus destroying the amplitude link between the two modalities, and second by a comparison with an individual ICA analysis of both ERPs and fMRI maps, thus excluding any intermodal interaction. Furthermore, we investigate the effect of the ERP quality for JointICA by comparing recordings inside and outside of the scanner and the dependence of the JointICA results on the sample size.
Based on the results obtained with these different analyses we will discuss why and when (under which conditions) JointICA works well, hopefully providing a better understanding for potential future users.
Section snippets
Subjects
Twenty-six healthy subjects (11 female and 15 male, aged 18–44) with no history of neurological or cardiological disorders participated in this study. Written informed consent was obtained in accordance with the local ethical committee guidelines. During the simultaneous measurements subjects were lying supine in the scanner on a cushion that ameliorated the pressure from the EEG electrodes on the head, and with soft cushions to the side to restrict head movement in the coil. Subjects were
Application of JointICA — simultaneously recorded data
In this section, we present the results of the analyses on data for the simultaneous measurements in two ways. In Fig. 2, the JointICA decomposition is shown for the down-left visual field stimuli. The ERP data were derived from the electrode PO8, on which the P1, N1, and P2 ERP peaks are clearly visible, and the corresponding fMRI IC activations obtained from JointICA are shown.
Additionally, since the C1 ERP peak reflecting the initial bottom–up neural response of primary visual neurons
Discussion
In this study, we evaluated and presented the performance of JointICA, a recently proposed method for a symmetric integration of EEG and fMRI. To this end, we recorded EEG and fMRI data during a simple visual detection task. As this particular task is very well described in the literature (Di Russo et al., 2002, Di Russo et al., 2003, Di Russo et al., 2005, Martinez et al., 2001), we use this information to assess the relevance of the results in our experiments. For this reason, we did not
Conclusion
In this study, we investigated the performance of JointICA for EEG–fMRI data on a visual detection task. We showed and compared the performance when the EEG was recorded both simultaneously with fMRI data and in a separate session (outside the scanner room). The importance of the ICA algorithm, which is used for the decomposition, is also analyzed and discussed. Although it was previously suggested to use Infomax, it was better explained in the Discussion section of this manuscript why exactly
Acknowledgments
This research is supported by
- •
Research Council KUL: GOA MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC), PFV/10/002 (OPTEC), IDO 05/010 EEG–fMRI, IDO 08/013 Autism;
- •
Flemish Government:
- o
FWO: PhD/postdoc grants, projects: FWO G.0302.07 (SVM), G.0341.07 (Data fusion), G.0427.10N (Integrated EEG–fMRI) research communities (ICCoS, ANMMM);
- o
TBM080658-MRI (EEG–fMRI);
- o
IBBT
- o
- •
Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, ‘Dynamical systems, control and optimization’, 2007–2011); ESA PRODEX
References (72)
- et al.
A method for removing imaging artifact from continuous EEG recorded during functional MRI
Neuroimage
(2000) - et al.
Effect of the static magnetic field of the MR-scanner on ERPs: evaluation of visual, cognitive and motor potentials
Clin. Neurophysiol.
(2010) - et al.
Single trial variability of EEG and fMRI responses to visual stimuli
Neuroimage
(2007) - et al.
EEG–fMRI of epileptic spikes: concordance with EEG source localization and intracranial EEG
Neuroimage
(2006) - et al.
Neuronal chronometry of target detection: fusion of hemodynamic and event-related potential data
Neuroimage
(2006) Independent component analysis, a new concept?
Signal Process.
(1994)- et al.
Performance of blind source separation algorithms for fMRI using a group ICA method
Magn. Reson. Imaging
(2007) - et al.
Altered processing of acoustic stimuli during sleep: reduced auditory activation and visual deactivation detected by a combined fMRI/EEG study
Neuroimage
(2002) - et al.
Predicting EEG single trial responses with simultaneous fMRI and relevance vector machine regression
Neuroimage
(2011) - et al.
Canonical decomposition of ictal scalp EEG reliably detects the seizure onset zone
Neuroimage
(2007)
Single-trial EEG–fMRI reveals the dynamics of cognitive function
Trends Cogn. Sci.
EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis
J. Neurosci. Methods
Identification of the neural sources of the pattern-reversal VEP
Neuroimage
Unmixing concurrent EEG–fMRI with parallel independent component analysis
Int. J. Psychophysiol.
Acquiring simultaneous EEG and functional MRI
J. Clin. Neurophysiol.
Validating the independent components of neuroimaging time-series via clustering and visualization
Neuroimage
EEG-correlated fMRI of human alpha activity
Neuroimage
A parallel framework for simultaneous EEG/fMRI analysis: methodology and simulation
Neuroimage
Event-related fMRI with simultaneous and continuous EEG: description of the method and initial case report
Neuroimage
Simultaneous ERP and fMRI of the auditory cortex in a passive oddball paradigm
Neuroimage
Putting spatioal attention on the map: timing and localization of stimulus selection process in striate and extrastriate visual areas
Vision Res.
Concurrent EEG/fMRI analysis by multiway partial least squares
Neuroimage
EEG signatures of auditory activity correlate with simultaneously recorded fMRI responses in humans
Neuroimage
Detection of consistently task-related activations in fMRI data with hybrid independent component analysis
Neuroimage
Correlates of alpha rhythm in functional magnetic resonance imaging and near infrared spectroscopy
Neuroimage
Joint independent component analysis for simultaneous EEG–fMRI: principle and simulation
Int. J. Psychophysiol.
Realignment parameter-informed artefact correction for simultaneous EEG–fMRI recordings
Neuroimage
Integration of fMRI and simultaneous EEG: towards a comprehensive understanding of localization and time-course of brain activity in target detection
Neuroimage
Single-trial coupling of EEG and fMRI reveals the involvement of early anterior cingulate cortex activation in effortful decision making
Neuroimage
Removal of FMRI environment artifacts from EEG data using optimal basis sets
Neuroimage
The BOLD correlates of the visual P1 and N1 in single-trial analysis of simultaneous EEG–fMRI recordings during a spatial detection task
Neuroimage
Functional source separation improves the quality of single trial visual evoked potentials recorded during concurrent EEG–fMRI
Neuroimage
Non-invasive epileptic focus localization using EEG-triggered functional MRI and electromagnetic tomography
Electroencephalogr. Clin. Neurophysiol.
Removal of BCG artifacts from EEG recordings inside the MR scanner: a comparison of methodological and validation-related aspects
Neuroimage
Optimization of experimental design in fMRI: a general framework using a genetic algorithm
Neuroimage
Joint source based morphometry identifies linked gray and white matter group differences
Neuroimage
Cited by (45)
Coupled tensor decompositions for data fusion
2021, Tensors for Data Processing: Theory, Methods, and ApplicationsExamining the role of feedback in TMS-induced visual suppression: A cautionary tale
2019, Consciousness and CognitionA method to compare the discriminatory power of data-driven methods: Application to ICA and IVA
2019, Journal of Neuroscience MethodsReal-time EEG feedback during simultaneous EEG-fMRI identifies the cortical signature of motor imagery
2015, NeuroImageCitation Excerpt :We consider this a feasible scenario for successful neurorehabilitation, and this scenario could not be achieved with EEG or fMRI in isolation. Associations between electrophysiological and hemodynamic signatures at between- and within-subject levels have previously been confirmed with concurrent EEG–fMRI (Debener et al., 2005; Mijović et al., 2012b). For MI, a negative correlation between contralateral EEG sensorimotor rhythm amplitude and fMRI hemodynamic signals in sensorimotor areas has also previously been observed (Formaggio et al., 2010; Yuan et al., 2010).
Comparison of BCG artifact removal methods for evoked responses in simultaneous EEG-fMRI
2015, Journal of Neuroscience Methods