Machine learning for real-time single-trial EEG-analysis: From brain–computer interfacing to mental state monitoring

doi:10.1016/j.jneumeth.2007.09.022

Journal of Neuroscience Methods

Volume 167, Issue 1, 15 January 2008, Pages 82-90

https://doi.org/10.1016/j.jneumeth.2007.09.022 Get rights and content

Abstract

Machine learning methods are an excellent choice for compensating the high variability in EEG when analyzing single-trial data in real-time. This paper briefly reviews preprocessing and classification techniques for efficient EEG-based brain–computer interfacing (BCI) and mental state monitoring applications. More specifically, this paper gives an outline of the Berlin brain–computer interface (BBCI), which can be operated with minimal subject training. Also, spelling with the novel BBCI-based Hex-o-Spell text entry system, which gains communication speeds of 6–8 letters per minute, is discussed. Finally the results of a real-time arousal monitoring experiment are presented.

Introduction

Recently, advances in single-trial EEG-analysis have achieved the efficient online differentiation of neuroelectric signals. The present contribution distinguishes between two main application fields of these analysis techniques: (a) brain–computer interfacing and (b) online monitoring of brain states. This paper reviews both along with the machine learning and signal analysis machinery that is necessary for such online EEG processing.

Brain–computer interfaces (BCI) allow for communication that is solely based on brain signals, independent from muscles or peripheral nerves (see Wolpaw et al., 2002, Kübler et al., 2001, Curran and Stokes, 2003, Kübler and Müller, 2007, Dornhege et al., 2007a, Carmena et al., 2003 for a broader overview and background information). The Berlin brain–computer interface (BBCI) is a non-invasive, EEG-based system whose key features are (1) the use of motor imagery for control tasks, (2) advanced machine learning techniques that automatically extract complex high-dimensional features and classify in a robust manner, and – as a consequence – (3) no need for subject training. The latter characteristic is considered an important contribution since typical BCI systems rely on classical conditioning (cf. Elbert et al., 1980, Rockstroh et al., 1984, Birbaumer et al., 2000) and require extensive subject training of 50–100 h.¹ In contrast, the BBCI approach allows to shift the training effort from the user towards the machine (cf. Section 3). Section 4.1 will demonstrate a BBCI application: the Hex-o-Spell interface for spelling.

Brain–computer interfacing is certainly not the only interesting application when decoding brain activity. General online monitoring of generic brain states beyond voluntarily altered brain activity has in the past been under study, e.g. for the detection of sleep stages, tiredness, arousal, for emotion monitoring and for cognitive workload analysis Kohlmorgen et al., 2007, Haynes and Rees, 2006, Müller et al., 1995. In Section 4.3 the real-time monitoring of mental states using EEG are discussed and the example of monitoring a subject’s arousal to estimate it’s concentration ability within an industrial problem setting is briefly outlined.

Section snippets

Machine learning for BCI

Since brain data is non-stationary, it offers formidable challenges from the viewpoint of a data analyst. It is characterized by significant trial-to-trial and subject-to-subject variability. Often signals are high-dimensional with only relatively few samples available for fitting models to the data and finally the signal-to-noise ratio is highly unfavorable, In fact, it typically is even ill-defined what signal and, respectively, what noise are (cf. Blankertz et al., 2006c, Dornhege et al.,

BCI control based on movement imagery without subject training

So far the BBCI has mainly studied two paradigms: (a) the discriminability of pre-movement potentials in self-paced executed movements Blankertz et al., 2003, Blankertz et al., 2006a, Blankertz et al., 2006c, where it can be shown that high information transfer rates can be obtained from single-trial classification of fast-paced motor commands and (b) motor imagery (Blankertz et al., 2007a). Both paradigms do not require subject training. Due to space limitations only the results of the second

Applications of BBCI

The machine learning tools that have been developed for the BBCI system enable us to analyze EEG signals in real-time and on a single-trial basis. As a prerequisite the algorithms generally have to be calibrated based on examples of the specific brain patterns of an individual. In the following, two applications of single-trial analysis are presented. First, Hex-o-Spell, a text entry system for communicating, which is a classical BCI feedback application. Second, the online monitoring of

Concluding discussion

Analyzing EEG signals robustly and in real-time, despite their high variability, and the obviously noisy signal characteristics, is a major challenge. Recently modern machine learning and adaptive signal processing techniques have been able to successfully contribute to this exciting field. In particular, it has become possible to analyze EEG on a single-trial basis for two fields of applications: (a) new insights into general mental state monitoring can be gained and (b) brain–computer

Acknowledgments

We gratefully acknowledge financial support by the Bundesministerium für Bildung und Forschung (BMBF), 01IBE01A/B and 01IGQ0414, by the Deutsche Forschungsgemeinschaft (DFG), FOR 375/B1, and by the IST Programme of the European Community, under the PASCAL Network of Excellence, IST-2002-506778.

The authors would like to thank their co-authors for granting the use of materials published in Blankertz et al. (2006b).

References (45)

E.A. Curran et al.
Learning to control brain activity: a review of the production and control of EEG components for driving brain–computer interface (BCI) systems
Brain Cogn
(2003)
T. Elbert et al.
Biofeedback of slow cortical potentials.I.
Electroencephalogr Clin Neurophysiol
(1980)
R. Krepki et al.
Berlin brain–computer interface—the hci communication channel for discovery.
Int J Human Comput Studies
(2007)
D.J. McFarland et al.
Brain–computer interface (BCI) operation: optimizing information transfer rates
Biol Psychol
(2003)
G. Pfurtscheller et al.
Event-related EEG/MEG synchronization and desynchronization: basic principles
Clin Neurophysiol
(1999)
J.R. Wolpaw et al.
Brain–computer interfaces for communication and control
Clin Neurophysiol
(2002)
K. Bennett et al.
Robust linear programming discrimination of two linearly inseparable sets
Optim Methods Software
(1992)
N. Birbaumer et al.
The though translation device (TTD) for completly paralyzed patients
IEEE Trans Rehab Eng
(2000)
B. Blankertz et al.
Classifying single trial EEG: towards brain–computer interfacing.
Blankertz B, Dornhege G, Krauledat M, Müller K-R, Curio G. The non-invasive Berlin Brain–Computer Interface: fast...

Blankertz B, Dornhege G, Krauledat M, Müller K-R, Curio G. The Berlin brain–computer interface: Report from the...

B. Blankertz et al.

The Berlin brain–computer interface: EEG-based communication without subject training

IEEE Trans Neural Syst Rehab Eng

(2006)

B. Blankertz et al.

The Berlin brain–computer interface presents the novel mental typewriter Hex-o-Spell

B. Blankertz et al.

The Berlin brain–computer interface: machine learning based detection of user specific brain states

J Univ Comput Sci

(2006)

B. Blankertz et al.

Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG-analysis

IEEE Trans Neural Syst Rehab Eng

(2003)

B. Blankertz et al.

A note on brain actuated spelling with the Berlin brain–computer interface

J.M. Carmena et al.

Learning to control a brain–machine interface for reaching and grasping by primates

PLoS Biol

(2003)

Dornhege G. Increasing information transfer rates for brain–computer interfacing. Ph.D. Thesis. University of Potsdam,...

G. Dornhege et al.

Combining features for BCI

G. Dornhege et al.

Boosting bit rates in non-invasive EEG single-trial classifications by feature combination and multi-class paradigms

IEEE Trans Biomed Eng

(2004)

G. Dornhege et al.

Combined optimization of spatial and temporal filters for improving brain–computer interfacing

IEEE Trans Biomed Eng

(2006)

Dornhege G, del R Millán J, Hinterberger T, McFarland D, Müller K-R, editors. Towards brain-computer interfacing. MIT...

Cited by (384)

Single-trial stimuli classification from detected P300 for augmented Brain–Computer Interface: A deep learning approach
2022, Machine Learning with Applications
The purpose of advanced Brain–Computer Interfaces (BCIs) is to connect the human brain with an external device without using the muscular system. To do this, they must effectively process mental activity and infer information on the users’ intentions and directives. This work proposes a novel and explainable BCI system capable of recognizing P300 deflection in single-trial EEGs with higher accuracy compared to the literature gold standard. Moreover, the proposed deep-learning approach allows us to go beyond the mere P300 detection, which is, to our best knowledge, the current state of the art. Indeed, we first identify the P300-related signal in the single-trial EEG signal, and then, we further discriminate the ERPs associated to the detected P300 between visual and auditory stimuli-related. To do this, we employ a CNN–LSTM neural network, which manages a 3D data representation of the acquired EEG signals. The performance of the approach is tested on experiments carried out on 22 subjects, revealing a 82.4% F1-score in P300 identification and 82.4% discriminating between visual and auditory stimuli. The employed algorithmic procedure also reports the most relevant each EEG channels in determining the predictions, adding interpretability to the proposed AI-based tools. These results pave the way for more sophisticated BCIs, capable of extending the set of available actions for the patients. The project was pre-approved by the Research Assessment Committee of the Department of Psychology (CRIP) for minimal risk projects, under the aegis of the Ethical Committee of University of Milano-Bicocca, on May 27th, 2019, protocol number RM-2019-193.
LSTM network as a screening tool to detect moderate traumatic brain injury from resting-state electroencephalogram
2022, Expert Systems with Applications
Fraud and abuse in insurance and health care claims have received major attention as they can cause increasing losses of revenue. Processing health care claims requires extensive workloads because the staffs have to investigate the legibility of the report. For the investigation of brain injury claims, the related insurance company may request medical images of the brain from the hospital and subsequently get opinions from the medical staff. Conventionally, computed tomography (CT) or magnetic resonance imaging (MRI) is utilized for this purpose. However, to perform a CT scan or an MRI scan for every patient that requested medical claims is impractical due to the limited resources. Thus, we proposed a screening approach that uses the resting-state electroencephalogram (EEG) recordings as the input to a long short-term memory (LSTM) network. This LSTM architecture can classify the resting state EEG into two classes, which are either as a moderate traumatic brain injury (TBI) patient or a healthy person. Experimental results show that the proposed approach is able to outperform two similar recent works by achieving a classification accuracy of 74.33%.
Facial Emotions Are Accurately Encoded in the Neural Signal of Those With Autism Spectrum Disorder: A Deep Learning Approach
2022, Biological Psychiatry: Cognitive Neuroscience and Neuroimaging
Individuals with autism spectrum disorder (ASD) exhibit frequent behavioral deficits in facial emotion recognition (FER). It remains unknown whether these deficits arise because facial emotion information is not encoded in their neural signal or because it is encodes but fails to translate to FER behavior (deployment). This distinction has functional implications, including constraining when differences in social information processing occur in ASD, and guiding interventions (i.e., developing prosthetic FER vs. reinforcing existing skills).
We utilized a discriminative and contemporary machine learning approach—deep convolutional neural networks—to classify facial emotions viewed by individuals with and without ASD (N = 88) from concurrently recorded electroencephalography signals.
The convolutional neural network classified facial emotions with high accuracy for both ASD and non-ASD groups, even though individuals with ASD performed more poorly on the concurrent FER task. In fact, convolutional neural network accuracy was greater in the ASD group and was not related to behavioral performance. This pattern of results replicated across three independent participant samples. Moreover, feature importance analyses suggested that a late temporal window of neural activity (1000–1500 ms) may be uniquely important in facial emotion classification for individuals with ASD.
Our results reveal for the first time that facial emotion information is encoded in the neural signal of individuals with (and without) ASD. Thus, observed difficulties in behavioral FER associated with ASD likely arise from difficulties in decoding or deployment of facial emotion information within the neural signal. Interventions should focus on capitalizing on this intact encoding rather than promoting compensation or FER prostheses.
Exploring Convolutional Neural Network Architectures for EEG Feature Extraction
2024, Sensors
Modeling Complex EEG Data Distribution on the Riemannian Manifold Toward Outlier Detection and Multimodal Classification
2024, IEEE Transactions on Biomedical Engineering
A proposal of Knowledge-Based Feedback for BCI-training to control EEG
2023, AIP Conference Proceedings

View all citing articles on Scopus

View full text

Machine learning for real-time single-trial EEG-analysis: From brain–computer interfacing to mental state monitoring

Abstract

Introduction

Section snippets

Machine learning for BCI

BCI control based on movement imagery without subject training

Applications of BBCI

Concluding discussion

Acknowledgments

Brain Cogn

Electroencephalogr Clin Neurophysiol

Int J Human Comput Studies

Biol Psychol

Clin Neurophysiol

Clin Neurophysiol

Robust linear programming discrimination of two linearly inseparable sets

Optim Methods Software

The though translation device (TTD) for completly paralyzed patients

IEEE Trans Rehab Eng

Classifying single trial EEG: towards brain–computer interfacing.

The Berlin brain–computer interface: EEG-based communication without subject training

IEEE Trans Neural Syst Rehab Eng

The Berlin brain–computer interface presents the novel mental typewriter Hex-o-Spell

The Berlin brain–computer interface: machine learning based detection of user specific brain states

J Univ Comput Sci

Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG-analysis

IEEE Trans Neural Syst Rehab Eng

A note on brain actuated spelling with the Berlin brain–computer interface

Learning to control a brain–machine interface for reaching and grasping by primates

PLoS Biol

Combining features for BCI

Boosting bit rates in non-invasive EEG single-trial classifications by feature combination and multi-class paradigms

IEEE Trans Biomed Eng

Combined optimization of spatial and temporal filters for improving brain–computer interfacing

IEEE Trans Biomed Eng