A review of feature extraction and classification algorithms for image RSVP-based BCI
---
paper_title: A framework for rapid visual image search using single-trial brain evoked responses

paper_content:

We report the design and performance of a brain computer interface for single-trial detection of viewed images based on human dynamic brain response signatures in 32-channel electroencephalography (EEG) acquired during a rapid serial visual presentation. The system explores the feasibility of speeding up image analysis by tapping into split-second perceptual judgments of humans. We present an incremental learning system with less memory storage and computational cost for single-trial event-related potential (ERP) detection, which is trained using cross-session data. We demonstrate the efficacy of the method on the task of target image detection. We apply linear and nonlinear support vector machines (SVMs) and a linear logistic classifier (LLC) for single-trial ERP detection using data collected from image analysts and naive subjects. For our data the detection performance of the nonlinear SVM is better than the linear SVM and the LLC. We also show that our ERP-based target detection system is five-fold faster than the traditional image viewing paradigm.
---
paper_title: Brain Activity-Based Image Classification From Rapid Serial Visual Presentation

paper_content:

We report the design and performance of a brain-computer interface (BCI) system for real-time single-trial binary classification of viewed images based on participant-specific dynamic brain response signatures in high-density (128-channel) electroencephalographic (EEG) data acquired during a rapid serial visual presentation (RSVP) task. Image clips were selected from a broad area image and presented in rapid succession (12/s) in 4.1-s bursts. Participants indicated by subsequent button press whether or not each burst of images included a target airplane feature. Image clip creation and search path selection were designed to maximize user comfort and maintain user awareness of spatial context. Independent component analysis (ICA) was used to extract a set of independent source time-courses and their minimally-redundant low-dimensional informative features in the time and time-frequency amplitude domains from 128-channel EEG data recorded during clip burst presentations in a training session. The naive Bayes fusion of two Fisher discriminant classifiers, computed from the 100 most discriminative time and time-frequency features, respectively, was used to estimate the likelihood that each clip contained a target feature. This estimator was applied online in a subsequent test session. Across eight training/test session pairs from seven participants, median area under the receiver operator characteristic curve, by tenfold cross validation, was 0.97 for within-session and 0.87 for between-session estimates, and was nearly as high (0.83) for targets presented in bursts that participants mistakenly reported to include no target features.
---
paper_title: Updating P300: An integrative theory of P3a and P3b

paper_content:

The empirical and theoretical development of the P300 event-related brain potential (ERP) is reviewed by considering factors that contribute to its amplitude, latency, and general characteristics. The neuropsychological origins of the P3a and P3b subcomponents are detailed, and how target/standard discrimination difficulty modulates scalp topography is discussed. The neural loci of P3a and P3b generation are outlined, and a cognitive model is proffered: P3a originates from stimulus-driven frontal attention mechanisms during task processing, whereas P3b originates from temporal-parietal activity associated with attention and appears related to subsequent memory processing. Neurotransmitter actions associating P3a to frontal/dopaminergic and P3b to parietal/norepinephrine pathways are highlighted. Neuroinhibition is suggested as an overarching theoretical mechanism for P300, which is elicited when stimulus detection engages memory operations.
---
paper_title: A review of classification algorithms for EEG-based brain-computer interfaces

paper_content:

In this paper we review classification algorithms used to design brain–computer interface (BCI) systems based on electroencephalography (EEG). We briefly present the commonly employed algorithms and describe their critical properties. Based on the literature, we compare them in terms of performance and provide guidelines to choose the suitable classification algorithm(s) for a specific BCI.
---
paper_title: A Tutorial on Principal Component Analysis

paper_content:

Principal component analysis (PCA) is a mainstay of modern data analysis - a black box that is widely used but (sometimes) poorly understood. The goal of this paper is to dispel the magic behind this black box. This manuscript focuses on building a solid intuition for how and why principal component analysis works. This manuscript crystallizes this knowledge by deriving from simple intuitions, the mathematics behind PCA. This tutorial does not shy away from explaining the ideas informally, nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers of all levels will be able to gain a better understanding of PCA as well as the when, the how and the why of applying this technique.
---
paper_title: High-throughput image search via single-trial event detection in a rapid serial visual presentation task

paper_content:

We describe a method, using linear discrimination, for detecting single-trial EEG signatures of object recognition events in a rapid serial visual presentation (RSVP) task. We record EEG using a high spatial density array (87 electrodes) during the rapid presentation (50-200 msec per image) of natural images. Subjects were instructed to release a button when they recognized a target image (an image with a person/people). Trials consisted of 100 images each, with a 50% chance of a single target being in a trial. Subject EEG was analyzed on a single-trial basis with an optimal spatial linear discriminator learned at multiple time windows after the presentation of an image. Linear discrimination enables the estimation of a forward model and thus allows for an approximate localization of the discriminating activity. Results show multiple loci for discriminating activity (e.g. motor and visual). Using these detected EEG signatures, we show that in many cases we can detect targets more accurately than the overt response (button release) and that such signatures can be used to prioritize images for high-throughput search.
---
paper_title: Cortically coupled computer vision for rapid image search

paper_content:

We describe a real-time electroencephalography (EEG)-based brain-computer interface system for triaging imagery presented using rapid serial visual presentation. A target image in a sequence of nontarget distractor images elicits in the EEG a stereotypical spatiotemporal response, which can be detected. A pattern classifier uses this response to reprioritize the image sequence, placing detected targets in the front of an image stack. We use single-trial analysis based on linear discrimination to recover spatial components that reflect differences in EEG activity evoked by target versus nontarget images. We find an optimal set of spatial weights for 59 EEG sensors within a sliding 50-ms time window. Using this simple classifier allows us to process EEG in real time. The detection accuracy across five subjects is on average 92%, i.e., in a sequence of 2500 images, resorting images based on detector output results in 92% of target images being moved from a random position in the sequence to one of the first 250 images (first 10% of the sequence). The approach leverages the highly robust and invariant object recognition capabilities of the human visual system, using single-trial EEG analysis to efficiently detect neural signatures correlated with the recognition event.
---
paper_title: An investigation of triggering approaches for the rapid serial visual presentation paradigm in brain computer interfacing

paper_content:

The rapid serial visual presentation (RSVP) paradigm is a method that can be used to extend the P300 based brain computer interface (BCI) approach to enable high throughput target image recognition applications. The method requires high temporal resolution and hence, generating reliable and accurate stimulus triggers is critical for high performance execution. The traditional RSVP paradigm is normally deployed on two computers where software triggers generated at runtime by the image presentation software on a presentation computer are acquired along with the raw electroencephalography (EEG) signals by a dedicated data acquisition system connected to a second computer. It is often assumed that the stimulus presentation timing as acquired via events arising in the stimulus presentation code is an accurate reflection of the physical stimulus presentation. This is not necessarily the case due to various and variable latencies that may arise in the overall system. This paper describes a study to investigate in a representative RSVP implementation whether or not software-derived stimulus timing can be considered an accurate reflection of the physical stimuli timing. To investigate this, we designed a simple circuit consisting of a light diode resistor comparator circuit (LDRCC) for recording the physical presentation of stimuli and which in turn generates what we refer to as hardware triggered events. These hardware-triggered events constitute a measure of ground truth and are captured along with the corresponding stimulus presentation command timing events for comparison. Our experimental results show that using software-derived timing only may introduce uncertainty as to the true presentation times of the stimuli and this uncertainty itself is highly variable at least in the representative implementation described here. For BCI protocols such as those utilizing RSVP, the uncertainly introduced will cause impairment of performance and we recommend the use of additional circuitry to capture the physical presentation of stimuli and that these hardware-derived triggers should instead constitute the event markers to be used for subsequent analysis of the EEG.
---
paper_title: A framework for rapid visual image search using single-trial brain evoked responses

paper_content:

We report the design and performance of a brain computer interface for single-trial detection of viewed images based on human dynamic brain response signatures in 32-channel electroencephalography (EEG) acquired during a rapid serial visual presentation. The system explores the feasibility of speeding up image analysis by tapping into split-second perceptual judgments of humans. We present an incremental learning system with less memory storage and computational cost for single-trial event-related potential (ERP) detection, which is trained using cross-session data. We demonstrate the efficacy of the method on the task of target image detection. We apply linear and nonlinear support vector machines (SVMs) and a linear logistic classifier (LLC) for single-trial ERP detection using data collected from image analysts and naive subjects. For our data the detection performance of the nonlinear SVM is better than the linear SVM and the LLC. We also show that our ERP-based target detection system is five-fold faster than the traditional image viewing paradigm.
---
paper_title: Common Spatio-Temporal Pattern for Single-Trial Detection of Event-Related Potential in Rapid Serial Visual Presentation Triage

paper_content:

Searching for target images in large volume imagery is a challenging problem and the rapid serial visual presentation (RSVP) triage is potentially a promising solution to the problem. RSVP triage is essentially a cortically-coupled computer vision technique that relies on single-trial detection of event-related potentials (ERP). In RSVP triage, images are shown to a subject in a rapid serial sequence. When a target image is seen by the subject, unique ERP characterized by P300 are elicited. Thus, in RSVP triage, accurate detection of such distinct ERP allows for fast searching of target images in large volume imagery. The accuracy of the distinct ERP detection in RSVP triage depends on the feature extraction method, for which the common spatial pattern analysis (CSP) was used with limited success. This paper presents a novel feature extraction method, termed common spatio-temporal pattern (CSTP), which is critical for robust single-trial detection of ERP. Unlike the conventional CSP, whereby only spatial patterns of ERP are considered, the present proposed method exploits spatial and temporal patterns of ERP separately, providing complementary spatial and temporal features for high accurate single-trial ERP detection. Numerical study using data collected from 20 subjects in RSVP triage experiments demonstrates that the proposed method offers significant performance improvement over the conventional CSP method (corrected p -value <; 0.05, Pearson r=0.64) and other competing methods in the literature. This paper further shows that the main idea of CSTP can be easily applied to other methods.
---
paper_title: Brain Activity-Based Image Classification From Rapid Serial Visual Presentation

paper_content:

We report the design and performance of a brain-computer interface (BCI) system for real-time single-trial binary classification of viewed images based on participant-specific dynamic brain response signatures in high-density (128-channel) electroencephalographic (EEG) data acquired during a rapid serial visual presentation (RSVP) task. Image clips were selected from a broad area image and presented in rapid succession (12/s) in 4.1-s bursts. Participants indicated by subsequent button press whether or not each burst of images included a target airplane feature. Image clip creation and search path selection were designed to maximize user comfort and maintain user awareness of spatial context. Independent component analysis (ICA) was used to extract a set of independent source time-courses and their minimally-redundant low-dimensional informative features in the time and time-frequency amplitude domains from 128-channel EEG data recorded during clip burst presentations in a training session. The naive Bayes fusion of two Fisher discriminant classifiers, computed from the 100 most discriminative time and time-frequency features, respectively, was used to estimate the likelihood that each clip contained a target feature. This estimator was applied online in a subsequent test session. Across eight training/test session pairs from seven participants, median area under the receiver operator characteristic curve, by tenfold cross validation, was 0.97 for within-session and 0.87 for between-session estimates, and was nearly as high (0.83) for targets presented in bursts that participants mistakenly reported to include no target features.
---
paper_title: Updating P300: An integrative theory of P3a and P3b

paper_content:

The empirical and theoretical development of the P300 event-related brain potential (ERP) is reviewed by considering factors that contribute to its amplitude, latency, and general characteristics. The neuropsychological origins of the P3a and P3b subcomponents are detailed, and how target/standard discrimination difficulty modulates scalp topography is discussed. The neural loci of P3a and P3b generation are outlined, and a cognitive model is proffered: P3a originates from stimulus-driven frontal attention mechanisms during task processing, whereas P3b originates from temporal-parietal activity associated with attention and appears related to subsequent memory processing. Neurotransmitter actions associating P3a to frontal/dopaminergic and P3b to parietal/norepinephrine pathways are highlighted. Neuroinhibition is suggested as an overarching theoretical mechanism for P300, which is elicited when stimulus detection engages memory operations.
---
paper_title: The steady-state visual evoked potential in vision research: A review

paper_content:

Periodic visual stimulation and analysis of the resulting steady-state visual evoked potentials were first introduced over 80 years ago as a means to study visual sensation and perception. From the first single-channel recording of responses to modulated light to the present use of sophisticated digital displays composed of complex visual stimuli and high-density recording arrays, steady-state methods have been applied in a broad range of scientific and applied settings.The purpose of this article is to describe the fundamental stimulation paradigms for steady-state visual evoked potentials and to illustrate these principles through research findings across a range of applications in vision science.
---
paper_title: Towards a Cure for BCI Illiteracy

paper_content:

Brain–Computer Interfaces (BCIs) allow a user to control a computer application by brain activity as acquired, e.g., by EEG. One of the biggest challenges in BCI research is to understand and solve the problem of “BCI Illiteracy”, which is that BCI control does not work for a non-negligible portion of users (estimated 15 to 30%). Here, we investigate the illiteracy problem in BCI systems which are based on the modulation of sensorimotor rhythms. In this paper, a sophisticated adaptation scheme is presented which guides the user from an initial subject-independent classifier that operates on simple features to a subject-optimized state-of-the-art classifier within one session while the user interacts the whole time with the same feedback application. While initial runs use supervised adaptation methods for robust co-adaptive learning of user and machine, final runs use unsupervised adaptation and therefore provide an unbiased measure of BCI performance. Using this approach, which does not involve any offline calibration measurement, good performance was obtained by good BCI participants (also one novice) after 3–6 min of adaptation. More importantly, the use of machine learning techniques allowed users who were unable to achieve successful feedback before to gain significant control over the BCI system. In particular, one participant had no peak of the sensory motor idle rhythm in the beginning of the experiment, but could develop such peak during the course of the session (and use voluntary modulation of its amplitude to control the feedback application).
---
paper_title: Overview of NTCIR-13 NAILS task

paper_content:

In this paper we review the NTCIR-13 NAILS (Neurally Augmented Image Labelling Strategies) pilot task at NTCIR-13. We describe a first-of-its-kind RSVP (Rapid Serial Visual Presentation) - EEG (Electroencephalography) dataset released as part of the NTCIR-13 participation conference and the results of the participating organisations who benchmarked machine-learning strategies against each other using the provided unlabelled test data.
---
paper_title: The Foundations of Cost-Sensitive Learning

paper_content:

This paper revisits the problem of optimal learning and decision-making when different misclassification errors incur different penalties. We characterize precisely but intuitively when a cost matrix is reasonable, and we show how to avoid the mistake of defining a cost matrix that is economically incoherent. For the two-class case, we prove a theorem that shows how to change the proportion of negative examples in a training set in order to make optimal cost-sensitive classification decisions using a classifier learned by a standard non-costsensitive learning method. However, we then argue that changing the balance of negative and positive training examples has little effect on the classifiers produced by standard Bayesian and decision tree learning methods. Accordingly, the recommended way of applying one of these methods in a domain with differing misclassification costs is to learn a classifier from the training set as given, and then to compute optimal decisions explicitly using the probability estimates given by the classifier. 1 Making decisions based on a cost matrix Given a specification of costs for correct and incorrect predictions, an example should be predicted to have the class that leads to the lowest expected cost, where the expectation is computed using the conditional probability of each class given the example. Mathematically, let the entry in a cost matrix be the cost of predicting class when the true class is . If then the prediction is correct, while if the prediction is incorrect. The optimal prediction for an example is the class that minimizes ! (1) Costs are not necessarily monetary. A cost can also be a waste of time, or the severity of an illness, for example. For each , is a sum over the alternative possibilities for the true class of . In this framework, the role of a learning algorithm is to produce a classifier that for any example can estimate the probability " # of each class being the true class of . For an example , making the prediction means acting as if is the true class of . The essence of cost-sensitive decision-making is that it can be optimal to act as if one class is true even when some other class is more probable. For example, it can be rational not to approve a large credit card transaction even if the transaction is most likely legitimate. 1.1 Cost matrix properties A cost matrix always has the following structure when there are only two classes: actual negative actual positive predict negative $% $& ' )(!*+* $% -,. /(!*10 predict positive 2,& $& ' )(302* 2,& -,. /(30+0 Recent papers have followed the convention that cost matrix rows correspond to alternative predicted classes, while columns correspond to actual classes, i.e. row/column = / = predicted/actual. In our notation, the cost of a false positive is (302* while the cost of a false negative is (!*!0 . Conceptually, the cost of labeling an example incorrectly should always be greater than the cost of labeling it correctly. Mathematically, it should always be the case that ( 0 *54 ( *+* and ( *!064 ( 0 0 . We call these conditions the “reasonableness” conditions. Suppose that the first reasonableness condition is violated, so (!*+*879(302* but still (!*!0 4 (30+0 . In this case the optimal policy is to label all examples positive. Similarly, if (:0 * 4 (!*+* but (30 0;7 in a cost matrix if for all , = ?7 > @ A . In this case the cost of predicting > is no greater than the cost of predicting = , regardless of what the true class is. So it is optimal never to predict = . As a special case, the optimal prediction is always > if row > is dominated by all other rows in a cost matrix. The two reasonableness conditions for a two-class cost matrix imply that neither row in the matrix dominates the other. Given a cost matrix, the decisions that are optimal are unchanged if each entry in the matrix is multiplied by a positive constant. This scaling corresponds to changing the unit of account for costs. Similarly, the decisions that are optimal are unchanged B if a constant is added to each entry in the matrix. This shifting corresponds to changing the baseline away from which costs are measured. By scaling and shifting entries, any two-class cost matrix that satisfies the reasonableness conditions can be transformed into a simpler matrix that always leads to the same decisions:
---
paper_title: Combining EEG and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data

paper_content:

Eye movements introduce large artifacts to electroencephalographic recordings (EEG) and thus render data analysis difficult or even impossible. Trials contaminated by eye movement and blink artifacts have to be discarded, hence in standard EEG-paradigms subjects are required to fixate on the screen. To overcome this restriction, several correction methods including regression and blind source separation have been proposed. Yet, there is no automated standard procedure established. By simultaneously recording eye movements and 64-channel-EEG during a guided eye movement paradigm, we investigate and review the properties of eye movement artifacts, including corneo-retinal dipole changes, saccadic spike potentials and eyelid artifacts, and study their interrelations during different types of eye- and eyelid movements. In concordance with earlier studies our results confirm that these artifacts arise from different independent sources and that depending on electrode site, gaze direction, and choice of reference these sources contribute differently to the measured signal. We assess the respective implications for artifact correction methods and therefore compare the performance of two prominent approaches, namely linear regression and independent component analysis (ICA). We show and discuss that due to the independence of eye artifact sources, regression-based correction methods inevitably over- or under-correct individual artifact components, while ICA is in principle suited to address such mixtures of different types of artifacts. Finally, we propose an algorithm, which uses eye tracker information to objectively identify eye-artifact related ICA-components (ICs) in an automated manner. In the data presented here, the algorithm performed very similar to human experts when those were given both, the topographies of the ICs and their respective activations in a large amount of trials. Moreover it performed more reliable and almost twice as effective than human experts when those had to base their decision on IC topographies only. Furthermore, a receiver operating characteristic (ROC) analysis demonstrated an optimal balance of false positive and false negative at an area under curve (AUC) of more than 0.99. Removing the automatically detected ICs from the data resulted in removal or substantial suppression of ocular artifacts including microsaccadic spike potentials, while the relevant neural signal remained unaffected. In conclusion the present work aims at a better understanding of individual eye movement artifacts, their interrelations and the respective implications for eye artifact correction. Additionally, the proposed ICA-procedure provides a tool for optimized detection and correction of eye movement-related artifact components.
---
paper_title: P300 brain computer interface: current challenges and emerging trends

paper_content:

A brain-computer interface (BCI) enables communication without movement based on brain signals measured with electroencephalography (EEG). BCIs usually rely on one of three types of signals: the P300 and other components of the event-related potential (ERP), steady state visual evoked potential (SSVEP), or event related desynchronization (ERD). Although P300 BCIs were introduced over twenty years ago, the past few years have seen a strong increase in P300 BCI research. This closed-loop BCI approach relies on the P300 and other components of the ERP, based on an oddball paradigm presented to the subject. In this paper, we overview the current status of P300 BCI technology, and then discuss new directions: paradigms for eliciting P300s; signal processing methods; applications; and hybrid BCIs. We conclude that P300 BCIs are quite promising, as several emerging directions have not yet been fully explored and could lead to improvements in bit rate, reliability, usability, and flexibility.
---
paper_title: A framework for rapid visual image search using single-trial brain evoked responses

paper_content:

We report the design and performance of a brain computer interface for single-trial detection of viewed images based on human dynamic brain response signatures in 32-channel electroencephalography (EEG) acquired during a rapid serial visual presentation. The system explores the feasibility of speeding up image analysis by tapping into split-second perceptual judgments of humans. We present an incremental learning system with less memory storage and computational cost for single-trial event-related potential (ERP) detection, which is trained using cross-session data. We demonstrate the efficacy of the method on the task of target image detection. We apply linear and nonlinear support vector machines (SVMs) and a linear logistic classifier (LLC) for single-trial ERP detection using data collected from image analysts and naive subjects. For our data the detection performance of the nonlinear SVM is better than the linear SVM and the LLC. We also show that our ERP-based target detection system is five-fold faster than the traditional image viewing paradigm.
---
paper_title: Brain Activity-Based Image Classification From Rapid Serial Visual Presentation

paper_content:

We report the design and performance of a brain-computer interface (BCI) system for real-time single-trial binary classification of viewed images based on participant-specific dynamic brain response signatures in high-density (128-channel) electroencephalographic (EEG) data acquired during a rapid serial visual presentation (RSVP) task. Image clips were selected from a broad area image and presented in rapid succession (12/s) in 4.1-s bursts. Participants indicated by subsequent button press whether or not each burst of images included a target airplane feature. Image clip creation and search path selection were designed to maximize user comfort and maintain user awareness of spatial context. Independent component analysis (ICA) was used to extract a set of independent source time-courses and their minimally-redundant low-dimensional informative features in the time and time-frequency amplitude domains from 128-channel EEG data recorded during clip burst presentations in a training session. The naive Bayes fusion of two Fisher discriminant classifiers, computed from the 100 most discriminative time and time-frequency features, respectively, was used to estimate the likelihood that each clip contained a target feature. This estimator was applied online in a subsequent test session. Across eight training/test session pairs from seven participants, median area under the receiver operator characteristic curve, by tenfold cross validation, was 0.97 for within-session and 0.87 for between-session estimates, and was nearly as high (0.83) for targets presented in bursts that participants mistakenly reported to include no target features.
---
paper_title: The Balanced Accuracy and Its Posterior Distribution

paper_content:

Evaluating the performance of a classification algorithm critically requires a measure of the degree to which unseen examples have been identified with their correct class labels. In practice, generalizability is frequently estimated by averaging the accuracies obtained on individual cross-validation folds. This procedure, however, is problematic in two ways. First, it does not allow for the derivation of meaningful confidence intervals. Second, it leads to an optimistic estimate when a biased classifier is tested on an imbalanced dataset. We show that both problems can be overcome by replacing the conventional point estimate of accuracy by an estimate of the posterior distribution of the balanced accuracy.
---
paper_title: A Tutorial on Principal Component Analysis

paper_content:

Principal component analysis (PCA) is a mainstay of modern data analysis - a black box that is widely used but (sometimes) poorly understood. The goal of this paper is to dispel the magic behind this black box. This manuscript focuses on building a solid intuition for how and why principal component analysis works. This manuscript crystallizes this knowledge by deriving from simple intuitions, the mathematics behind PCA. This tutorial does not shy away from explaining the ideas informally, nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers of all levels will be able to gain a better understanding of PCA as well as the when, the how and the why of applying this technique.
---
paper_title: Cortically coupled computer vision for rapid image search

paper_content:

We describe a real-time electroencephalography (EEG)-based brain-computer interface system for triaging imagery presented using rapid serial visual presentation. A target image in a sequence of nontarget distractor images elicits in the EEG a stereotypical spatiotemporal response, which can be detected. A pattern classifier uses this response to reprioritize the image sequence, placing detected targets in the front of an image stack. We use single-trial analysis based on linear discrimination to recover spatial components that reflect differences in EEG activity evoked by target versus nontarget images. We find an optimal set of spatial weights for 59 EEG sensors within a sliding 50-ms time window. Using this simple classifier allows us to process EEG in real time. The detection accuracy across five subjects is on average 92%, i.e., in a sequence of 2500 images, resorting images based on detector output results in 92% of target images being moved from a random position in the sequence to one of the first 250 images (first 10% of the sequence). The approach leverages the highly robust and invariant object recognition capabilities of the human visual system, using single-trial EEG analysis to efficiently detect neural signatures correlated with the recognition event.
---
paper_title: Common Spatial Pattern Method for Channel Selelction in Motor Imagery Based Brain-computer Interface

paper_content:

A brain-computer interface (BCI) based on motor imagery (MI) translates the subject's motor intention into a control signal through classifying the electroencephalogram (EEG) patterns of different imagination tasks, e.g. hand and foot movements. Characteristic EEG spatial patterns make MI tasks substantially discriminable. Multi-channel EEGs are usually necessary for spatial pattern identification and therefore MI-based BCI is still in the stage of laboratory demonstration, to some extent, due to the need for constantly troublesome recording preparation. This paper presents a method for channel reduction in Mi-based BCI. Common spatial pattern (CSP) method was employed to analyze spatial patterns of imagined hand and foot movements. Significant channels were selected by searching the maximums of spatial pattern vectors in scalp mappings. A classification algorithm was developed by means of combining linear discriminant analysis towards event-related desynchronization (ERD) and readiness potential (RP). The classification accuracies with four optimal channels were 93.45% and 91.88% for two subjects
---
paper_title: Comparison of beamformers for EEG source signal reconstruction

paper_content:

Abstract Recently, several new beamformers have been introduced for reconstruction and localization of neural sources from EEG and MEG. Although studies have compared the accuracy of beamformers for localization of strong sources in the brain, a comparison of new and conventional beamformers for time-course reconstruction of a desired source has not been previously undertaken. In this study, 8 beamformers were examined with respect to several parameters, including variations in depth, orientation, magnitude, and frequency of the simulated source to determine their (i) effectiveness at time-course reconstruction of the sources, and (ii) stability of their performances with respect to the input changes. The spatial and directional pass-bands of the beamformers were estimated via simulated and real EEG sources to determine spatial resolution. White-noise spatial maps of the beamformers were calculated to show which beamformers have a location bias. Simulated EEG data were produced by projection via forward head modelling of simulated sources onto scalp electrodes, then superimposed on real background EEG. Real EEG was recorded from a patient with essential tremor and deep brain implanted electrodes. Gain – the ratio of SNR of the reconstructed time-course to the input SNR – was the primary measure of performance of the beamformers. Overall, minimum-variance beamformers had higher Gains and superior spatial resolution to those of the minimum-norm beamformers, although their performance was more sensitive to changes in magnitude, depth, and frequency of the simulated source. White-noise spatial maps showed that several, but not all, beamformers have an undesirable location bias.
---
paper_title: Optimizing Spatial filters for Robust EEG Single-Trial Analysis

paper_content:

Due to the volume conduction multichannel electroencephalogram (EEG) recordings give a rather blurred image of brain activity. Therefore spatial filters are extremely useful in single-trial analysis in order to improve the signal-to-noise ratio. There are powerful methods from machine learning and signal processing that permit the optimization of spatio-temporal filters for each subject in a data dependent fashion beyond the fixed filters based on the sensor geometry, e.g., Laplacians. Here we elucidate the theoretical background of the common spatial pattern (CSP) algorithm, a popular method in brain-computer interface (BCD research. Apart from reviewing several variants of the basic algorithm, we reveal tricks of the trade for achieving a powerful CSP performance, briefly elaborate on theoretical aspects of CSP, and demonstrate the application of CSP-type preprocessing in our studies of the Berlin BCI (BBCI) project.
---
paper_title: Common Spatio-Temporal Pattern for Single-Trial Detection of Event-Related Potential in Rapid Serial Visual Presentation Triage

paper_content:

Searching for target images in large volume imagery is a challenging problem and the rapid serial visual presentation (RSVP) triage is potentially a promising solution to the problem. RSVP triage is essentially a cortically-coupled computer vision technique that relies on single-trial detection of event-related potentials (ERP). In RSVP triage, images are shown to a subject in a rapid serial sequence. When a target image is seen by the subject, unique ERP characterized by P300 are elicited. Thus, in RSVP triage, accurate detection of such distinct ERP allows for fast searching of target images in large volume imagery. The accuracy of the distinct ERP detection in RSVP triage depends on the feature extraction method, for which the common spatial pattern analysis (CSP) was used with limited success. This paper presents a novel feature extraction method, termed common spatio-temporal pattern (CSTP), which is critical for robust single-trial detection of ERP. Unlike the conventional CSP, whereby only spatial patterns of ERP are considered, the present proposed method exploits spatial and temporal patterns of ERP separately, providing complementary spatial and temporal features for high accurate single-trial ERP detection. Numerical study using data collected from 20 subjects in RSVP triage experiments demonstrates that the proposed method offers significant performance improvement over the conventional CSP method (corrected p -value <; 0.05, Pearson r=0.64) and other competing methods in the literature. This paper further shows that the main idea of CSTP can be easily applied to other methods.
---
paper_title: xDAWN Algorithm to Enhance Evoked Potentials: Application to Brain–Computer Interface

paper_content:

A brain-computer interface (BCI) is a communication system that allows to control a computer or any other device thanks to the brain activity. The BCI described in this paper is based on the P300 speller BCI paradigm introduced by Farwell and Donchin. An unsupervised algorithm is proposed to enhance P300 evoked potentials by estimating spatial filters; the raw EEG signals are then projected into the estimated signal subspace. Data recorded on three subjects were used to evaluate the proposed method. The results, which are presented using a Bayesian linear discriminant analysis classifier, show that the proposed method is efficient and accurate.
---
paper_title: The LDA beamformer: Optimal estimation of ERP source time series using linear discriminant analysis.

paper_content:

We introduce a novel beamforming approach for estimating event-related potential (ERP) source time series based on regularized linear discriminant analysis (LDA). The optimization problems in LDA and linearly-constrained minimum-variance (LCMV) beamformers are formally equivalent. The approaches differ in that, in LCMV beamformers, the spatial patterns are derived from a source model, whereas in an LDA beamformer the spatial patterns are derived directly from the data (i.e., the ERP peak). Using a formal proof and MEG simulations, we show that the LDA beamformer is robust to correlated sources and offers a higher signal-to-noise ratio than the LCMV beamformer and PCA. As an application, we use EEG data from an oddball experiment to show how the LDA beamformer can be harnessed to detect single-trial ERP latencies and estimate connectivity between ERP sources. Concluding, the LDA beamformer optimally reconstructs ERP sources by maximizing the ERP signal-to-noise ratio. Hence, it is a highly suited tool for analyzing ERP source time series, particularly in EEG/MEG studies wherein a source model is not available.
---
paper_title: Single-Trial Classification of Event-Related Potentials in Rapid Serial Visual Presentation Tasks Using Supervised Spatial Filtering

paper_content:

Accurate detection of single-trial event-related potentials (ERPs) in the electroencephalogram (EEG) is a difficult problem that requires efficient signal processing and machine learning techniques. Supervised spatial filtering methods that enhance the discriminative information in EEG data are commonly used to improve single-trial ERP detection. We propose a convolutional neural network (CNN) with a layer dedicated to spatial filtering for the detection of ERPs and with training based on the maximization of the area under the receiver operating characteristic curve (AUC). The CNN is compared with three common classifiers: 1) Bayesian linear discriminant analysis; 2) multilayer perceptron (MLP); and 3) support vector machines. Prior to classification, the data were spatially filtered with xDAWN (for the maximization of the signal-to-signal-plus-noise ratio), common spatial pattern, or not spatially filtered. The 12 analytical techniques were tested on EEG data recorded in three rapid serial visual presentation experiments that required the observer to discriminate rare target stimuli from frequent nontarget stimuli. Classification performance discriminating targets from nontargets depended on both the spatial filtering method and the classifier. In addition, the nonlinear classifier MLP outperformed the linear methods. Finally, training based AUC maximization provided better performance than training based on the minimization of the mean square error. The results support the conclusion that the choice of the systems architecture is critical and both spatial filtering and classification must be considered together.
---
paper_title: xDAWN Algorithm to Enhance Evoked Potentials: Application to Brain–Computer Interface

paper_content:

A brain-computer interface (BCI) is a communication system that allows to control a computer or any other device thanks to the brain activity. The BCI described in this paper is based on the P300 speller BCI paradigm introduced by Farwell and Donchin. An unsupervised algorithm is proposed to enhance P300 evoked potentials by estimating spatial filters; the raw EEG signals are then projected into the estimated signal subspace. Data recorded on three subjects were used to evaluate the proposed method. The results, which are presented using a Bayesian linear discriminant analysis classifier, show that the proposed method is efficient and accurate.
---
paper_title: Optimum principal components for spatial filtering of EEG to detect imaginary movement by coherence

paper_content:

Several techniques have been used to improve the signal-to-noise ratio to increase the detection rate of Event Related Potentials (ERPs). This work investigates the application of spatial filtering based on principal component analysis (PCA) to detect ERP due to left-hand index finger movement imagination. The EEG signals were recorded of central derivations (C4, C2, Cz, C1 and C3), positioned according to 10–10 International System. The optimal spatial filter was found by using the first principal component and the ERP detection was obtained by magnitude squared coherence technique. The best detection rate, by using original signal (without filtering), was obtained at C2 derivation, with 54.73% for significance level of 5%. For the same significance level, the detection rate of the filtered signal was drastically improved to 96.84%. Results suggest that spatial filter by using PCA might be a very useful tool in assisting the ERP detection for movement imagination for applications on brain machine interface.
---
paper_title: Dimensionality Reduction and Channel Selection of Motor Imagery Electroencephalographic Data

paper_content:

The performance of spatial filters based on independent components analysis (ICA) was evaluated by employing principal component analysis (PCA) preprocessing for dimensional reduction. The PCA preprocessing was not found to be a suitable method that could retain motor imagery information in a smaller set of components. In contrast, 6 ICA components selected on the basis of visual inspection performed comparably (61.9%) to the full range of 22 components (63.9%). An automated selection of ICA components based on a variance criterion was also carried out. Only 8 components chosen this way performed better (63.1%) than visually selected components. A similar analysis on the reduced set of electrodes over mid-central and centroparietal regions of the brain revealed that common spatial patterns (CSPs) and Infomax were able to detect motor imagery activity with a satisfactory accuracy.
---
paper_title: Brain Activity-Based Image Classification From Rapid Serial Visual Presentation

paper_content:

We report the design and performance of a brain-computer interface (BCI) system for real-time single-trial binary classification of viewed images based on participant-specific dynamic brain response signatures in high-density (128-channel) electroencephalographic (EEG) data acquired during a rapid serial visual presentation (RSVP) task. Image clips were selected from a broad area image and presented in rapid succession (12/s) in 4.1-s bursts. Participants indicated by subsequent button press whether or not each burst of images included a target airplane feature. Image clip creation and search path selection were designed to maximize user comfort and maintain user awareness of spatial context. Independent component analysis (ICA) was used to extract a set of independent source time-courses and their minimally-redundant low-dimensional informative features in the time and time-frequency amplitude domains from 128-channel EEG data recorded during clip burst presentations in a training session. The naive Bayes fusion of two Fisher discriminant classifiers, computed from the 100 most discriminative time and time-frequency features, respectively, was used to estimate the likelihood that each clip contained a target feature. This estimator was applied online in a subsequent test session. Across eight training/test session pairs from seven participants, median area under the receiver operator characteristic curve, by tenfold cross validation, was 0.97 for within-session and 0.87 for between-session estimates, and was nearly as high (0.83) for targets presented in bursts that participants mistakenly reported to include no target features.
---
paper_title: Spatiotemporal Representations of Rapid Visual Target Detection: A Single-Trial EEG Classification Algorithm

paper_content:

Brain computer interface applications, developed for both healthy and clinical populations, critically depend on decoding brain activity in single trials. The goal of the present study was to detect distinctive spatiotemporal brain patterns within a set of event related responses. We introduce a novel classification algorithm, the spatially weighted FLD-PCA (SWFP), which is based on a two-step linear classification of event-related responses, using fisher linear discriminant (FLD) classifier and principal component analysis (PCA) for dimensionality reduction. As a benchmark algorithm, we consider the hierarchical discriminant component Analysis (HDCA), introduced by Parra, et al. 2007. We also consider a modified version of the HDCA, namely the hierarchical discriminant principal component analysis algorithm (HDPCA). We compare single-trial classification accuracies of all the three algorithms, each applied to detect target images within a rapid serial visual presentation (RSVP, 10 Hz) of images from five different object categories, based on single-trial brain responses. We find a systematic superiority of our classification algorithm in the tested paradigm. Additionally, HDPCA significantly increases classification accuracies compared to the HDCA. Finally, we show that presenting several repetitions of the same image exemplars improve accuracy, and thus may be important in cases where high accuracy is crucial.
---
paper_title: Removing Electroencephalographic Artifacts by Blind Source Separation

paper_content:

Eye movements, eye blinks, cardiac signals, muscle noise, and line noise present serious problems for electroencephalographic (EEG) interpretation and analysis when rejecting contaminated EEG segments results in an unacceptable data loss. Many methods have been proposed to remove artifacts from EEG recordings, especially those arising from eye movements and blinks. Often regression in the time or frequency domain is performed on parallel EEG and electrooculographic (EOG) recordings to derive parameters characterizing the appearance and spread of EOG artifacts in the EEG channels. Because EEG and ocular activity mix bidirectionally, regressing out eye artifacts inevitably involves subtracting relevant EEG signals from each record as well. Regression methods become even more problematic when a good regressing channel is not available for each artifact source, as in the case of muscle artifacts. Use of principal component analysis (PCA) has been proposed to remove eye artifacts from multichannel EEG. However, PCA cannot completely separate eye artifacts from brain signals, especially when they have comparable amplitudes. Here, we propose a new and generally applicable method for removing a wide variety of artifacts from EEG records based on blind source separation by independent component analysis (ICA). Our results on EEG data collected from normal and autistic subjects show that ICA can effectively detect, separate, and remove contamination from a wide variety of artifactual sources in EEG records with results comparing favorably with those obtained using regression and PCA methods. ICA can also be used to analyze blink-related brain activity.
---
paper_title: A Tutorial on Principal Component Analysis

paper_content:

Principal component analysis (PCA) is a mainstay of modern data analysis - a black box that is widely used but (sometimes) poorly understood. The goal of this paper is to dispel the magic behind this black box. This manuscript focuses on building a solid intuition for how and why principal component analysis works. This manuscript crystallizes this knowledge by deriving from simple intuitions, the mathematics behind PCA. This tutorial does not shy away from explaining the ideas informally, nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers of all levels will be able to gain a better understanding of PCA as well as the when, the how and the why of applying this technique.
---
paper_title: Characterization and Robust Classification of EEG Signal from Image RSVP Events with Independent Time-Frequency Features

paper_content:

This paper considers the problem of automatic characterization and detection of target images in a rapid serial visual presentation (RSVP) task based on EEG data. A novel method that aims to identify single-trial event-related potentials (ERPs) in time-frequency is proposed, and a robust classifier with feature clustering is developed to better utilize the correlated ERP features. The method is applied to EEG recordings of a RSVP experiment with multiple sessions and subjects. The results show that the target image events are mainly characterized by 3 distinct patterns in the time-frequency domain, i.e., a theta band (4.3 Hz) power boosting 300–700 ms after the target image onset, an alpha band (12 Hz) power boosting 500– 1000 ms after the stimulus onset, and a delta band (2 Hz) power boosting after 500 ms. The most discriminant timefrequency features are power boosting and are relatively consistent among multiple sessions and subjects. Since the original discriminant time-frequency features are highly correlated, we constructed the uncorrelated features using hierarchical clustering for better classification of target and non-target images. With feature clustering, performance (area under ROC) improved from 0.85 to 0.89 on within-session tests, and from 0.76 to 0.84 on cross-subject tests. The constructed uncorrelated features were more robust than the original discriminant features and corresponded to a number of local regions on the time-frequency plane.
---
paper_title: Analyzing Neural Time Series Data: Theory and Practice

paper_content:

This book offers a comprehensive guide to the theory and practice of analyzing electrical brain signals. It explains the conceptual, mathematical, and implementational (via Matlab programming) aspects of time-, time-frequency- and synchronization-based analyses of magnetoencephalography (MEG), electroencephalography (EEG), and local field potential (LFP) recordings from humans and nonhuman animals. It is the only book on the topic that covers both the theoretical background and the implementation in language that can be understood by readers without extensive formal training in mathematics, including cognitive scientists, neuroscientists, and psychologists. Readers who go through the book chapter by chapter and implement the examples in Matlab will develop an understanding of why and how analyses are performed, how to interpret results, what the methodological issues are, and how to perform single-subject-level and group-level analyses. Researchers who are familiar with using automated programs to perform advanced analyses will learn what happens when they click the "analyze now" button. The book provides sample data and downloadable Matlab code. Each of the 38 chapters covers one analysis topic, and these topics progress from simple to advanced. Most chapters conclude with exercises that further develop the material covered in the chapter. Many of the methods presented (including convolution, the Fourier transform, and Euler's formula) are fundamental and form the groundwork for other advanced data analysis methods. Readers who master the methods in the book will be well prepared to learn other approaches.
---
paper_title: High-throughput image search via single-trial event detection in a rapid serial visual presentation task

paper_content:

We describe a method, using linear discrimination, for detecting single-trial EEG signatures of object recognition events in a rapid serial visual presentation (RSVP) task. We record EEG using a high spatial density array (87 electrodes) during the rapid presentation (50-200 msec per image) of natural images. Subjects were instructed to release a button when they recognized a target image (an image with a person/people). Trials consisted of 100 images each, with a 50% chance of a single target being in a trial. Subject EEG was analyzed on a single-trial basis with an optimal spatial linear discriminator learned at multiple time windows after the presentation of an image. Linear discrimination enables the estimation of a forward model and thus allows for an approximate localization of the discriminating activity. Results show multiple loci for discriminating activity (e.g. motor and visual). Using these detected EEG signatures, we show that in many cases we can detect targets more accurately than the overt response (button release) and that such signatures can be used to prioritize images for high-throughput search.
---
paper_title: A framework for rapid visual image search using single-trial brain evoked responses

paper_content:

We report the design and performance of a brain computer interface for single-trial detection of viewed images based on human dynamic brain response signatures in 32-channel electroencephalography (EEG) acquired during a rapid serial visual presentation. The system explores the feasibility of speeding up image analysis by tapping into split-second perceptual judgments of humans. We present an incremental learning system with less memory storage and computational cost for single-trial event-related potential (ERP) detection, which is trained using cross-session data. We demonstrate the efficacy of the method on the task of target image detection. We apply linear and nonlinear support vector machines (SVMs) and a linear logistic classifier (LLC) for single-trial ERP detection using data collected from image analysts and naive subjects. For our data the detection performance of the nonlinear SVM is better than the linear SVM and the LLC. We also show that our ERP-based target detection system is five-fold faster than the traditional image viewing paradigm.
---
paper_title: Spatiotemporal Representations of Rapid Visual Target Detection: A Single-Trial EEG Classification Algorithm

paper_content:

Brain computer interface applications, developed for both healthy and clinical populations, critically depend on decoding brain activity in single trials. The goal of the present study was to detect distinctive spatiotemporal brain patterns within a set of event related responses. We introduce a novel classification algorithm, the spatially weighted FLD-PCA (SWFP), which is based on a two-step linear classification of event-related responses, using fisher linear discriminant (FLD) classifier and principal component analysis (PCA) for dimensionality reduction. As a benchmark algorithm, we consider the hierarchical discriminant component Analysis (HDCA), introduced by Parra, et al. 2007. We also consider a modified version of the HDCA, namely the hierarchical discriminant principal component analysis algorithm (HDPCA). We compare single-trial classification accuracies of all the three algorithms, each applied to detect target images within a rapid serial visual presentation (RSVP, 10 Hz) of images from five different object categories, based on single-trial brain responses. We find a systematic superiority of our classification algorithm in the tested paradigm. Additionally, HDPCA significantly increases classification accuracies compared to the HDCA. Finally, we show that presenting several repetitions of the same image exemplars improve accuracy, and thus may be important in cases where high accuracy is crucial.
---
paper_title: Generative Adversarial Networks

paper_content:

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.
---
paper_title: Regularized discriminant analysis

paper_content:

Abstract Linear and quadratic discriminant analysis are considered in the small-sample, high-dimensional setting. Alternatives to the usual maximum likelihood (plug-in) estimates for the covariance matrices are proposed. These alternatives are characterized by two parameters, the values of which are customized to individual situations by jointly minimizing a sample-based estimate of future misclassification risk. Computationally fast implementations are presented, and the efficacy of the approach is examined through simulation studies and application to data. These studies indicate that in many circumstances dramatic gains in classification accuracy can be achieved.
---
paper_title: Characterization and Robust Classification of EEG Signal from Image RSVP Events with Independent Time-Frequency Features

paper_content:

This paper considers the problem of automatic characterization and detection of target images in a rapid serial visual presentation (RSVP) task based on EEG data. A novel method that aims to identify single-trial event-related potentials (ERPs) in time-frequency is proposed, and a robust classifier with feature clustering is developed to better utilize the correlated ERP features. The method is applied to EEG recordings of a RSVP experiment with multiple sessions and subjects. The results show that the target image events are mainly characterized by 3 distinct patterns in the time-frequency domain, i.e., a theta band (4.3 Hz) power boosting 300–700 ms after the target image onset, an alpha band (12 Hz) power boosting 500– 1000 ms after the stimulus onset, and a delta band (2 Hz) power boosting after 500 ms. The most discriminant timefrequency features are power boosting and are relatively consistent among multiple sessions and subjects. Since the original discriminant time-frequency features are highly correlated, we constructed the uncorrelated features using hierarchical clustering for better classification of target and non-target images. With feature clustering, performance (area under ROC) improved from 0.85 to 0.89 on within-session tests, and from 0.76 to 0.84 on cross-subject tests. The constructed uncorrelated features were more robust than the original discriminant features and corresponded to a number of local regions on the time-frequency plane.
---
paper_title: Brain Activity-Based Image Classification From Rapid Serial Visual Presentation

paper_content:

We report the design and performance of a brain-computer interface (BCI) system for real-time single-trial binary classification of viewed images based on participant-specific dynamic brain response signatures in high-density (128-channel) electroencephalographic (EEG) data acquired during a rapid serial visual presentation (RSVP) task. Image clips were selected from a broad area image and presented in rapid succession (12/s) in 4.1-s bursts. Participants indicated by subsequent button press whether or not each burst of images included a target airplane feature. Image clip creation and search path selection were designed to maximize user comfort and maintain user awareness of spatial context. Independent component analysis (ICA) was used to extract a set of independent source time-courses and their minimally-redundant low-dimensional informative features in the time and time-frequency amplitude domains from 128-channel EEG data recorded during clip burst presentations in a training session. The naive Bayes fusion of two Fisher discriminant classifiers, computed from the 100 most discriminative time and time-frequency features, respectively, was used to estimate the likelihood that each clip contained a target feature. This estimator was applied online in a subsequent test session. Across eight training/test session pairs from seven participants, median area under the receiver operator characteristic curve, by tenfold cross validation, was 0.97 for within-session and 0.87 for between-session estimates, and was nearly as high (0.83) for targets presented in bursts that participants mistakenly reported to include no target features.
---
paper_title: An efficient P300-based brain-computer interface for disabled subjects

paper_content:

A brain-computer interface (BCI) is a communication system that translates brain-activity into commands for a computer or other devices. In other words, a BCI allows users to act on their environment by using only brain-activity, without using peripheral nerves and muscles. In this paper, we present a BCI that achieves high classification accuracy and high bitrates for both disabled and able-bodied subjects. The system is based on the P300 evoked potential and is tested with five severely disabled and four able-bodied subjects. For four of the disabled subjects classification accuracies of 100% are obtained. The bitrates obtained for the disabled subjects range between 10 and 25 bits/min. The effect of different electrode configurations and machine learning algorithms on classification accuracy is tested. Further factors that are possibly important for obtaining good classification accuracy in P300-based BCI systems for disabled subjects are discussed.
---
paper_title: Boosting linear logistic regression for single trial ERP detection in rapid serial visual presentation tasks.

paper_content:

In this paper, we employ the AdaBoost algorithm to the linear logistic regression model to detect encephalography (EEG) signatures, called evoked response potentials of visual recognition events in a single trial. In the experiments, a large amount of images were displayed at a very high presentation rate, named rapid serial visual presentation. The EEG was recorded using 32 electrodes during the rapid image presentation. Subjects were instructed to click the mouse when they recognize a target image. The results demonstrated that the boosting method improves the detection performance compared with the base classifier by approximately 3% as measured by area under the ROC curve.
---
paper_title: Optimising the Number of Channels in EEG-Augmented Image Search

paper_content:

Recent proof-of-concept research has appeared showing the applicability of Brain Computer Interface (BCI) technology in combination with the human visual system, to classify images. The basic premise here is that images that arouse a participant's attention generate a detectable response in their brainwaves, measurable using an electroencephalograph (EEG). When a participant is given a target class of images to search for, each image belonging to that target class presented within a stream of images should elicit a distinctly detectable neural response. Previous work in this domain has primarily focused on validating the technique on proof of concept image sets that demonstrate desired properties and on examining the capabilities of the technique at various image presentation speeds. In this paper we expand on this by examining the capability of the technique when using a reduced number of channels in the EEG, and its impact on the detection accuracy.
---
paper_title: Exploring EEG for Object Detection and Retrieval

paper_content:

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. Several experiments are performed using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We compare the feedback from the BCI and mouse-based interfaces in a subset of TRECVid images, finding that, when users have limited time to annotate the images, both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback can outperform mouse-based feedback.
---
paper_title: Common Spatio-Temporal Pattern for Single-Trial Detection of Event-Related Potential in Rapid Serial Visual Presentation Triage

paper_content:

Searching for target images in large volume imagery is a challenging problem and the rapid serial visual presentation (RSVP) triage is potentially a promising solution to the problem. RSVP triage is essentially a cortically-coupled computer vision technique that relies on single-trial detection of event-related potentials (ERP). In RSVP triage, images are shown to a subject in a rapid serial sequence. When a target image is seen by the subject, unique ERP characterized by P300 are elicited. Thus, in RSVP triage, accurate detection of such distinct ERP allows for fast searching of target images in large volume imagery. The accuracy of the distinct ERP detection in RSVP triage depends on the feature extraction method, for which the common spatial pattern analysis (CSP) was used with limited success. This paper presents a novel feature extraction method, termed common spatio-temporal pattern (CSTP), which is critical for robust single-trial detection of ERP. Unlike the conventional CSP, whereby only spatial patterns of ERP are considered, the present proposed method exploits spatial and temporal patterns of ERP separately, providing complementary spatial and temporal features for high accurate single-trial ERP detection. Numerical study using data collected from 20 subjects in RSVP triage experiments demonstrates that the proposed method offers significant performance improvement over the conventional CSP method (corrected p -value <; 0.05, Pearson r=0.64) and other competing methods in the literature. This paper further shows that the main idea of CSTP can be easily applied to other methods.
---
paper_title: An overview of gradient descent optimization algorithms

paper_content:

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.
---
paper_title: A review of classification algorithms for EEG-based brain-computer interfaces

paper_content:

In this paper we review classification algorithms used to design brain–computer interface (BCI) systems based on electroencephalography (EEG). We briefly present the commonly employed algorithms and describe their critical properties. Based on the literature, we compare them in terms of performance and provide guidelines to choose the suitable classification algorithm(s) for a specific BCI.
---
paper_title: Neural Networks For Pattern Recognition

paper_content:

Highlights of adaptive resonance theory classifying spatial patterns classifying temporal patterns multilayer networks and the use of attention representing synonyms specific architectures that use presynaptic inhibition. Appendices: feedforward circuits for normalization and noise suppression network equations used in the simulations of chapter 3 network equations used in the simulations of chapter 4.
---
paper_title: Statistical Pattern Recognition: A Review

paper_content:

The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.
---
paper_title: Multilayer perceptrons for the classification of brain computer interface data

paper_content:

Fast and simple classification methods for brain computer interfacing (BCI) signals are indispensable for the design of successful BCI applications. This paper presents a computationally simple algorithm to classify BCI data into left and right finger movements of the subjects. A two-class output multilayer perceptron (MLP) performs the classification. Our approach is attractive for providing an optimal combination of 1) computational efficiency, 2) classification accuracy (training: 100% and testing: 64%) and 3) minimal feature extraction (two channels out of a 28-channel EEG trial). The channels selected to be extracted (C3 and C4) not only greatly reduce dimensionality, but also refer to the central parts of the brain that decide left- right cognition, greatly enhancing the classification task. The results obtained are promising, and hold much potential for further investigation.
---
paper_title: The Segmentation of the Left Ventricle of the Heart From Ultrasound Data Using Deep Learning Architectures and Derivative-Based Search Methods

paper_content:

We present a new supervised learning model designed for the automatic segmentation of the left ventricle (LV) of the heart in ultrasound images. We address the following problems inherent to supervised learning models: 1) the need of a large set of training images; 2) robustness to imaging conditions not present in the training data; and 3) complex search process. The innovations of our approach reside in a formulation that decouples the rigid and nonrigid detections, deep learning methods that model the appearance of the LV, and efficient derivative-based search algorithms. The functionality of our approach is evaluated using a data set of diseased cases containing 400 annotated images (from 12 sequences) and another data set of normal cases comprising 80 annotated images (from two sequences), where both sets present long axis views of the LV. Using several error measures to compute the degree of similarity between the manual and automatic segmentations, we show that our method not only has high sensitivity and specificity but also presents variations with respect to a gold standard (computed from the manual annotations of two experts) within interuser variability on a subset of the diseased cases. We also compare the segmentations produced by our approach and by two state-of-the-art LV segmentation models on the data set of normal cases, and the results show that our approach produces segmentations that are comparable to these two approaches using only 20 training images and increasing the training set to 400 images causes our approach to be generally more accurate. Finally, we show that efficient search methods reduce up to tenfold the complexity of the method while still producing competitive segmentations. In the future, we plan to include a dynamical model to improve the performance of the algorithm, to use semisupervised learning methods to reduce even more the dependence on rich and large training sets, and to design a shape model less dependent on the training set.
---
paper_title: Recurrent convolutional neural network for object recognition

paper_content:

In recent years, the convolutional neural network (CNN) has achieved great success in many computer vision tasks. Partially inspired by neuroscience, CNN shares many properties with the visual system of the brain. A prominent difference is that CNN is typically a feed-forward architecture while in the visual system recurrent connections are abundant. Inspired by this fact, we propose a recurrent CNN (RCNN) for object recognition by incorporating recurrent connections into each convolutional layer. Though the input is static, the activities of RCNN units evolve over time so that the activity of each unit is modulated by the activities of its neighboring units. This property enhances the ability of the model to integrate the context information, which is important for object recognition. Like other recurrent neural networks, unfolding the RCNN through time can result in an arbitrarily deep network with a fixed number of parameters. Furthermore, the unfolded network has multiple paths, which can facilitate the learning process. The model is tested on four benchmark object recognition datasets: CIFAR-10, CIFAR-100, MNIST and SVHN. With fewer trainable parameters, RCNN outperforms the state-of-the-art models on all of these datasets. Increasing the number of parameters leads to even better performance. These results demonstrate the advantage of the recurrent structure over purely feed-forward structure for object recognition.
---
paper_title: Convolutional Neural Network Architectures for Matching Natural Language Sentences

paper_content:

Semantic matching is of central importance to many natural language tasks \cite{bordes2014semantic,RetrievalQA}. A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them. As a step toward this goal, we propose convolutional neural network models for matching two sentences, by adapting the convolutional strategy in vision and speech. The proposed models not only nicely represent the hierarchical structures of sentences with their layer-by-layer composition and pooling, but also capture the rich matching patterns at different levels. Our models are rather generic, requiring no prior knowledge on language, and can hence be applied to matching tasks of different nature and in different languages. The empirical study on a variety of matching tasks demonstrates the efficacy of the proposed model on a variety of matching tasks and its superiority to competitor models.
---
paper_title: EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces

paper_content:

Brain computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct characteristics of its expected EEG control signal, limiting its application to that specific signal. Convolutional Neural Networks (CNNs), which have been used in computer vision and speech recognition, have successfully been applied to EEG-based BCIs; however, they have mainly been applied to single BCI paradigms and thus it remains unclear how these architectures generalize to other paradigms. Here, we ask if we can design a single CNN architecture to accurately classify EEG signals from different BCI paradigms, while simultaneously being as compact as possible. In this work we introduce EEGNet, a compact convolutional network for EEG-based BCIs. We introduce the use of depthwise and separable convolutions to construct an EEG-specific model which encapsulates well-known EEG feature extraction concepts for BCI. We compare EEGNet to current state-of-the-art approaches across four BCI paradigms: P300 visual-evoked potentials, error-related negativity responses (ERN), movement-related cortical potentials (MRCP), and sensory motor rhythms (SMR). We show that EEGNet generalizes across paradigms better than the reference algorithms when only limited training data is available. We demonstrate three different approaches to visualize the contents of a trained EEGNet model to enable interpretation of the learned features. Our results suggest that EEGNet is robust enough to learn a wide variety of interpretable features over a range of BCI tasks, suggesting that the observed performances were not due to artifact or noise sources in the data.
---
paper_title: Single-Trial Classification of Event-Related Potentials in Rapid Serial Visual Presentation Tasks Using Supervised Spatial Filtering

paper_content:

Accurate detection of single-trial event-related potentials (ERPs) in the electroencephalogram (EEG) is a difficult problem that requires efficient signal processing and machine learning techniques. Supervised spatial filtering methods that enhance the discriminative information in EEG data are commonly used to improve single-trial ERP detection. We propose a convolutional neural network (CNN) with a layer dedicated to spatial filtering for the detection of ERPs and with training based on the maximization of the area under the receiver operating characteristic curve (AUC). The CNN is compared with three common classifiers: 1) Bayesian linear discriminant analysis; 2) multilayer perceptron (MLP); and 3) support vector machines. Prior to classification, the data were spatially filtered with xDAWN (for the maximization of the signal-to-signal-plus-noise ratio), common spatial pattern, or not spatially filtered. The 12 analytical techniques were tested on EEG data recorded in three rapid serial visual presentation experiments that required the observer to discriminate rare target stimuli from frequent nontarget stimuli. Classification performance discriminating targets from nontargets depended on both the spatial filtering method and the classifier. In addition, the nonlinear classifier MLP outperformed the linear methods. Finally, training based AUC maximization provided better performance than training based on the minimization of the mean square error. The results support the conclusion that the choice of the systems architecture is critical and both spatial filtering and classification must be considered together.
---
paper_title: Face Recognition : A Convolutional Neural Network Approach

paper_content:

We present a hybrid neural-network for human face recognition which compares favourably with other methods. The system combines local image sampling, a self-organizing map (SOM) neural network, and a convolutional neural network. The SOM provides a quantization of the image samples into a topological space where inputs that are nearby in the original space are also nearby in the output space, thereby providing dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides partial invariance to translation, rotation, scale, and deformation. The convolutional network extracts successively larger features in a hierarchical set of layers. We present results using the Karhunen-Loeve transform in place of the SOM, and a multilayer perceptron (MLP) in place of the convolutional network for comparison. We use a database of 400 images of 40 individuals which contains quite a high degree of variability in expression, pose, and facial details. We analyze the computational complexity and discuss how new classes could be added to the trained recognizer.
---
paper_title: Single-trial EEG RSVP classification using convolutional neural networks

paper_content:

Traditionally, Brain-Computer Interfaces (BCI) have been explored as a means to return function to paralyzed or otherwise debilitated individuals. An emerging use for BCIs is in human-autonomy sensor fusion where physiological data from healthy subjects is combined with machine-generated information to enhance the capabilities of artificial systems. While human-autonomy fusion of physiological data and computer vision have been shown to improve classification during visual search tasks, to date these approaches have relied on separately trained classification models for each modality. We aim to improve human-autonomy classification performance by developing a single framework that builds codependent models of human electroencephalograph (EEG) and image data to generate fused target estimates. As a first step, we developed a novel convolutional neural network (CNN) architecture and applied it to EEG recordings of subjects classifying target and non-target image presentations during a rapid serial visual presentation (RSVP) image triage task. The low signal-to-noise ratio (SNR) of EEG inherently limits the accuracy of single-trial classification and when combined with the high dimensionality of EEG recordings, extremely large training sets are needed to prevent overfitting and achieve accurate classification from raw EEG data. This paper explores a new deep CNN architecture for generalized multi-class, single-trial EEG classification across subjects. We compare classification performance from the generalized CNN architecture trained across all subjects to the individualized XDAWN, HDCA, and CSP neural classifiers which are trained and tested on single subjects. Preliminary results show that our CNN meets and slightly exceeds the performance of the other classifiers despite being trained across subjects.
---
paper_title: Classification of EEG during imagined mental tasks by forecasting with Elman Recurrent Neural Networks

paper_content:

The ability to classify EEG recorded while a subject performs varying imagined mental tasks may lay the foundation for building usable Brain-Computer Interfaces as well as improve the performance of EEG analysis software used in clinical settings. Although a number of research groups have produced EEG classifiers, these methods have not yet reached a level of performance that is acceptable for use in many practical applications. We assert that current approaches are limited by their ability to capture the temporal and spatial patterns contained within EEG. In order to address these problems, we propose a new generative technique for EEG classification that uses Elman Recurrent Neural Networks. EEG recorded while a subject performs one of several imagined mental tasks is first modeled by training a network to forecast the signal a single step ahead in time. We show that these models are able to forecast EEG well with an RMSE as low as 0.110. A separate model is then trained over EEG belonging to each class. Classification of previously unseen data is performed by applying each model and assigning the class label associated with the network that produced the lowest forecasting error. This approach is tested on EEG collected from two able-bodied subjects and one subject with a high-level spinal cord injury. Classification rates as high as 93.3% are achieved for a two-task problem with decisions made every second yielding a bitrate of 38.7 bits per minute.
---
paper_title: Analysis of EEG signals by implementing eigenvector methods/recurrent neural networks

paper_content:

The implementation of recurrent neural network (RNN) employing eigenvector methods is presented for classification of electroencephalogram (EEG) signals. In practical applications of pattern recognition, there are often diverse features extracted from raw data which needs recognizing. Because of the importance of making the right decision, the present work is carried out for searching better classification procedures for the EEG signals. Decision making was performed in two stages: feature extraction by eigenvector methods and classification using the classifiers trained on the extracted features. The aim of the study is classification of the EEG signals by the combination of eigenvector methods and the RNN. The present research demonstrated that the power levels of the power spectral density (PSD) estimates obtained by the eigenvector methods are the features which well represent the EEG signals and the RNN trained on these features achieved high classification accuracies.
---
paper_title: Recurrent neural network based language model

paper_content:

A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
---
paper_title: A Deep Learning method for classification of images RSVP events with EEG data

paper_content:

In this paper, we investigated Deep Learning (DL) for characterizing and detecting target images in an image rapid serial visual presentation (RSVP) task based on EEG data. We exploited DL technique with input feature clusters to handle high dimensional features related to time - frequency events. The method was applied to EEG recordings of a RSVP experiment with multiple sessions and subjects. For classification of target and non-target images, a deep belief net (DBN) classifier was based on the uncorrelated features, which was constructed from original correlated features using clustering method. The performance of the proposed DBN was tested for different combinations of hidden units and hidden layers on multiple subjects. The results of DBN were compared with cluster Linear Discriminant Analysis (cLDA) and Support vector machine (SVM) and DBN demonstrated better performance in all tested cases. There was an improvement of 10 - 25% for certain cases. We also demonstrated how DBN is used to characterize brain activities.
---
paper_title: A Tutorial on Principal Component Analysis

paper_content:

Principal component analysis (PCA) is a mainstay of modern data analysis - a black box that is widely used but (sometimes) poorly understood. The goal of this paper is to dispel the magic behind this black box. This manuscript focuses on building a solid intuition for how and why principal component analysis works. This manuscript crystallizes this knowledge by deriving from simple intuitions, the mathematics behind PCA. This tutorial does not shy away from explaining the ideas informally, nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers of all levels will be able to gain a better understanding of PCA as well as the when, the how and the why of applying this technique.
---

Title: A Review of Feature Extraction and Classification Algorithms for Image RSVP-Based BCI
Section 1: Introduction
Description 1: This section provides an overview of RSVP technology in BCI, its importance, and the necessity of a review dedicated to feature extraction and classification algorithms for RSVP-based BCI research.

Section 2: RSVP Experiment for EEG Data Acquisition
Description 2: This section describes the typical setup and procedure for data acquisition in RSVP-EEG experiments, including the hardware and software used.

Section 3: Brief Introduction to RSVP-EEG Pattern
Description 3: This section introduces the commonly observed EEG patterns in RSVP-BCI, with a focus on the P300 ERP and its significance.

Section 4: RSVP-EEG Data Preprocessing and Properties
Description 4: This section discusses the preprocessing steps required for RSVP-EEG data, including filtering, epoching, and addressing issues like low SNR and overlapping epochs.

Section 5: Performance Evaluation Metrics
Description 5: This section outlines various metrics to evaluate the performance of machine-learning models used in RSVP-based BCI, emphasizing ROC-AUC and balanced accuracy (BA).

Section 6: Feature Extraction Methods Used in RSVP-Based BCI Research
Description 6: This section reviews various feature extraction methods, including supervised and unsupervised spatial filtering techniques, and time-frequency representation specific to RSVP-EEG.

Section 7: Survey of Classifiers Used in RSVP-Based BCI Research
Description 7: This section surveys different classifiers, categorizing them into linear classifiers and neural networks, discussing their applicability and performance in RSVP-based BCI.

Section 8: Conclusion
Description 8: This section summarizes the key findings on feature extraction and classification methods in RSVP-based BCI, highlighting the advantages of different approaches and suggesting future research directions.