Deep Representation Learning for Open Vocabulary
  Electroencephalography-to-Text Decoding
Hamza Amrani Daniela Micucci Paolo Napoletano
Previous research has demonstrated the potential of using pre-trained
language models for decoding open vocabulary Electroencephalography (EEG)
signals captured through a non-invasive Brain-Computer Interface (BCI).
However, the impact of embedding EEG signals in the context of language models
and the effect of subjectivity, remain unexplored, leading to uncertainty about
the best approach to enhance decoding performance. Additionally, current
evaluation metrics used to assess decoding effectiveness are predominantly
syntactic and do not provide insights into the comprehensibility of the decoded
output for human understanding. We present an end-to-end deep learning
framework for non-invasive brain recordings that brings modern representational
learning approaches to neuroscience. Our proposal introduces the following
innovations: 1) an end-to-end deep learning architecture for open vocabulary
EEG decoding, incorporating a subject-dependent representation learning module
for raw EEG encoding, a BART language model, and a GPT-4 sentence refinement
module; 2) a more comprehensive sentence-level evaluation metric based on the
BERTScore; 3) an ablation study that analyses the contributions of each module
within our proposal, providing valuable insights for future research. We
evaluate our approach on two publicly available datasets, ZuCo v1.0 and v2.0,
comprising EEG recordings of 30 subjects engaged in natural reading tasks. Our
model achieves a BLEU-1 score of 42.75%, a ROUGE-1-F of 33.28%, and a
BERTScore-F of 53.86%, outperforming the previous state-of-the-art methods by
3.38%, 8.43%, and 6.31%, respectively.