scHyena: Foundation Model for Full-Length Single-Cell RNA-Seq Analysis
  in Brain
Gyutaek Oh Baekgyu Choi Inkyung Jung Jong Chul Ye
Single-cell RNA sequencing (scRNA-seq) has made significant strides in
unraveling the intricate cellular diversity within complex tissues. This is
particularly critical in the brain, presenting a greater diversity of cell
types than other tissue types, to gain a deeper understanding of brain function
within various cellular contexts. However, analyzing scRNA-seq data remains a
challenge due to inherent measurement noise stemming from dropout events and
the limited utilization of extensive gene expression information. In this work,
we introduce scHyena, a foundation model designed to address these challenges
and enhance the accuracy of scRNA-seq analysis in the brain. Specifically,
inspired by the recent Hyena operator, we design a novel Transformer
architecture called singe-cell Hyena (scHyena) that is equipped with a linear
adaptor layer, the positional encoding via gene-embedding, and a
{bidirectional} Hyena operator. This enables us to process full-length
scRNA-seq data without losing any information from the raw data. In particular,
our model learns generalizable features of cells and genes through pre-training
scHyena using the full length of scRNA-seq data. We demonstrate the superior
performance of scHyena compared to other benchmark methods in downstream tasks,
including cell type classification and scRNA-seq imputation.