PEvoLM: Protein Sequence Evolutionary Information Language Model
Issar Arab
With the exponential increase of the protein sequence databases over time,
multiple-sequence alignment (MSA) methods, like PSI-BLAST, perform exhaustive
and time-consuming database search to retrieve evolutionary information. The
resulting position-specific scoring matrices (PSSMs) of such search engines
represent a crucial input to many machine learning (ML) models in the field of
bioinformatics and computational biology. A protein sequence is a collection of
contiguous tokens or characters called amino acids (AAs). The analogy to
natural language allowed us to exploit the recent advancements in the field of
Natural Language Processing (NLP) and therefore transfer NLP state-of-the-art
algorithms to bioinformatics. This research presents an Embedding Language
Model (ELMo), converting a protein sequence to a numerical vector
representation. While the original ELMo trained a 2-layer bidirectional Long
Short-Term Memory (LSTMs) network following a two-path architecture, one for
the forward and the second for the backward pass, by merging the idea of PSSMs
with the concept of transfer-learning, this work introduces a novel
bidirectional language model (bi-LM) with four times less free parameters and
using rather a single path for both passes. The model was trained not only on
predicting the next AA but also on the probability distribution of the next AA
derived from similar, yet different sequences as summarized in a PSSM,
simultaneously for multi-task learning, hence learning evolutionary information
of protein sequences as well. The network architecture and the pre-trained
model are made available as open source under the permissive MIT license on
GitHub at https://github.com/issararab/PEvoLM.