Optimizing a Transformer-based network for a deep learning seismic
  processing workflow
Randy Harsuko Tariq Alkhalifah
StorSeismic is a recently introduced model based on the Transformer to adapt
to various seismic processing tasks through its pretraining and fine-tuning
training strategy. In the original implementation, StorSeismic utilized a
sinusoidal positional encoding and a conventional self-attention mechanism,
both borrowed from the natural language processing (NLP) applications. For
seismic processing they admitted good results, but also hinted to limitations
in efficiency and expressiveness. We propose modifications to these two key
components, by utilizing relative positional encoding and low-rank attention
matrices as replacements to the vanilla ones. The proposed changes are tested
on processing tasks applied to a realistic Marmousi and offshore field data as
a sequential strategy, starting from denoising, direct arrival removal,
multiple attenuation, and finally root-mean-squared velocity ($V_{RMS}$)
prediction for normal moveout (NMO) correction. We observe faster pretraining
and competitive results on the fine-tuning tasks and, additionally, fewer
parameters to train compared to the vanilla model.