Efficient Representation Learning for Healthcare with
  Cross-Architectural Self-Supervision
Pranav Singh Jacopo Cirrone
In healthcare and biomedical applications, extreme computational requirements
pose a significant barrier to adopting representation learning. Representation
learning can enhance the performance of deep learning architectures by learning
useful priors from limited medical data. However, state-of-the-art
self-supervised techniques suffer from reduced performance when using smaller
batch sizes or shorter pretraining epochs, which are more practical in clinical
settings. We present Cross Architectural - Self Supervision (CASS) in response
to this challenge. This novel siamese self-supervised learning approach
synergistically leverages Transformer and Convolutional Neural Networks (CNN)
for efficient learning. Our empirical evaluation demonstrates that CASS-trained
CNNs and Transformers outperform existing self-supervised learning methods
across four diverse healthcare datasets. With only 1% labeled data for
finetuning, CASS achieves a 3.8% average improvement; with 10% labeled data, it
gains 5.9%; and with 100% labeled data, it reaches a remarkable 10.13%
enhancement. Notably, CASS reduces pretraining time by 69% compared to
state-of-the-art methods, making it more amenable to clinical implementation.
We also demonstrate that CASS is considerably more robust to variations in
batch size and pretraining epochs, making it a suitable candidate for machine
learning in healthcare applications.