ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning
  of Heterogeneous Microscopy Images
Nicolas Bourriez Ihab Bendidi Ethan Cohen Gabriel Watkinson Maxime Sanchez Guillaume Bollot Auguste Genovesio
Unlike color photography images, which are consistently encoded into RGB
channels, biological images encompass various modalities, where the type of
microscopy and the meaning of each channel varies with each experiment.
Importantly, the number of channels can range from one to a dozen and their
correlation is often comparatively much lower than RGB, as each of them brings
specific information content. This aspect is largely overlooked by methods
designed out of the bioimage field, and current solutions mostly focus on
intra-channel spatial attention, often ignoring the relationship between
channels, yet crucial in most biological applications. Importantly, the
variable channel type and count prevent the projection of several experiments
to a unified representation for large scale pre-training. In this study, we
propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture
employing an Inter-Channel Attention mechanism on images with an arbitrary
number, order and type of channels. We also introduce IDRCell100k, a bioimage
dataset with a rich set of 79 experiments covering 7 microscope modalities,
with a multitude of channel types, and channel counts varying from 1 to 10 per
experiment. Our proposed architecture, trained in a self-supervised manner,
outperforms existing approaches in several biologically relevant downstream
tasks. Additionally, it can be used to bridge the gap for the first time
between assays with different microscopes, channel numbers or types by
embedding various image and experimental modalities into a unified biological
image representation. The latter should facilitate interdisciplinary studies
and pave the way for better adoption of deep learning in biological image-based
analyses. Code and Data to be released soon.