Learning Nuclei Representations with Masked Image Modelling
Piotr Wójcik Hussein Naji Adrian Simon Reinhard Büttner Katarzyna Bożek
Masked image modelling (MIM) is a powerful self-supervised representation
learning paradigm, whose potential has not been widely demonstrated in medical
image analysis. In this work, we show the capacity of MIM to capture rich
semantic representations of Haemotoxylin & Eosin (H&E)-stained images at the
nuclear level. Inspired by Bidirectional Encoder representation from Image
Transformers (BEiT), we split the images into smaller patches and generate
corresponding discrete visual tokens. In addition to the regular grid-based
patches, typically used in visual Transformers, we introduce patches of
individual cell nuclei. We propose positional encoding of the irregular
distribution of these structures within an image. We pre-train the model in a
self-supervised manner on H&E-stained whole-slide images of diffuse large
B-cell lymphoma, where cell nuclei have been segmented. The pre-training
objective is to recover the original discrete visual tokens of the masked image
on the one hand, and to reconstruct the visual tokens of the masked object
instances on the other. Coupling these two pre-training tasks allows us to
build powerful, context-aware representations of nuclei. Our model generalizes
well and can be fine-tuned on downstream classification tasks, achieving
improved cell classification accuracy on PanNuke dataset by more than 5%
compared to current instance segmentation methods.