Knowledge-enhanced Visual-Language Pretraining for Computational
  Pathology
Xiao Zhou Xiaoman Zhang Chaoyi Wu Ya Zhang Weidi Xie Yanfeng Wang
In this paper, we consider the problem of visual representation learning for
computational pathology, by exploiting large-scale image-text pairs gathered
from public resources, along with the domain specific knowledge in pathology.
Specifically, we make the following contributions: (i) We curate a pathology
knowledge tree that consists of 50,470 informative attributes for 4,718
diseases requiring pathology diagnosis from 32 human tissues. To our knowledge,
this is the first comprehensive structured pathology knowledge base; (ii) We
develop a knowledge-enhanced visual-language pretraining approach, where we
first project pathology-specific knowledge into latent embedding space via
language model, and use it to guide the visual representation learning; (iii)
We conduct thorough experiments to validate the effectiveness of our proposed
components, demonstrating significant performance improvement on various
downstream tasks, including cross-modal retrieval, zero-shot classification on
pathology patches, and zero-shot tumor subtyping on whole slide images (WSIs).
All codes, models and the pathology knowledge tree will be released to the
research community