HistGen: Histopathology Report Generation via Local-Global Feature
  Encoding and Cross-modal Context Interaction
Zhengrui Guo Jiabo Ma Yingxue Xu Yihui Wang Liansheng Wang Hao Chen
Histopathology serves as the gold standard in cancer diagnosis, with clinical
reports being vital in interpreting and understanding this process, guiding
cancer treatment and patient care. The automation of histopathology report
generation with deep learning stands to significantly enhance clinical
efficiency and lessen the labor-intensive, time-consuming burden on
pathologists in report writing. In pursuit of this advancement, we introduce
HistGen, a multiple instance learning-empowered framework for histopathology
report generation together with the first benchmark dataset for evaluation.
Inspired by diagnostic and report-writing workflows, HistGen features two
delicately designed modules, aiming to boost report generation by aligning
whole slide images (WSIs) and diagnostic reports from local and global
granularity. To achieve this, a local-global hierarchical encoder is developed
for efficient visual feature aggregation from a region-to-slide perspective.
Meanwhile, a cross-modal context module is proposed to explicitly facilitate
alignment and interaction between distinct modalities, effectively bridging the
gap between the extensive visual sequences of WSIs and corresponding highly
summarized reports. Experimental results on WSI report generation show the
proposed model outperforms state-of-the-art (SOTA) models by a large margin.
Moreover, the results of fine-tuning our model on cancer subtyping and survival
analysis tasks further demonstrate superior performance compared to SOTA
methods, showcasing strong transfer learning capability. Dataset, model
weights, and source code are available in
https://github.com/dddavid4real/HistGen.