HEMIT: H&E to Multiplex-immunohistochemistry Image Translation with
  Dual-Branch Pix2pix Generator
Chang Bian Beth Philips Tim Cootes Martin Fergie
Computational analysis of multiplexed immunofluorescence histology data is
emerging as an important method for understanding the tumour micro-environment
in cancer. This work presents HEMIT, a dataset designed for translating
Hematoxylin and Eosin (H&E) sections to multiplex-immunohistochemistry (mIHC)
images, featuring DAPI, CD3, and panCK markers. Distinctively, HEMIT's mIHC
images are multi-component and cellular-level aligned with H&E, enriching
supervised stain translation tasks. To our knowledge, HEMIT is the first
publicly available cellular-level aligned dataset that enables H&E to
multi-target mIHC image translation. This dataset provides the computer vision
community with a valuable resource to develop novel computational methods which
have the potential to gain new insights from H&E slide archives.
  We also propose a new dual-branch generator architecture, using residual
Convolutional Neural Networks (CNNs) and Swin Transformers which achieves
better translation outcomes than other popular algorithms. When evaluated on
HEMIT, it outperforms pix2pixHD, pix2pix, U-Net, and ResNet, achieving the
highest overall score on key metrics including the Structural Similarity Index
Measure (SSIM), Pearson correlation score (R), and Peak signal-to-noise Ratio
(PSNR). Additionally, downstream analysis has been used to further validate the
quality of the generated mIHC images. These results set a new benchmark in the
field of stain translation tasks.