Multimodal Pathology Image Search Between H&E Slides and Multiplexed
  Immunofluorescent Images
Amir Hajighasemi MD Jillur Rahman Saurav Mohammad S Nasr Jai Prakash Veerla Aarti Darji Parisa Boodaghi Malidarreh Michael Robben Helen H Shang Jacob M Luber
We present an approach for multimodal pathology image search, using dynamic
time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed
into a ranked choice voting scheme to retrieve multiplexed immunofluorescent
imaging (mIF) that is most similar to a query H&E slide. Through training the
VAE and applying DTW, we align and compare mIF and H&E slides. Our method
improves differential diagnosis and therapeutic decisions by integrating
morphological H&E data with immunophenotyping from mIF, providing clinicians a
rich perspective of disease states. This facilitates an understanding of the
spatial relationships in tissue samples and could revolutionize the diagnostic
process, enhancing precision and enabling personalized therapy selection. Our
technique demonstrates feasibility using colorectal cancer and healthy tonsil
samples. An exhaustive ablation study was conducted on a search engine designed
to explore the correlation between multiplexed Immunofluorescence (mIF) and
Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map
these distinct modalities into a unified vector space. Despite extreme class
imbalance, the system demonstrated robustness and utility by returning similar
results across various data features, which suggests potential for future use
in multimodal histopathology data analysis.