In-context learning enables multimodal large language models to classify
  cancer pathology images
Dyke Ferber Georg Wölflein Isabella C. Wiest Marta Ligero Srividhya Sainath Narmin Ghaffari Laleh Omar S. M. El Nahhas Gustav Müller-Franzes Dirk Jäger Daniel Truhn Jakob Nikolas Kather
Medical image classification requires labeled, task-specific datasets which
are used to train deep learning networks de novo, or to fine-tune foundation
models. However, this process is computationally and technically demanding. In
language processing, in-context learning provides an alternative, where models
learn from within prompts, bypassing the need for parameter updates. Yet,
in-context learning remains underexplored in medical image analysis. Here, we
systematically evaluate the model Generative Pretrained Transformer 4 with
Vision capabilities (GPT-4V) on cancer image processing with in-context
learning on three cancer histopathology tasks of high importance:
Classification of tissue subtypes in colorectal cancer, colon polyp subtyping
and breast tumor detection in lymph node sections. Our results show that
in-context learning is sufficient to match or even outperform specialized
neural networks trained for particular tasks, while only requiring a minimal
number of samples. In summary, this study demonstrates that large vision
language models trained on non-domain specific data can be applied out-of-the
box to solve medical image-processing tasks in histopathology. This
democratizes access of generalist AI models to medical experts without
technical background especially for areas where annotated data is scarce.