One Model to Rule them All: Towards Universal Segmentation for Medical
  Images with Text Prompts
Ziheng Zhao Yao Zhang Chaoyi Wu Xiaoman Zhang Ya Zhang Yanfeng Wang Weidi Xie
In this study, we focus on building up a model that can Segment Anything in
medical scenarios, driven by Text prompts, termed as SAT. Our main
contributions are three folds: (i) on data construction, we combine multiple
knowledge sources to construct a multi-modal medical knowledge tree; Then we
build up a large-scale segmentation dataset for training, by collecting over
11K 3D medical image scans from 31 segmentation datasets with careful
standardization on both visual scans and label space; (ii) on model training,
we formulate a universal segmentation model, that can be prompted by inputting
medical terminologies in text form. We present a knowledge-enhanced
representation learning framework, and a series of strategies for effectively
training on the combination of a large number of datasets; (iii) on model
evaluation, we train a SAT-Nano with only 107M parameters, to segment 31
different segmentation datasets with text prompt, resulting in 362 categories.
We thoroughly evaluate the model from three aspects: averaged by body regions,
averaged by classes, and average by datasets, demonstrating comparable
performance to 36 specialist nnUNets, i.e., we train nnUNet models on each
dataset/subset, resulting in 36 nnUNets with around 1000M parameters for the 31
datasets. We will release all the codes, and models used in this report, i.e.,
SAT-Nano. Moreover, we will offer SAT-Ultra in the near future, which is
trained with model of larger size, on more diverse datasets. Webpage URL:
https://zhaoziheng.github.io/MedUniSeg.