Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion
Meng Zheng Benjamin Planche Xuan Gong Fan Yang Terrence Chen Ziyan Wu
3D patient body modeling is critical to the success of automated patient
positioning for smart medical scanning and operating rooms. Existing CNN-based
end-to-end patient modeling solutions typically require a) customized network
designs demanding large amount of relevant training data, covering extensive
realistic clinical scenarios (e.g., patient covered by sheets), which leads to
suboptimal generalizability in practical deployment, b) expensive 3D human
model annotations, i.e., requiring huge amount of manual effort, resulting in
systems that scale poorly. To address these issues, we propose a generic
modularized 3D patient modeling method consists of (a) a multi-modal keypoint
detection module with attentive fusion for 2D patient joint localization, to
learn complementary cross-modality patient body information, leading to
improved keypoint localization robustness and generalizability in a wide
variety of imaging (e.g., CT, MRI etc.) and clinical scenarios (e.g., heavy
occlusions); and (b) a self-supervised 3D mesh regression module which does not
require expensive 3D mesh parameter annotations to train, bringing immediate
cost benefits for clinical deployment. We demonstrate the efficacy of the
proposed method by extensive patient positioning experiments on both public and
clinical data. Our evaluation results achieve superior patient positioning
performance across various imaging modalities in real clinical scenarios.