Annotated Biomedical Video Generation using Denoising Diffusion
  Probabilistic Models and Flow Fields
Rüveyda Yilmaz Dennis Eschweiler Johannes Stegmaier
The segmentation and tracking of living cells play a vital role within the
biomedical domain, particularly in cancer research, drug development, and
developmental biology. These are usually tedious and time-consuming tasks that
are traditionally done by biomedical experts. Recently, to automatize these
processes, deep learning based segmentation and tracking methods have been
proposed. These methods require large-scale datasets and their full potential
is constrained by the scarcity of annotated data in the biomedical imaging
domain. To address this limitation, we propose Biomedical Video Diffusion Model
(BVDM), capable of generating realistic-looking synthetic microscopy videos.
Trained only on a single real video, BVDM can generate videos of arbitrary
length with pixel-level annotations that can be used for training data-hungry
models. It is composed of a denoising diffusion probabilistic model (DDPM)
generating high-fidelity synthetic cell microscopy images and a flow prediction
model (FPM) predicting the non-rigid transformation between consecutive video
frames. During inference, initially, the DDPM imposes realistic cell textures
on synthetic cell masks which are generated based on real data statistics. The
flow prediction model predicts the flow field between consecutive masks and
applies that to the DDPM output from the previous time frame to create the next
one while keeping temporal consistency. BVDM outperforms state-of-the-art
synthetic live cell microscopy video generation models. Furthermore, we
demonstrate that a sufficiently large synthetic dataset enhances the
performance of cell segmentation and tracking models compared to using a
limited amount of available real data.