Flow Matching in Latent Space
Quan Dao Hao Phung Binh Nguyen Anh Tran
Flow matching is a recent framework to train generative models that exhibits
impressive empirical performance while being relatively easier to train
compared with diffusion-based models. Despite its advantageous properties,
prior methods still face the challenges of expensive computing and a large
number of function evaluations of off-the-shelf solvers in the pixel space.
Furthermore, although latent-based generative methods have shown great success
in recent years, this particular model type remains underexplored in this area.
In this work, we propose to apply flow matching in the latent spaces of
pretrained autoencoders, which offers improved computational efficiency and
scalability for high-resolution image synthesis. This enables flow-matching
training on constrained computational resources while maintaining their quality
and flexibility. Additionally, our work stands as a pioneering contribution in
the integration of various conditions into flow matching for conditional
generation tasks, including label-conditioned image generation, image
inpainting, and semantic-to-image generation. Through extensive experiments,
our approach demonstrates its effectiveness in both quantitative and
qualitative results on various datasets, such as CelebA-HQ, FFHQ, LSUN Church &
Bedroom, and ImageNet. We also provide a theoretical control of the
Wasserstein-2 distance between the reconstructed latent flow distribution and
true data distribution, showing it is upper-bounded by the latent flow matching
objective. Our code will be available at
https://github.com/VinAIResearch/LFM.git.