Latent Space Editing in Transformer-Based Flow Matching
Vincent Tao Hu David W Zhang Pascal Mettes Meng Tang Deli Zhao Cees G. M. Snoek
This paper strives for image editing via generative models. Flow Matching is
an emerging generative modeling technique that offers the advantage of simple
and efficient training. Simultaneously, a new transformer-based U-ViT has
recently been proposed to replace the commonly used UNet for better scalability
and performance in generative modeling. Hence, Flow Matching with a transformer
backbone offers the potential for scalable and high-quality generative
modeling, but their latent structure and editing ability are as of yet unknown.
Hence, we adopt this setting and explore how to edit images through latent
space manipulation. We introduce an editing space, which we call $u$-space,
that can be manipulated in a controllable, accumulative, and composable manner.
Additionally, we propose a tailored sampling solution to enable sampling with
the more efficient adaptive step-size ODE solvers. Lastly, we put forth a
straightforward yet powerful method for achieving fine-grained and nuanced
editing using text prompts. Our framework is simple and efficient, all while
being highly effective at editing images while preserving the essence of the
original content. Our code will be publicly available at https://taohu.me/lfm/