Diffusion-based Holistic Texture Rectification and Synthesis
Guoqing Hao Satoshi Iizuka Kensho Hara Edgar Simo-Serra Hirokatsu Kataoka Kazuhiro Fukui
We present a novel framework for rectifying occlusions and distortions in
degraded texture samples from natural images. Traditional texture synthesis
approaches focus on generating textures from pristine samples, which
necessitate meticulous preparation by humans and are often unattainable in most
natural images. These challenges stem from the frequent occlusions and
distortions of texture samples in natural images due to obstructions and
variations in object surface geometry. To address these issues, we propose a
framework that synthesizes holistic textures from degraded samples in natural
images, extending the applicability of exemplar-based texture synthesis
techniques. Our framework utilizes a conditional Latent Diffusion Model (LDM)
with a novel occlusion-aware latent transformer. This latent transformer not
only effectively encodes texture features from partially-observed samples
necessary for the generation process of the LDM, but also explicitly captures
long-range dependencies in samples with large occlusions. To train our model,
we introduce a method for generating synthetic data by applying geometric
transformations and free-form mask generation to clean textures. Experimental
results demonstrate that our framework significantly outperforms existing
methods both quantitatively and quantitatively. Furthermore, we conduct
comprehensive ablation studies to validate the different components of our
proposed framework. Results are corroborated by a perceptual user study which
highlights the efficiency of our proposed approach.