Disentangled Representation Learning for Controllable Person Image
  Generation
Wenju Xu Chengjiang Long Yongwei Nie Guanghui Wang
In this paper, we propose a novel framework named DRL-CPG to learn
disentangled latent representation for controllable person image generation,
which can produce realistic person images with desired poses and human
attributes (e.g., pose, head, upper clothes, and pants) provided by various
source persons. Unlike the existing works leveraging the semantic masks to
obtain the representation of each component, we propose to generate
disentangled latent code via a novel attribute encoder with transformers
trained in a manner of curriculum learning from a relatively easy step to a
gradually hard one. A random component mask-agnostic strategy is introduced to
randomly remove component masks from the person segmentation masks, which aims
at increasing the difficulty of training and promoting the transformer encoder
to recognize the underlying boundaries between each component. This enables the
model to transfer both the shape and texture of the components. Furthermore, we
propose a novel attribute decoder network to integrate multi-level attributes
(e.g., the structure feature and the attribute representation) with
well-designed Dual Adaptive Denormalization (DAD) residual blocks. Extensive
experiments strongly demonstrate that the proposed approach is able to transfer
both the texture and shape of different human parts and yield realistic
results. To our knowledge, we are the first to learn disentangled latent
representations with transformers for person image generation.