---
author:
- 'Yang-Tian Sun, Qian-Cheng Fu, Yue-Ren Jiang, Zitao Liu, Yu-Kun Lai, Hongbo Fu, Lin Gao[^1]'
bibliography:
- 'IEEEabrv.bib'
- 'egbib.bib'
title: Human Motion Transfer with 3D Constraints and Detail Enhancement
---

![image](./image/figure1.jpg){width="95.00000%"}

problem of video-based human motion transfer is an interesting but challenging research problem. Given two monocular video clips, one for a source subject and the other for a target subject, the goal of this problem is to transfer the motion from the source person to the target, while maintaining the target person’s appearance. Specifically, in the synthesized video, the subject should have the same motion as the source person, and the same appearance as the target person (including human clothes and background). To achieve this, it is essential to produce high-quality image-to-image translation of frames, while ensuring temporal coherence.

The difficulty of this problem is how to effectively decouple and recombine the posture information and appearance information of the source and target characters. Based on generative adversarial networks (GANs), a powerful tool for high-quality image-to-image translation, Chan [*et al.*]{} [@chan18:every_dance_now] proposed to first learn a mapping from a 2D pose to a subject image from the target video, and then use the pose of the source subject as the input to the learned mapping for video synthesis. However, due to the difference between the source and target poses, this approach often results in noticeable artifacts, especially for the self-occlusion of body parts.

Observing that the self-occlusion issue is difficult to handle in the image domain, we propose to first reconstruct a 3D human model from a 2D image of both the source and target subjects, and then adjust the pose of the target human body to match the source (while maintaining the target person’s body shape). Intrinsic geometric description of the deformed target is then projected back to 2D to form an image that reflects 3D structure.

This along with the 2D pose figure extracted from the source image is used as a constraint during GAN-based image-to-image translation, to effectively maintain the structural characteristics of human body under different poses.

In addition, previous methods [@chan18:every_dance_now; @wang2018vid2vid] only use the appearance of the target person in the training process of pose-to-image translation, and does not fully utilize the appearance of the source. When an input pose is very different from any poses seen during the training process, such solutions might lead to blurry results. Observing that the source video frame corresponding to the input pose might contain reusable rich details (especially for the body parts like hands where the source and target subjects share some similarity), we intend to selectively transfer details from real source frames to the synthesized video frames. This is achieved by our detail enhancement network. [Figure \[fig:teaser\] shows representative motion transfer results with rich details]{}.

We summarize our contributions as follows: 1) We propose to reconstruct a 3D human body with its shape from a target frame and its pose from a source frame, and project it to 2D to serve as a GAN-based network condition. This contains rich 3D information including body shape, pose, occlusion to help maintain the structural characteristics of the human body in the generated images. 2) We introduce the detail enhancement net (DE-Net), which utilizes the information from the real source frames to enhance details in the generated results.

Related Work
============

Over the last decades, motion transfer has been extensively studied due to its ability for fast video content production. Some early solutions have mainly revolved around realigning existing video footage according to the similarity to the desired pose [@Bregler1997VideoRD; @Efros03]. However, it is not an easy task to find an accurate similarity measure for different actions of different subjects. Several other approaches have also attempted to address this problem in 3D, but they focus on the use of inverse kinematic solvers [@DBLP:conf/siggraph/LeeS99] and transfer motion between 3D *skeletons* [@Hecker2008RealtimeMR], whereas we consider using a reconstructed 3D body mesh to guide motion transfer in the image domain.

Recently, the rapid advances of deep learning, especially generative adversarial networks (GANs) and their variations (e.g., cGAN [@mirza2014conditional], CoGAN [@DBLP:conf/nips/LiuT16], CycleGAN [@CycleGAN2017], DiscoGAN [@DBLP:journals/corr/KimCKLK17]) have provided a powerful tool for image-to-image translation, which has yielded impressive results across a wide spectrum of synthesis tasks and shows its ability to synthesize visually pleasing images from conditional labels. Pix2pix [@isola2017image], based on a conditional GAN framework, is one of the pioneering works. CycleGAN [@CycleGAN2017] further presents the idea of cycle consistency loss for learning to translate between two domains in the absence of paired images, and Recycle-GAN [@bansal2018recycle] combines both spatial and temporal constraints for video retargeting tasks. Pix2pixHD [@wang17:pix2pixHD] introduces a multi-scale conditional GAN to synthesize high-resolution images using both global and local generators, and vid2vid [@wang2018vid2vid] designs specific spatial and temporal adversarial constraints for video synthesis.

Based on these variants of GANs, a lot of approaches [@Balakrishnan2018SynthesizingIO; @chan18:every_dance_now; @vunet2018; @ma2017pose; @ma2018disentangled] have been proposed for human motion transfer between two domains. The key idea of these approaches is to decouple the pose information from the input image and use it as the input of a GAN network to generate a realistic output image. For example, in [@Balakrishnan2018SynthesizingIO], the input image is separated into two parts: the foreground (or different body parts) and background, and the final realistic image is generated by separately processing and cross fusion of the two parts. Chan [*et al.*]{}[@chan18:every_dance_now] extract pose information with an off-the-shelf human pose detector OpenPose [@cao2018openpose; @cao2017realtime; @simon2017hand; @wei2016cpm] and use the pix2pixHD [@wang17:pix2pixHD] framework together with a specialized Face GAN to learn a mapping from a 2D pose figure to an image. Neverova [*et al.*]{}[@neverova2018dense] adopt a similar idea but use the estimation of DensePose [@Guler2018DensePose] to guide image generation. Wang [*et al.*]{}make a step further to adopt both OpenPose and DensePose in [@wang2018vid2vid]. However, due to the lack of 3D semantic information, these approaches are highly sensitive to problems such as self-occlusions.

To solve the above problems, it is natural to add 3D information to the condition of generative networks. There are many robust 3D human mesh reconstruction methods such as [@joo2018total; @kanazawaHMR18; @SMPL-X:2019; @xiang2019monocular], which can reconstruct a 3D model corresponding pose from a single image or a video clip. Benefiting from these accurate and reliable 3D body reconstruction techniques, we can study the issue of human motion transfer in a new perspective. Liu [*et al.*]{}[@lwb2019] present a novel warping strategy, which also uses the projection of 3D models to tackle motion transfer, appearance transfer and novel view synthesis within a unified model. However, due to the diversity of their network functionalities, it does not perform particularly well in the aspect of motion transfer.

Method
======

![image](./image/pipeline.jpg){width="95.00000%"}

We aim to generate a new video of the target person imitating the character movements in the source video, while keeping the structural integrity and detail features of the target subject as much as possible. To accomplish this, we use the mesh projection containing 3D information as the condition for the GAN, and introduce a detail enhancement mechanism to improve the details.

Overview
--------

We denote $\mathcal{S} = \{S_i\}$ as a set of source video frames, and $\mathcal{T} = \{T_j\}$ as a set of target frames. Our architecture can be divided into two modules, as shown in Figure \[fig:pipeline\]: the Motion Transfer Net (MT-Net) on account of motion transfer across two domains and the Detail Enhancement Net (DE-Net) used for the enhancement of details. More specifically, MT-Net takes [**two real frames**]{} $I_{app}$ and $I_{pose}$ as input, and generates an output image $I_{MT}$ that has the same appearance as $I_{app}$ and the pose as $I_{pose}$. DE-Net takes image $I_B$ as input, which is the blending of raw transfer result $I_{MT}$ with blurred details and corresponding real frame $I_{pose}$ with rich details, and aims to generate an image $I_{DE}$ in the [**target domain**]{} with the details enhanced. Our *training pipeline* is as follows:

[**Within-Domain Pre-Training of MT-Net.**]{} To stabilize the training process, we first pre-train MT-Net using within-domain samples. For the domain $\mathcal{S}$, let $I_{app}, I_{pose} \in \mathcal{S}$, and we can obtain $I_{MT}$, which is the reconstructed source frame. For the domain $\mathcal{T}$, let $I_{app}, I_{pose} \in \mathcal{T}$, and we should obtain the reconstructed target frame. This process initializes the MT-Net. Note that for each $I_{pose}$, $I_{app}$ is randomly selected from corresponding domain and fixed during the training process. [**Training of DE-Net.**]{} After the pre-training of MT-Net, we let $I_{app} \in \mathcal{S}$ and $I_{pose} \in \mathcal{T}$, which generates initial transferred image $I_{MT}$ by the MT-Net, which is often blurred. We then calculate a blended image $I_B$ which is an average of $I_{MT}$ and corresponding real frame $I_{pose}$ that contains clear details. We then train the DE-Net to discern and generate details from the blended result selectively to produce output image $I_{DE}$ with details enhanced that matches the target domain. Our *transfer pipeline* is as follows: Let $I_{app} \in \mathcal{T}$ and $I_{pose} \in \mathcal{S}$, we can get the initial transfer result $I_{MT}$ (with the source pose and target appearance) using MT-Net. And then we can obtain the final result $I_{DE}$ with details enhanced by the DE-Net.

Note that the domains of $I_{app}$ and $I_{pose}$ for training DE-Net and transfer are swapped, because in the training setting, $I_{MT}$ has the appearance as the source and pose as the target, and DE-Net aims to produce an image with the appearance and pose both in the target domain, so the ground truth of $I_{DE}$ is available (which is exactly corresponding $I_{pose} \in \mathcal{T}$). This provides supervision for training $I_{DE}$ to enhance details in the target domain (i.e. with the appearance of the target subject). Such supervision is not available if the transfer setting is used.

Motion Transfer Net
-------------------

![Architecture of MT-Net, which synthesizes transfer image with appearance of Input1 and pose of Input2.[]{data-label="fig:mt"}](./image/mt_net.jpg){width="0.9\linewidth"}

As illustrated in Figure \[fig:mt\], the Motion Transfer Net consists of 3 parts: Label Image Generator that produces label images that encode 2D/3D human pose and shape information, Appearance Encoder that encodes the appearance of the input image, and Pose GAN that produces an output image with given appearance and pose/shape constraints.

### Label Image Generation via 3D Human Models

To maintain the structural integrity of the generated results and produce realistic images for actions involving self-occlusions, we utilize the 3D geometry information of the underlying subject to produce label images as the GAN condition to regularize the generative network. The architecture of our label image generator is shown in Figure \[fig:pose extractor\].

[**3D human model reconstruction.**]{} We first extract the 3D body shape $\beta$ and pose $\theta$ information for both source and target videos using a state-of-the-art pre-trained 3D pose and shape estimator [@xiang2019monocular]. This leads to a 3D deformable mesh model including the details of body, face and fingers. When transferring between two domains, the 3D human models also allow the generation of a 3D mesh with the pose from one domain and shape from the other. The extracted deformable mesh sequences might exhibit temporal incoherence artifacts due to inevitable reconstruction errors. This can be alleviated by simply applying temporal smoothing to mesh vertices, since our mesh sequences have the same connectivity.

[**Human model projection.** ]{} We project the reconstructed 3D human model onto 2D to obtain a label image, which will be used as the condition to guide the generator. The image should ideally contain intrinsic 3D information (invariant to pose changes) to guide the synthesis process such that a particular color corresponds to a specific location on the human body. To achieve this, we propose to extract the three non-trivial eigenvectors corresponding to three smallest eigenvalues and consider them as a 3-channel image assigned to each vertex [@meyer2003discrete], which is projected to 2D to form a 3D constraint image. Although additional 3D information is available, 3D meshes extracted from 2D images may occasionally contain artifacts due to the inherent ambiguity. Therefore, we also adopt OpenPose [@cao2018openpose; @cao2017realtime; @simon2017hand; @wei2016cpm] to extract a 2D pose figure as part of the condition, which is less informative but more robust in the 2D space. Our label image therefore is 6-channel after combining both 2D and 3D constraints.

![Architecture of our label image generator. On the one hand, we reconstruct a 3D mesh of the transferred human body, assign eigenvectors corresponding to the three smallest eigenvalues of its Laplace matrix as intrinsic features (visualized in RGB color), and project it to form a 3D constraint image. On the other hand, we connect the keypoints extracted by a 2D pose detector to form a 2D pose figure. These are concatenated as the 6-channel label image, denoted as $I_{label}$.[]{data-label="fig:pose extractor"}](./image/pose_extractor.jpg){width="0.9\linewidth"}

### Pose to Image Translation

We learn the mapping from label image sequences to realistic image sequences by training a conditional GAN, consisting of Appearance Encoder and Pose GAN. The design of Pose GAN is similar to pix2pixHD[@wang17:pix2pixHD]: It is composed of a generator $G_{pose}$, and two multi-scale frame discriminators with the same architecture for images in the source and target domains, respectively. The two networks drive each other: the generator learns to synthesize more realistic images conditioned on the input to fool the discriminator, while the discriminator in turn learns to discern the “real” images (ground truth) and “fake” (generated) images. The difference of our network from pix2pixHD is that the data we use to learn the mapping includes not only target video, but also source video. We have done this by conditioning $G_{pose}$ on both label images and appearance features, extracted by the Label Image Generator and Appearance Encoder respectively. See Figure \[fig:GAN\] for the generative network architecture. It is worth mentioning that in order to solve the problem of poor continuity caused by single frame generation, similar to [@chan18:every_dance_now], adjacent frames are involved in training to improve temporal coherence.

![Architecture of the generative network. Appearance image $I_{app}$ is randomly selected and sent to Appearance Encoder $E_{app}$ to obtain appearance features. Adjacent label images $I_{label}$ (current frame) and $I'_{label}$ (previous frame) are sent to Pose GAN $G_{pose}$ together with appearance features to generate the reconstructed image or initial transfer result $I_{MT}$.[]{data-label="fig:GAN"}](./image/gan.jpg){width="0.9\linewidth"}

**Appearance Encoder and Pose GAN Generator.** As said above, to make full use of the given data and meet the need of the subsequent detail enhancement, we train the generative network using data from both source video and target video. However, training two separate conditional GANs has a high overhead for computing resources and time. In order to simplify this process, we introduce an Appearance Encoder, and use label images containing 2D/3D constraints and appearance features together to guide Pose GAN to produce the reconstructed image (for within-domain input) or initial transfer result $I_{MT}$ (for cross-domain input). Note that when the given subject video is replaced, our framework only needs to finetune the upsampling part of Pose GAN for the generation of a new subject.

Appearance Encoder is a fully convolutional network that extracts appearance features of the input image $I_{app}$, which is used as a condition for the Pose GAN. It takes randomly selected frames as input and outputs appearance features corresponding to that domain. Pose GAN is the main part of MT-Net, which consists of three submodules: Downsampling, ResNet blocks and Upsampling. It works on both label images and the appearance features extracted by the Appearance Encoder, and synthesize results with the corresponding pose and appearance. As shown in Figure \[fig:GAN\], the output of Appearance Encoder is added to the intermediate ResNet blocks in the generator $G_{app}$.

**Pose GAN Discriminator.** We use the multi-scale discriminator presented in pix2pixHD [@wang17:pix2pixHD]. Discriminators of different scales can give the discrimination of images at different levels. In our method, we use two discriminators $D^\mathcal{S}_{pose}$ and $D^\mathcal{T}_{pose}$ to discriminate the probability of generated images belonging to the corresponding domain, each with 3 scales. **Temporal Smoothing.** We use the time smoothing strategy in [@chan18:every_dance_now] to enhance the continuity between adjacent generated frames. The generation of the current frame is not only related to the current label image $I_{label}$, but also related to the previous frame $I'_{label}$.

Therefore, let $d \in \{\mathcal{S}, \mathcal{T}\}$ denote the domain in which the training images are selected, our conditional GAN has the following objective:

$$\begin{aligned}
\nonumber    \mathcal{L}^d_{\text{MT}} (&E_{app}, G_{pose}, D^d_{pose}) =  \\
\nonumber    &\mathbb{E}[\log D_{pose}^d(I_{label}, I'_{label}, I_{pose}, I'_{pose})] + \\
    &\mathbb{E}[\log(1-D_{pose}^d(I_{label}, I'_{label}, I_{MT}, I'_{MT})]
\label{equa:GAN}\end{aligned}$$

Here the discriminator $D^d_{pose}$ takes a pair of adjacent images in the domain $d$, and classifies them to real images (trained using the current frame $I_{pose}$, and previous frame $I'_{pose}$ from the training set), or fake images ($I_{MT}$ and the previous frame output $I'_{MT}$ generated by the Pose GAN).

Detail Enhancement Net
----------------------

Through the first stage of training, we can obtain initial transfer results with blurred details: source to target transfer result $I_{S2T}$ as well as target to source transfer result $I_{T2S}$. We can also construct paired data $(I_S, I_{S2T})$ and $(I_T, I_{T2S})$, as shown in Figure \[fig:comparison\], where images in the same pair have the same pose but different appearances, and different clarity of details, which motivates us to use a DE-Net to enhance the details from the blended image. Note that in different videos, subjects may have different builds or different positions relative to camera. In such cases, before sending to DE-Net we transform source frame by aligning the head position and height in accordance with target frame with simple scaling and translation.

![Comparison of the source image $I_S$ and source to target transfer result $I_{S2T}$, as well as the target image $I_T$ and target to source transfer result $I_{T2S}$.[]{data-label="fig:comparison"}](./image/compare.jpg){width="\linewidth"}

The purpose of our DE-Net is to generate clear details of target images from the blended image pair. It is a GAN where the generator $G_{DE}$ is a U-net which synthesizes images in the target domain with clear details, as illustrated in Figure \[fig:Refine\]. The discriminator $D_{DE}$ discerns the “real” images (ground truth) and “fake” images (synthesized by $G_{DE}$).

In the training stage, we use the blended image pair $(I_{T2S}, I_{T})$ as input and supervisely train DE-Net with $I_{T}$ as ground truth. The use of mean blended image instead of concatenation avoids the output overfitting to $I_T$. We optimize the DE-Net by the following objective: $$\begin{aligned}
\nonumber & \mathcal{L}_{\texttt{DE}}(G_{DE}, D_{DE}) = \mathbb{E}[\log D_{DE}(I_{label}, I'_{label}, I_T)] \\
& \qquad +\mathbb{E} [1 - \log D_{DE}(I_{label}, I'_{label}, I_{DE})]
\label{equa:refine}\end{aligned}$$ Where $I_{label}$ and $I'_{label}$ are the label images used to generate $I_{T2S}$.

![Architecture of Detail Enhancement Net (DE-Net). The main part of our DE-Net is a U-net, which takes the mean of paired data $(I_{T2S}, I_T)$ or $(I_S, I_{S2T})$ as input, and synthesizes a target image $I_{DE}$ with details enhanced.[]{data-label="fig:Refine"}](./image/refine_net.jpg){width="0.8\linewidth"}

In the transfer stage, we use the source to target transfer result $I_{S2T}$ and the corresponding source image $I_S$ to obtain enhanced transfer result.

Full Objective
--------------

The training of our network is divided into two stages. First we train the Motion Transfer Net, which consists of Pose GAN and Appearance Encoder. The full objective contains adversarial loss, perceptual loss and discriminator feature-matching loss, which has the following form: $$\begin{aligned}
\nonumber    \min \limits_{E_{app},G_{pose}}&((\max \limits_{D^\mathcal{S}_{pose}} \mathcal{L}^{\mathcal{S}}_{\text{MT}}(E_{app}, G_{pose}, D^{\mathcal{S}}_{pose})) \\
\nonumber    &+ (\max \limits_{D^\mathcal{T}_{pose}} \mathcal{L}^{\mathcal{T}}_{\text{MT}}(E_{app}, G_{pose}, D^{\mathcal{T}}_{pose})) \\
\nonumber    &+ \lambda_P \mathcal{L}_P(I_{MT}, I_{pose}) \\ 
\nonumber &+ \lambda_{FM}\mathcal{L}_{FM}((E_{app}, G_{pose}), D^\mathcal{S}_{pose}) \\
    &+ \lambda_{FM}\mathcal{L}_{FM}((E_{app}, G_{pose}), D^\mathcal{T}_{pose})).\end{aligned}$$ Here, $\mathcal{L}^{\mathcal{S}}_{\text{MT}}$ and $\mathcal{L}^{\mathcal{T}}_{\text{MT}}$ are defined in Eq. \[equa:GAN\]. The perceptual loss $\mathcal{L}_{P}$ regularizes the generated result $I_{MT}$ to be closer to the ground truth $I_{pose}$ in the VGG-19 [@simonyan2014very] feature space, defined as $$\label{eq:Lp}
\mathcal{L}_P(I_1, I_2)  = ||\text{VGG}(I_{1}) - \text{VGG}(I_{2})||_1.$$ The discriminator feature-matching loss $\mathcal{L}_{FM}$ is presented in pix2pixHD [@wang17:pix2pixHD] and similarly regularizes the output using intermediate result of the discriminator, calculated as $$\label{eq:Lfm}
\mathcal{L}_{FM}(G,D_k) = \mathbb{E} \sum \limits_{i=1} \limits^T \frac{1}{N_i}[||D_k^{(i)}(s,x)-D_k^{(i)}(s,G(s))||_1],$$ where $T$ is the number of layers, $N_i$ is the number of elements in the $i$th layer and $k$ is the index of discriminators in the multi-scale architecture. $s$ is the condition of cGAN and $x$ is the corresponding ground truth. The DE-Net is optimized with the following objective $$\begin{aligned}
\nonumber    \min \limits_{G_{DE}}&((\max \limits_ {D_{DE}} \mathcal{L}_{\text{DE}}(G_{DE}, D_{DE})) + \lambda_P \mathcal{L}_P(I_{DE}, I_{pose}) \\
&+ \lambda_{FM}\mathcal{L}_{FM}(G_{DE}, D_{DE})).\end{aligned}$$ Here $\mathcal{L}_{\text{DE}}$ is defined in Eq. \[equa:refine\]. The perceptual and discriminator feature-matching losses are defined in Eqs. \[eq:Lp\] and \[eq:Lfm\].

![image](image/fig1_jyrsample.jpg)

Experiment
==========

We compare our method with state-of-the-art methods and ablation variants, both quantitatively and qualitatively.

Setup
-----

[**Dataset.**]{} To verify the performance of our method, we collected 8 in-the-wild single-dancer videos from YouTube and filmed 5 videos ourselves, out of which 2 with ordinary background and 3 with green screen. All videos are at $1920 \times 1080$ resolution and each lasts 2-5 minutes. Each subject wears different clothes and performs different types of action such as freestyle dancing and stretching exercises. To prepare for training and testing, We cut the start and end parts that contain no action, and crop and normalize each frame to 512 $\times$ 1024 by simple scaling and translation. [**Implementation details.**]{} \[sec:training\] We adopt a multi-stage training strategy in our method using Adam optimizer with learning rate 0.0001. In the first stage, we pre-train the MT-Net for 20 epochs. In the next stage, the parameters of MT-Net are fixed and DE-Net is trained individually for 10 epoches. We set hyperparameters $\lambda_{FM} = 10$ and $\lambda_P = 5$ for both stages. More details about MT-Net and DE-Net are given in supplementary material.

[**Existing methods.**]{} We compare our performance with existing state-of-the-art methods vid2vid [@wang2018vid2vid], Everybody Dance Now [@chan18:every_dance_now] and Liquid Warping GAN [@lwb2019], using official implementation.

![Comparison with state-of-the-art methods. We show the generated results by vid2vid, Everybody Dance Now, Liquid Warping GAN and our method. Only our method has reconstructed the wriggling hand and smile face.[]{data-label="fig:contrast"}](image/fig2_jyrcontrast.jpg)

Quantitative Results
--------------------

[**Evaluation Metrics.**]{} We use objective metrics for quantitative evaluation under two different conditions: 1) To directly measure the quality of the generated images, we perform self-transfer in which the source and target are the same subject, and then use SSIM [@wang2004image] and learning-based perceptual similarity (LPIPS) [@zhang2018perceptual] to assess the similarity between source and target images. We split frames of each subject into training and test set at the ratio of 8:2 for this evaluation. 2) We also evaluate the performance of cross-subject transfer where the source and target are different subjects, using inception score [@salimans2016improved] and Fréchet Inception Distance [@heusel2017gans] as metrics. It should be noted that we compute the FID score between the original and generated target images since there exists no ground truth for comparison in this case. We exclude the green screen dataset in quantitative evaluation to focus on more challenging cases. The metrics mentioned above are all based on single frames, which cannot reflect the smoothness of generated image sequences. The effect of mesh filtering in time series can be observed in the video results and quantitatively measured by the user study.

### Comparison with existing methods.

Comparison results with state of the arts are reported in Table \[table1\].

It can be found that our method performs better than others.

  -- -- ------- ------- ------- -----------
                                
                        ours    
        0.781   0.836   0.790   **0.887**
        0.096   0.067   0.106   **0.041**
        3.186   3.238   3.064   **3.424**
        59.98   57.02   81.20   **53.76**
  -- -- ------- ------- ------- -----------

  : Quantitative comparison with state of the arts on the dance dataset. Metrics are averaged over the 6 subjects. For SSIM and IS, higher is better. For LPIPS and FID, lower is better.[]{data-label="table1"}

### Ablation study

In this part, we perform an ablation study to verify the impact of each component of our model, including using 3D constraints (“3D”) and DE-Net (“DE”). Our full pipeline is indicated “Full“. When 3D is disabled, we use 2D pose figures as default label images.

Table \[table2\] shows the results of the ablation study. It is obvious that our full proposed framework performs better than other variants. And both 3d constraints and DE-Net enhance the results. The comparison of **MT(3D only)** and **MT** proved the effect of 3D constraints. However, due to the projection results fail to fully characterize human facial features and expression, its score is slightly below **MT+3D**. Furthermore, we can observe that scores of self-transfer between **MT** and **MT+3D** (or **MT+DE** and **Full**) are similar. This is because source and target subjects share the same body shape in self-transfer, which somewhat limits the effectiveness of 3D constraints, where the scores of cross-subject transfer demonstrate the important role 3D information plays on transfer between different subjects with different shapes.

### User evaluation

We also conduct a user study to measure the human perceptual quality for cross-subject transfer results. In our experiments, we compare videos generated by vid2vid, Chan [*et al.*]{}, Liquid Warping GAN and our method. Specifically, we show to volunteers a series of videos by each of the methods at the resolution of $1024 \times 512$, and the volunteers are given unlimited time to make responses. 50 distinct participants are involved and each of them is asked to select: 1) the clearest result with rich details; 2) the most temporally stable result; and 3) the overall best result. As shown in Table \[table3\], our method is more realistic, with richer details and with better temporal stability in comparison with other methods.

Qualitative Results
-------------------

We visualize our generated results in Figure \[fig:sample\]. It can be seen that our method successfully drives the motions of different targets with structural integrity and rich details, particularly in the face and hands. We also demonstrate that our method outperforms existing methods in Figure \[fig:contrast\].

As illustrated in the first row of Figure \[fig:contrast\], our method can enhance the structural integrity of arms and legs, and avoid the missing hands in the case that other methods fail to generate. At the same time, our method can also characterize details of the generated results more accurately, such as facial expression shown in the second row of Figure \[fig:contrast\]. Figure \[fig:ablation\] shows the advantage of using the 3D constraints and DE-Net.

Limitations and Discussion
--------------------------

Although our model is able to synthesize motion transferred images with high authenticity and details, there are still several limitations. We show some failure cases with visual artifacts shown in Figure \[fig:failure case\]. In the left example, our model fails to eliminate the long hair of the source character in result, while in the right, some undesired part of clothes appears in the generated image because of the loose source clothes.

![Failure cases. For each case, the source image is shown on the left and our transfer result on the right.[]{data-label="fig:failure case"}](image/fig4_jyrfailure.jpg)

These failure cases are mainly attributed to the abnormal movement of source character which causes large change in human body shape, such as shaking hair or opening clothes suddenly, in which case the DE-Net fails to eliminate extra details. Our future work will focus on improving the ability of DE-Net to avoid the appearance of undesired details in the transfer results.

Conclusion
==========

We have proposed a new approach to human motion transfer. It employs the 3D body shape and pose constraints as a condition to regularize the generative adversarial learning framework, which is more expressive and complete than 2D. We also design a enhancement mechanism to reinforce the detail characteristics of synthesized results using detailed information from real source frames. Extensive experiments show that our method outperforms existing methods both visually and quantitatively.

[^1]: Corresponding Author.
