ORTexME: Occlusion-Robust Human Shape and Pose via Temporal Average
  Texture and Mesh Encoding
Yu Cheng Bo Wang Robby T. Tan
In 3D human shape and pose estimation from a monocular video, models trained
with limited labeled data cannot generalize well to videos with occlusion,
which is common in the wild videos. The recent human neural rendering
approaches focusing on novel view synthesis initialized by the off-the-shelf
human shape and pose methods have the potential to correct the initial human
shape. However, the existing methods have some drawbacks such as, erroneous in
handling occlusion, sensitive to inaccurate human segmentation, and ineffective
loss computation due to the non-regularized opacity field. To address these
problems, we introduce ORTexME, an occlusion-robust temporal method that
utilizes temporal information from the input video to better regularize the
occluded body parts. While our ORTexME is based on NeRF, to determine the
reliable regions for the NeRF ray sampling, we utilize our novel average
texture learning approach to learn the average appearance of a person, and to
infer a mask based on the average texture. In addition, to guide the
opacity-field updates in NeRF to suppress blur and noise, we propose the use of
human body mesh. The quantitative evaluation demonstrates that our method
achieves significant improvement on the challenging multi-person 3DPW dataset,
where our method achieves 1.8 P-MPJPE error reduction. The SOTA rendering-based
methods fail and enlarge the error up to 5.6 on the same dataset.