HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined
  RGB and Depth Inpainting
Ankan Dash Jingyi Gu Guiling Wang
Inpainting involves filling in missing pixels or areas in an image, a crucial
technique employed in Mixed Reality environments for various applications,
particularly in Diminished Reality (DR) where content is removed from a user's
visual environment. Existing methods rely on digital replacement techniques
which necessitate multiple cameras and incur high costs. AR devices and
smartphones use ToF depth sensors to capture scene depth maps aligned with RGB
images. Despite speed and affordability, ToF cameras create imperfect depth
maps with missing pixels. To address the above challenges, we propose
Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in
a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked
edge and segmentation label images respectively, while CombinedRGBD-GAN
combines their latent representation outputs and performs RGB and Depth
inpainting. Edge images and particularly segmentation label images as auxiliary
inputs significantly enhance inpainting performance by complementary context
and hierarchical optimization. We believe we make the first attempt to
incorporate label images into inpainting process.Unlike previous approaches
requiring multiple sequential models and separate outputs, our work operates in
an end-to-end manner, training all three models simultaneously and
hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized
separately and further optimized inside CombinedRGBD-GAN to enhance inpainting
quality. Experiments demonstrate that HI-GAN works seamlessly and achieves
overall superior performance compared with existing approaches.