Inpainting Normal Maps for Lightstage data
Hancheng Zuo Bernard Tiddeman
This study introduces a novel method for inpainting normal maps using a
generative adversarial network (GAN). Normal maps, often derived from a
lightstage, are crucial in performance capture but can have obscured areas due
to movement (e.g., by arms, hair, or props). Inpainting fills these missing
areas with plausible data. Our approach extends previous general image
inpainting techniques, employing a bow tie-like generator network and a
discriminator network, with alternating training phases. The generator aims to
synthesize images aligning with the ground truth and deceive the discriminator,
which differentiates between real and processed images. Periodically, the
discriminator undergoes retraining to enhance its ability to identify processed
images. Importantly, our method adapts to the unique characteristics of normal
map data, necessitating modifications to the loss function. We utilize a cosine
loss instead of mean squared error loss for generator training. Limited
training data availability, even with synthetic datasets, demands significant
augmentation, considering the specific nature of the input data. This includes
appropriate image flipping and in-plane rotations to accurately alter normal
vectors. Throughout training, we monitored key metrics such as average loss,
Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio
(PSNR) for the generator, along with average loss and accuracy for the
discriminator. Our findings suggest that the proposed model effectively
generates high-quality, realistic inpainted normal maps, suitable for
performance capture applications. These results establish a foundation for
future research, potentially involving more advanced networks and comparisons
with inpainting of source images used to create the normal maps.