Appearance Codes using Joint Embedding Learning of Multiple Modalities
Alex Zhang Evan Dogariu
The use of appearance codes in recent work on generative modeling has enabled
novel view renders with variable appearance and illumination, such as day-time
and night-time renders of a scene. A major limitation of this technique is the
need to re-train new appearance codes for every scene on inference, so in this
work we address this problem proposing a framework that learns a joint
embedding space for the appearance and structure of the scene by enforcing a
contrastive loss constraint between different modalities. We apply our
framework to a simple Variational Auto-Encoder model on the RADIATE dataset
\cite{sheeny2021radiate} and qualitatively demonstrate that we can generate new
renders of night-time photos using day-time appearance codes without additional
optimization iterations. Additionally, we compare our model to a baseline VAE
that uses the standard per-image appearance code technique and show that our
approach achieves generations of similar quality without learning appearance
codes for any unseen images on inference.