SAIR: Learning Semantic-aware Implicit Representation
Canyu Zhang Xiaoguang Li Qing Guo Song Wang
Implicit representation of an image can map arbitrary coordinates in the
continuous domain to their corresponding color values, presenting a powerful
capability for image reconstruction. Nevertheless, existing implicit
representation approaches only focus on building continuous appearance mapping,
ignoring the continuities of the semantic information across pixels. As a
result, they can hardly achieve desired reconstruction results when the
semantic information within input images is corrupted, for example, a large
region misses. To address the issue, we propose to learn semantic-aware
implicit representation (SAIR), that is, we make the implicit representation of
each pixel rely on both its appearance and semantic information (\eg, which
object does the pixel belong to). To this end, we propose a framework with two
modules: (1) building a semantic implicit representation (SIR) for a corrupted
image whose large regions miss. Given an arbitrary coordinate in the continuous
domain, we can obtain its respective text-aligned embedding indicating the
object the pixel belongs. (2) building an appearance implicit representation
(AIR) based on the SIR. Given an arbitrary coordinate in the continuous domain,
we can reconstruct its color whether or not the pixel is missed in the input.
We validate the novel semantic-aware implicit representation method on the
image inpainting task, and the extensive experiments demonstrate that our
method surpasses state-of-the-art approaches by a significant margin.