CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using
  Score-Based Diffusion Models
Maitreya Suin Rama Chellappa
Recent generative-prior-based methods have shown promising blind face
restoration performance. They usually project the degraded images to the latent
space and then decode high-quality faces either by single-stage latent
optimization or directly from the encoding. Generating fine-grained facial
details faithful to inputs remains a challenging problem. Most existing methods
produce either overly smooth outputs or alter the identity as they attempt to
balance between generation and reconstruction. This may be attributed to the
typical trade-off between quality and resolution in the latent space. If the
latent space is highly compressed, the decoded output is more robust to
degradations but shows worse fidelity. On the other hand, a more flexible
latent space can capture intricate facial details better, but is extremely
difficult to optimize for highly degraded faces using existing techniques. To
address these issues, we introduce a diffusion-based-prior inside a VQGAN
architecture that focuses on learning the distribution over uncorrupted latent
embeddings. With such knowledge, we iteratively recover the clean embedding
conditioning on the degraded counterpart. Furthermore, to ensure the reverse
diffusion trajectory does not deviate from the underlying identity, we train a
separate Identity Recovery Network and use its output to constrain the reverse
diffusion process. Specifically, using a learnable latent mask, we add
gradients from a face-recognition network to a subset of latent features that
correlates with the finer identity-related details in the pixel space, leaving
the other features untouched. Disentanglement between perception and fidelity
in the latent space allows us to achieve the best of both worlds. We perform
extensive evaluations on multiple real and synthetic datasets to validate the
superiority of our approach.