Defocus to focus: Photo-realistic bokeh rendering by fusing defocus and
  radiance priors
Xianrui Luo Juewen Peng Ke Xian Zijin Wu Zhiguo Cao
We consider the problem of realistic bokeh rendering from a single
all-in-focus image. Bokeh rendering mimics aesthetic shallow depth-of-field
(DoF) in professional photography, but these visual effects generated by
existing methods suffer from simple flat background blur and blurred in-focus
regions, giving rise to unrealistic rendered results. In this work, we argue
that realistic bokeh rendering should (i) model depth relations and distinguish
in-focus regions, (ii) sustain sharp in-focus regions, and (iii) render
physically accurate Circle of Confusion (CoC). To this end, we present a
Defocus to Focus (D2F) framework to learn realistic bokeh rendering by fusing
defocus priors with the all-in-focus image and by implementing radiance priors
in layered fusion. Since no depth map is provided, we introduce defocus
hallucination to integrate depth by learning to focus. The predicted defocus
map implies the blur amount of bokeh and is used to guide weighted layered
rendering. In layered rendering, we fuse images blurred by different kernels
based on the defocus map. To increase the reality of the bokeh, we adopt
radiance virtualization to simulate scene radiance. The scene radiance used in
weighted layered rendering reassigns weights in the soft disk kernel to produce
the CoC. To ensure the sharpness of in-focus regions, we propose to fuse
upsampled bokeh images and original images. We predict the initial fusion mask
from our defocus map and refine the mask with a deep network. We evaluate our
model on a large-scale bokeh dataset. Extensive experiments show that our
approach is capable of rendering visually pleasing bokeh effects in complex
scenes. In particular, our solution receives the runner-up award in the AIM
2020 Rendering Realistic Bokeh Challenge.