CoRF : Colorizing Radiance Fields using Knowledge Distillation
Ankit Dhiman R Srinath Srinjay Sarkar Lokesh R Boregowda R Venkatesh Babu
Neural radiance field (NeRF) based methods enable high-quality novel-view
synthesis for multi-view images. This work presents a method for synthesizing
colorized novel views from input grey-scale multi-view images. When we apply
image or video-based colorization methods on the generated grey-scale novel
views, we observe artifacts due to inconsistency across views. Training a
radiance field network on the colorized grey-scale image sequence also does not
solve the 3D consistency issue. We propose a distillation based method to
transfer color knowledge from the colorization networks trained on natural
images to the radiance field network. Specifically, our method uses the
radiance field network as a 3D representation and transfers knowledge from
existing 2D colorization methods. The experimental results demonstrate that the
proposed method produces superior colorized novel views for indoor and outdoor
scenes while maintaining cross-view consistency than baselines. Further, we
show the efficacy of our method on applications like colorization of radiance
field network trained from 1.) Infra-Red (IR) multi-view images and 2.) Old
grey-scale multi-view image sequences.