CVT-xRF: Contrastive In-Voxel Transformer for 3D Consistent Radiance
  Fields from Sparse Inputs
Yingji Zhong Lanqing Hong Zhenguo Li Dan Xu
Neural Radiance Fields (NeRF) have shown impressive capabilities for
photorealistic novel view synthesis when trained on dense inputs. However, when
trained on sparse inputs, NeRF typically encounters issues of incorrect density
or color predictions, mainly due to insufficient coverage of the scene causing
partial and sparse supervision, thus leading to significant performance
degradation. While existing works mainly consider ray-level consistency to
construct 2D learning regularization based on rendered color, depth, or
semantics on image planes, in this paper we propose a novel approach that
models 3D spatial field consistency to improve NeRF's performance with sparse
inputs. Specifically, we first adopt a voxel-based ray sampling strategy to
ensure that the sampled rays intersect with a certain voxel in 3D space. We
then randomly sample additional points within the voxel and apply a Transformer
to infer the properties of other points on each ray, which are then
incorporated into the volume rendering. By backpropagating through the
rendering loss, we enhance the consistency among neighboring points.
Additionally, we propose to use a contrastive loss on the encoder output of the
Transformer to further improve consistency within each voxel. Experiments
demonstrate that our method yields significant improvement over different
radiance fields in the sparse inputs setting, and achieves comparable
performance with current works.