VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
Yiying Yang Wen Liu Fukun Yin Xin Chen Gang Yu Jiayuan Fan Tao Chen
Recent advancements in implicit neural representations have contributed to
high-fidelity surface reconstruction and photorealistic novel view synthesis.
However, the computational complexity inherent in these methodologies presents
a substantial impediment, constraining the attainable frame rates and
resolutions in practical applications. In response to this predicament, we
propose VQ-NeRF, an effective and efficient pipeline for enhancing implicit
neural representations via vector quantization. The essence of our method
involves reducing the sampling space of NeRF to a lower resolution and
subsequently reinstating it to the original size utilizing a pre-trained VAE
decoder, thereby effectively mitigating the sampling time bottleneck
encountered during rendering. Although the codebook furnishes representative
features, reconstructing fine texture details of the scene remains challenging
due to high compression rates. To overcome this constraint, we design an
innovative multi-scale NeRF sampling scheme that concurrently optimizes the
NeRF model at both compressed and original scales to enhance the network's
ability to preserve fine details. Furthermore, we incorporate a semantic loss
function to improve the geometric fidelity and semantic coherence of our 3D
reconstructions. Extensive experiments demonstrate the effectiveness of our
model in achieving the optimal trade-off between rendering quality and
efficiency. Evaluation on the DTU, BlendMVS, and H3DS datasets confirms the
superior performance of our approach.