CLNeRF: Continual Learning Meets NeRF
Zhipeng Cai Matthias Mueller
Novel view synthesis aims to render unseen views given a set of calibrated
images. In practical applications, the coverage, appearance or geometry of the
scene may change over time, with new images continuously being captured.
Efficiently incorporating such continuous change is an open challenge. Standard
NeRF benchmarks only involve scene coverage expansion. To study other practical
scene changes, we propose a new dataset, World Across Time (WAT), consisting of
scenes that change in appearance and geometry over time. We also propose a
simple yet effective method, CLNeRF, which introduces continual learning (CL)
to Neural Radiance Fields (NeRFs). CLNeRF combines generative replay and the
Instant Neural Graphics Primitives (NGP) architecture to effectively prevent
catastrophic forgetting and efficiently update the model when new data arrives.
We also add trainable appearance and geometry embeddings to NGP, allowing a
single compact model to handle complex scene changes. Without the need to store
historical images, CLNeRF trained sequentially over multiple scans of a
changing scene performs on-par with the upper bound model trained on all scans
at once. Compared to other CL baselines CLNeRF performs much better across
standard benchmarks and WAT. The source code, and the WAT dataset are available
at https://github.com/IntelLabs/CLNeRF. Video presentation is available at:
https://youtu.be/nLRt6OoDGq0?si=8yD6k-8MMBJInQPs