Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion
Weng Fei Low Gim Hee Lee
Event cameras offer many advantages over standard cameras due to their
distinctive principle of operation: low power, low latency, high temporal
resolution and high dynamic range. Nonetheless, the success of many downstream
visual applications also hinges on an efficient and effective scene
representation, where Neural Radiance Field (NeRF) is seen as the leading
candidate. Such promise and potential of event cameras and NeRF inspired recent
works to investigate on the reconstruction of NeRF from moving event cameras.
However, these works are mainly limited in terms of the dependence on dense and
low-noise event streams, as well as generalization to arbitrary contrast
threshold values and camera speed profiles. In this work, we propose Robust
e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving
event cameras under various real-world conditions, especially from sparse and
noisy events generated under non-uniform motion. It consists of two key
components: a realistic event generation model that accounts for various
intrinsic parameters (e.g. time-independent, asymmetric threshold and
refractory period) and non-idealities (e.g. pixel-to-pixel threshold
variation), as well as a complementary pair of normalized reconstruction losses
that can effectively generalize to arbitrary speed profiles and intrinsic
parameter values without such prior knowledge. Experiments on real and novel
realistically simulated sequences verify our effectiveness. Our code, synthetic
dataset and improved event simulator are public.