Urban Radiance Field Representation with Deformable Neural Mesh
  Primitives
Fan Lu Yan Xu Guang Chen Hongsheng Li Kwan-Yee Lin Changjun Jiang
Neural Radiance Fields (NeRFs) have achieved great success in the past few
years. However, most current methods still require intensive resources due to
ray marching-based rendering. To construct urban-level radiance fields
efficiently, we design Deformable Neural Mesh Primitive~(DNMP), and propose to
parameterize the entire scene with such primitives. The DNMP is a flexible and
compact neural variant of classic mesh representation, which enjoys both the
efficiency of rasterization-based rendering and the powerful neural
representation capability for photo-realistic image synthesis. Specifically, a
DNMP consists of a set of connected deformable mesh vertices with paired vertex
features to parameterize the geometry and radiance information of a local area.
To constrain the degree of freedom for optimization and lower the storage
budgets, we enforce the shape of each primitive to be decoded from a relatively
low-dimensional latent space. The rendering colors are decoded from the vertex
features (interpolated with rasterization) by a view-dependent MLP. The DNMP
provides a new paradigm for urban-level scene representation with appealing
properties: $(1)$ High-quality rendering. Our method achieves leading
performance for novel view synthesis in urban scenarios. $(2)$ Low
computational costs. Our representation enables fast rendering (2.07ms/1k
pixels) and low peak memory usage (110MB/1k pixels). We also present a
lightweight version that can run 33$\times$ faster than vanilla NeRFs, and
comparable to the highly-optimized Instant-NGP (0.61 vs 0.71ms/1k pixels).
Project page: \href{https://dnmp.github.io/}{https://dnmp.github.io/}.