Coordinate-Aware Modulation for Neural Fields
Joo Chan Lee Daniel Rho Seungtae Nam Jong Hwan Ko Eunbyung Park
Neural fields, mapping low-dimensional input coordinates to corresponding
signals, have shown promising results in representing various signals. Numerous
methodologies have been proposed, and techniques employing MLPs and grid
representations have achieved substantial success. MLPs allow compact and high
expressibility, yet often suffer from spectral bias and slow convergence speed.
On the other hand, methods using grids are free from spectral bias and achieve
fast training speed, however, at the expense of high spatial complexity. In
this work, we propose a novel way for exploiting both MLPs and grid
representations in neural fields. Unlike the prevalent methods that combine
them sequentially (extract features from the grids first and feed them to the
MLP), we inject spectral bias-free grid representations into the intermediate
features in the MLP. More specifically, we suggest a Coordinate-Aware
Modulation (CAM), which modulates the intermediate features using scale and
shift parameters extracted from the grid representations. This can maintain the
strengths of MLPs while mitigating any remaining potential biases, facilitating
the rapid learning of high-frequency components. In addition, we empirically
found that the feature normalizations, which have not been successful in neural
filed literature, proved to be effective when applied in conjunction with the
proposed CAM. Experimental results demonstrate that CAM enhances the
performance of neural representation and improves learning stability across a
range of signals. Especially in the novel view synthesis task, we achieved
state-of-the-art performance with the least number of parameters and fast
training speed for dynamic scenes and the best performance under 1MB memory for
static scenes. CAM also outperforms the best-performing video compression
methods using neural fields by a large margin.