Preconditioners for the Stochastic Training of Implicit Neural
  Representations
Shin-Fang Chng Hemanth Saratchandran Simon Lucey
Implicit neural representations have emerged as a powerful technique for
encoding complex continuous multidimensional signals as neural networks,
enabling a wide range of applications in computer vision, robotics, and
geometry. While Adam is commonly used for training due to its stochastic
proficiency, it entails lengthy training durations. To address this, we explore
alternative optimization techniques for accelerated training without
sacrificing accuracy. Traditional second-order optimizers like L-BFGS are
suboptimal in stochastic settings, making them unsuitable for large-scale data
sets. Instead, we propose stochastic training using curvature-aware diagonal
preconditioners, showcasing their effectiveness across various signal
modalities such as images, shape reconstruction, and Neural Radiance Fields
(NeRF).