PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D
  representations for agricultural robotics
Claus Smitt Michael Halstead Patrick Zimmer Thomas Läbe Esra Guclu Cyrill Stachniss Chris McCool
Precise scene understanding is key for most robot monitoring and intervention
tasks in agriculture. In this work we present PAg-NeRF which is a novel
NeRF-based system that enables 3D panoptic scene understanding. Our
representation is trained using an image sequence with noisy robot odometry
poses and automatic panoptic predictions with inconsistent IDs between frames.
Despite this noisy input, our system is able to output scene geometry,
photo-realistic renders and 3D consistent panoptic representations with
consistent instance IDs. We evaluate this novel system in a very challenging
horticultural scenario and in doing so demonstrate an end-to-end trainable
system that can make use of noisy robot poses rather than precise poses that
have to be pre-calculated. Compared to a baseline approach the peak signal to
noise ratio is improved from 21.34dB to 23.37dB while the panoptic quality
improves from 56.65% to 70.08%. Furthermore, our approach is faster and can be
tuned to improve inference time by more than a factor of 2 while being memory
efficient with approximately 12 times fewer parameters.