Customizable Perturbation Synthesis for Robust SLAM Benchmarking
Xiaohao Xu Tianyi Zhang Sibo Wang Xiang Li Yongqi Chen Ye Li Bhiksha Raj Matthew Johnson-Roberson Xiaonan Huang
Robustness is a crucial factor for the successful deployment of robots in
unstructured environments, particularly in the domain of Simultaneous
Localization and Mapping (SLAM). Simulation-based benchmarks have emerged as a
highly scalable approach for robustness evaluation compared to real-world data
collection. However, crafting a challenging and controllable noisy world with
diverse perturbations remains relatively under-explored. To this end, we
propose a novel, customizable pipeline for noisy data synthesis, aimed at
assessing the resilience of multi-modal SLAM models against various
perturbations. This pipeline incorporates customizable hardware setups,
software components, and perturbed environments. In particular, we introduce
comprehensive perturbation taxonomy along with a perturbation composition
toolbox, allowing the transformation of clean simulations into challenging
noisy environments. Utilizing the pipeline, we instantiate the Robust-SLAM
benchmark, which includes diverse perturbation types, to evaluate the risk
tolerance of existing advanced multi-modal SLAM models. Our extensive analysis
uncovers the susceptibilities of existing SLAM models to real-world
disturbance, despite their demonstrated accuracy in standard benchmarks. Our
perturbation synthesis toolbox, SLAM robustness evaluation pipeline, and
Robust-SLAM benchmark will be made publicly available at
https://github.com/Xiaohao-Xu/SLAM-under-Perturbation/.