Zero-shot detection of buildings in mobile LiDAR using Language Vision
  Model
June Moh Goo Zichao Zeng Jan Boehm
Recent advances have demonstrated that Language Vision Models (LVMs) surpass
the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision
tasks, motivating attempts to apply LVMs to three-dimensional (3D) data. While
LVMs are efficient and effective in addressing various downstream 2D vision
tasks without training, they face significant challenges when it comes to point
clouds, a representative format for representing 3D data. It is more difficult
to extract features from 3D data and there are challenges due to large data
sizes and the cost of the collection and labelling, resulting in a notably
limited availability of datasets. Moreover, constructing LVMs for point clouds
is even more challenging due to the requirements for large amounts of data and
training time. To address these issues, our research aims to 1) apply the
Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2)
experiment with synthetic data to evaluate its effectiveness in bridging the
gap between synthetic and real-world data domains. Our approach exhibited high
performance with an accuracy of 0.96, an IoU of 0.85, precision of 0.92, recall
of 0.91, and an F1 score of 0.92, confirming its potential. However, challenges
such as occlusion problems and pixel-level overlaps of multi-label points
during spherical image generation remain to be addressed in future studies.