OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object
  Detection
Hu Zhang Jianhua Xu Tao Tang Haiyang Sun Xin Yu Zi Huang Kaicheng Yu
Traditional LiDAR-based object detection research primarily focuses on
closed-set scenarios, which falls short in complex real-world applications.
Directly transferring existing 2D open-vocabulary models with some known LiDAR
classes for open-vocabulary ability, however, tends to suffer from over-fitting
problems: The obtained model will detect the known objects, even presented with
a novel category. In this paper, we propose OpenSight, a more advanced 2D-3D
modeling framework for LiDAR-based open-vocabulary detection. OpenSight
utilizes 2D-3D geometric priors for the initial discernment and localization of
generic objects, followed by a more specific semantic interpretation of the
detected objects. The process begins by generating 2D boxes for generic objects
from the accompanying camera images of LiDAR. These 2D boxes, together with
LiDAR points, are then lifted back into the LiDAR space to estimate
corresponding 3D boxes. For better generic object perception, our framework
integrates both temporal and spatial-aware constraints. Temporal awareness
correlates the predicted 3D boxes across consecutive timestamps, recalibrating
the missed or inaccurate boxes. The spatial awareness randomly places some
``precisely'' estimated 3D boxes at varying distances, increasing the
visibility of generic objects. To interpret the specific semantics of detected
objects, we develop a cross-modal alignment and fusion module to first align 3D
features with 2D image embeddings and then fuse the aligned 3D-2D features for
semantic decoding. Our experiments indicate that our method establishes
state-of-the-art open-vocabulary performance on widely used 3D detection
benchmarks and effectively identifies objects for new categories of interest.