CLIPose: Category-Level Object Pose Estimation with Pre-trained
  Vision-Language Knowledge
Xiao Lin Minghao Zhu Ronghao Dang Guangliang Zhou Shaolong Shu Feng Lin Chengju Liu Qijun Chen
Most of existing category-level object pose estimation methods devote to
learning the object category information from point cloud modality. However,
the scale of 3D datasets is limited due to the high cost of 3D data collection
and annotation. Consequently, the category features extracted from these
limited point cloud samples may not be comprehensive. This motivates us to
investigate whether we can draw on knowledge of other modalities to obtain
category information. Inspired by this motivation, we propose CLIPose, a novel
6D pose framework that employs the pre-trained vision-language model to develop
better learning of object category information, which can fully leverage
abundant semantic knowledge in image and text modalities. To make the 3D
encoder learn category-specific features more efficiently, we align
representations of three modalities in feature space via multi-modal
contrastive learning. In addition to exploiting the pre-trained knowledge of
the CLIP's model, we also expect it to be more sensitive with pose parameters.
Therefore, we introduce a prompt tuning approach to fine-tune image encoder
while we incorporate rotations and translations information in the text
descriptions. CLIPose achieves state-of-the-art performance on two mainstream
benchmark datasets, REAL275 and CAMERA25, and runs in real-time during
inference (40FPS).