Exploiting GPT-4 Vision for Zero-shot Point Cloud Understanding
Qi Sun Xiao Cui Wengang Zhou Houqiang Li
In this study, we tackle the challenge of classifying the object category in
point clouds, which previous works like PointCLIP struggle to address due to
the inherent limitations of the CLIP architecture. Our approach leverages GPT-4
Vision (GPT-4V) to overcome these challenges by employing its advanced
generative abilities, enabling a more adaptive and robust classification
process. We adapt the application of GPT-4V to process complex 3D data,
enabling it to achieve zero-shot recognition capabilities without altering the
underlying model architecture. Our methodology also includes a systematic
strategy for point cloud image visualization, mitigating domain gap and
enhancing GPT-4V's efficiency. Experimental validation demonstrates our
approach's superiority in diverse scenarios, setting a new benchmark in
zero-shot point cloud classification.