Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation
Xueting Hu Ce Zhang Yi Zhang Bowen Hai Ke Yu Zhihai He
Pre-trained Vision-Language Models (VLMs), such as CLIP, have shown enhanced
performance across a range of tasks that involve the integration of visual and
linguistic modalities. When CLIP is used for depth estimation tasks, the
patches, divided from the input images, can be combined with a series of
semantic descriptions of the depth information to obtain similarity results.
The coarse estimation of depth is then achieved by weighting and summing the
depth values, called depth bins, corresponding to the predefined semantic
descriptions. The zero-shot approach circumvents the computational and
time-intensive nature of traditional fully-supervised depth estimation methods.
However, this method, utilizing fixed depth bins, may not effectively
generalize as images from different scenes may exhibit distinct depth
distributions. To address this challenge, we propose a few-shot-based method
which learns to adapt the VLMs for monocular depth estimation to balance
training costs and generalization capabilities. Specifically, it assigns
different depth bins for different scenes, which can be selected by the model
during inference. Additionally, we incorporate learnable prompts to preprocess
the input text to convert the easily human-understood text into easily
model-understood vectors and further enhance the performance. With only one
image per scene for training, our extensive experiment results on the NYU V2
and KITTI dataset demonstrate that our method outperforms the previous
state-of-the-art method by up to 10.6\% in terms of MARE.