Evaluating VLMs for Score-Based, Multi-Probe Annotation of 3D Objects
Rishabh Kabra Loic Matthey Alexander Lerchner Niloy J. Mitra
Unlabeled 3D objects present an opportunity to leverage pretrained vision
language models (VLMs) on a range of annotation tasks -- from describing object
semantics to physical properties. An accurate response must take into account
the full appearance of the object in 3D, various ways of phrasing the
question/prompt, and changes in other factors that affect the response. We
present a method to marginalize over any factors varied across VLM queries,
utilizing the VLM's scores for sampled responses. We first show that this
probabilistic aggregation can outperform a language model (e.g., GPT4) for
summarization, for instance avoiding hallucinations when there are contrasting
details between responses. Secondly, we show that aggregated annotations are
useful for prompt-chaining; they help improve downstream VLM predictions (e.g.,
of object material when the object's type is specified as an auxiliary input in
the prompt). Such auxiliary inputs allow ablating and measuring the
contribution of visual reasoning over language-only reasoning. Using these
evaluations, we show how VLMs can approach, without additional training or
in-context learning, the quality of human-verified type and material
annotations on the large-scale Objaverse dataset.