Learning Concise and Descriptive Attributes for Visual Recognition
An Yan Yu Wang Yiwu Zhong Chengyu Dong Zexue He Yujie Lu William Wang Jingbo Shang Julian McAuley
Recent advances in foundation models present new opportunities for
interpretable visual recognition -- one can first query Large Language Models
(LLMs) to obtain a set of attributes that describe each class, then apply
vision-language models to classify images via these attributes. Pioneering work
shows that querying thousands of attributes can achieve performance competitive
with image features. However, our further investigation on 8 datasets reveals
that LLM-generated attributes in a large quantity perform almost the same as
random words. This surprising finding suggests that significant noise may be
present in these attributes. We hypothesize that there exist subsets of
attributes that can maintain the classification performance with much smaller
sizes, and propose a novel learning-to-search method to discover those concise
sets of attributes. As a result, on the CUB dataset, our method achieves
performance close to that of massive LLM-generated attributes (e.g., 10k
attributes for CUB), yet using only 32 attributes in total to distinguish 200
bird species. Furthermore, our new paradigm demonstrates several additional
benefits: higher interpretability and interactivity for humans, and the ability
to summarize knowledge for a recognition task.