CLIP Brings Better Features to Visual Aesthetics Learners
Liwu Xu Jinjin Xu Yuzhe Yang Yijie Huang Yanchun Xie Yaqian Li
The success of pre-training approaches on a variety of downstream tasks has
revitalized the field of computer vision. Image aesthetics assessment (IAA) is
one of the ideal application scenarios for such methods due to subjective and
expensive labeling procedure. In this work, an unified and flexible two-phase
\textbf{C}LIP-based \textbf{S}emi-supervised \textbf{K}nowledge
\textbf{D}istillation paradigm is proposed, namely \textbf{\textit{CSKD}}.
Specifically, we first integrate and leverage a multi-source unlabeled dataset
to align rich features between a given visual encoder and an off-the-shelf CLIP
image encoder via feature alignment loss. Notably, the given visual encoder is
not limited by size or structure and, once well-trained, it can seamlessly
serve as a better visual aesthetic learner for both student and teacher. In the
second phase, the unlabeled data is also utilized in semi-supervised IAA
learning to further boost student model performance when applied in
latency-sensitive production scenarios. By analyzing the attention distance and
entropy before and after feature alignment, we notice an alleviation of feature
collapse issue, which in turn showcase the necessity of feature alignment
instead of training directly based on CLIP image encoder. Extensive experiments
indicate the superiority of CSKD, which achieves state-of-the-art performance
on multiple widely used IAA benchmarks.