SAM-IQA: Can Segment Anything Boost Image Quality Assessment?
Xinpeng Li Ting Jiang Haoqiang Fan Shuaicheng Liu
Image Quality Assessment (IQA) is a challenging task that requires training
on massive datasets to achieve accurate predictions. However, due to the lack
of IQA data, deep learning-based IQA methods typically rely on pre-trained
networks trained on massive datasets as feature extractors to enhance their
generalization ability, such as the ResNet network trained on ImageNet. In this
paper, we utilize the encoder of Segment Anything, a recently proposed
segmentation model trained on a massive dataset, for high-level semantic
feature extraction. Most IQA methods are limited to extracting spatial-domain
features, while frequency-domain features have been shown to better represent
noise and blur. Therefore, we leverage both spatial-domain and frequency-domain
features by applying Fourier and standard convolutions on the extracted
features, respectively. Extensive experiments are conducted to demonstrate the
effectiveness of all the proposed components, and results show that our
approach outperforms the state-of-the-art (SOTA) in four representative
datasets, both qualitatively and quantitatively. Our experiments confirm the
powerful feature extraction capabilities of Segment Anything and highlight the
value of combining spatial-domain and frequency-domain features in IQA tasks.
Code: https://github.com/Hedlen/SAM-IQA