Local Distortion Aware Efficient Transformer Adaptation for Image
  Quality Assessment
Kangmin Xu Liang Liao Jing Xiao Chaofeng Chen Haoning Wu Qiong Yan Weisi Lin
Image Quality Assessment (IQA) constitutes a fundamental task within the
field of computer vision, yet it remains an unresolved challenge, owing to the
intricate distortion conditions, diverse image contents, and limited
availability of data. Recently, the community has witnessed the emergence of
numerous large-scale pretrained foundation models, which greatly benefit from
dramatically increased data and parameter capacities. However, it remains an
open problem whether the scaling law in high-level tasks is also applicable to
IQA task which is closely related to low-level clues. In this paper, we
demonstrate that with proper injection of local distortion features, a larger
pretrained and fixed foundation model performs better in IQA tasks.
Specifically, for the lack of local distortion structure and inductive bias of
vision transformer (ViT), alongside the large-scale pretrained ViT, we use
another pretrained convolution neural network (CNN), which is well known for
capturing the local structure, to extract multi-scale image features. Further,
we propose a local distortion extractor to obtain local distortion features
from the pretrained CNN and a local distortion injector to inject the local
distortion features into ViT. By only training the extractor and injector, our
method can benefit from the rich knowledge in the powerful foundation models
and achieve state-of-the-art performance on popular IQA datasets, indicating
that IQA is not only a low-level problem but also benefits from stronger
high-level features drawn from large-scale pretrained models.