CLiF-VQA: Enhancing Video Quality Assessment by Incorporating High-Level
  Semantic Information related to Human Feelings
Yachun Mi Yu Li Yan Shu Chen Hui Puchao Zhou Shaohui Liu
Video Quality Assessment (VQA) aims to simulate the process of perceiving
video quality by the human visual system (HVS). The judgments made by HVS are
always influenced by human subjective feelings. However, most of the current
VQA research focuses on capturing various distortions in the spatial and
temporal domains of videos, while ignoring the impact of human feelings. In
this paper, we propose CLiF-VQA, which considers both features related to human
feelings and spatial features of videos. In order to effectively extract
features related to human feelings from videos, we explore the consistency
between CLIP and human feelings in video perception for the first time.
Specifically, we design multiple objective and subjective descriptions closely
related to human feelings as prompts. Further we propose a novel CLIP-based
semantic feature extractor (SFE) which extracts features related to human
feelings by sliding over multiple regions of the video frame. In addition, we
further capture the low-level-aware features of the video through a spatial
feature extraction module. The two different features are then aggregated
thereby obtaining the quality score of the video. Extensive experiments show
that the proposed CLiF-VQA exhibits excellent performance on several VQA
datasets.