On the Content Bias in Fréchet Video Distance
Songwei Ge Aniruddha Mahapatra Gaurav Parmar Jun-Yan Zhu Jia-Bin Huang
Fr\'echet Video Distance (FVD), a prominent metric for evaluating video
generation models, is known to conflict with human perception occasionally. In
this paper, we aim to explore the extent of FVD's bias toward per-frame quality
over temporal realism and identify its sources. We first quantify the FVD's
sensitivity to the temporal axis by decoupling the frame and motion quality and
find that the FVD increases only slightly with large temporal corruption. We
then analyze the generated videos and show that via careful sampling from a
large set of generated videos that do not contain motions, one can drastically
decrease FVD without improving the temporal quality. Both studies suggest FVD's
bias towards the quality of individual frames. We further observe that the bias
can be attributed to the features extracted from a supervised video classifier
trained on the content-biased dataset. We show that FVD with features extracted
from the recent large-scale self-supervised video models is less biased toward
image quality. Finally, we revisit a few real-world examples to validate our
hypothesis.