The Unreasonable Effectiveness of Linear Prediction as a Perceptual
  Metric
Daniel Severo Lucas Theis Johannes Ballé
We show how perceptual embeddings of the visual system can be constructed at
inference-time with no training data or deep neural network features. Our
perceptual embeddings are solutions to a weighted least squares (WLS) problem,
defined at the pixel-level, and solved at inference-time, that can capture
global and local image characteristics. The distance in embedding space is used
to define a perceptual similarity metric which we call LASI: Linear
Autoregressive Similarity Index. Experiments on full-reference image quality
assessment datasets show LASI performs competitively with learned deep feature
based methods like LPIPS (Zhang et al., 2018) and PIM (Bhardwaj et al., 2020),
at a similar computational cost to hand-crafted methods such as MS-SSIM (Wang
et al., 2003). We found that increasing the dimensionality of the embedding
space consistently reduces the WLS loss while increasing performance on
perceptual tasks, at the cost of increasing the computational complexity. LASI
is fully differentiable, scales cubically with the number of embedding
dimensions, and can be parallelized at the pixel-level. A Maximum
Differentiation (MAD) competition (Wang & Simoncelli, 2008) between LASI and
LPIPS shows that both methods are capable of finding failure points for the
other, suggesting these metrics can be combined.