Perceptual Kalman Filters: Online State Estimation under a Perfect
  Perceptual-Quality Constraint
Dror Freirich Tomer Michaeli Ron Meir
Many practical settings call for the reconstruction of temporal signals from
corrupted or missing data. Classic examples include decoding, tracking, signal
enhancement and denoising. Since the reconstructed signals are ultimately
viewed by humans, it is desirable to achieve reconstructions that are pleasing
to human perception. Mathematically, perfect perceptual-quality is achieved
when the distribution of restored signals is the same as that of natural
signals, a requirement which has been heavily researched in static estimation
settings (i.e. when a whole signal is processed at once). Here, we study the
problem of optimal causal filtering under a perfect perceptual-quality
constraint, which is a task of fundamentally different nature. Specifically, we
analyze a Gaussian Markov signal observed through a linear noisy
transformation. In the absence of perceptual constraints, the Kalman filter is
known to be optimal in the MSE sense for this setting. Here, we show that
adding the perfect perceptual quality constraint (i.e. the requirement of
temporal consistency), introduces a fundamental dilemma whereby the filter may
have to "knowingly" ignore new information revealed by the observations in
order to conform to its past decisions. This often comes at the cost of a
significant increase in the MSE (beyond that encountered in static settings).
Our analysis goes beyond the classic innovation process of the Kalman filter,
and introduces the novel concept of an unutilized information process. Using
this tool, we present a recursive formula for perceptual filters, and
demonstrate the qualitative effects of perfect perceptual-quality estimation on
a video reconstruction problem.