Online Overexposed Pixels Hallucination in Videos with Adaptive
  Reference Frame Selection
Yazhou Xing Amrita Mazumdar Anjul Patney Chao Liu Hongxu Yin Qifeng Chen Jan Kautz Iuri Frosio
Low dynamic range (LDR) cameras cannot deal with wide dynamic range inputs,
frequently leading to local overexposure issues. We present a learning-based
system to reduce these artifacts without resorting to complex acquisition
mechanisms like alternating exposures or costly processing that are typical of
high dynamic range (HDR) imaging. We propose a transformer-based deep neural
network (DNN) to infer the missing HDR details. In an ablation study, we show
the importance of using a multiscale DNN and train it with the proper cost
function to achieve state-of-the-art quality. To aid the reconstruction of the
overexposed areas, our DNN takes a reference frame from the past as an
additional input. This leverages the commonly occurring temporal instabilities
of autoexposure to our advantage: since well-exposed details in the current
frame may be overexposed in the future, we use reinforcement learning to train
a reference frame selection DNN that decides whether to adopt the current frame
as a future reference. Without resorting to alternating exposures, we obtain
therefore a causal, HDR hallucination algorithm with potential application in
common video acquisition settings. Our demo video can be found at
https://drive.google.com/file/d/1-r12BKImLOYCLUoPzdebnMyNjJ4Rk360/view