Color-aware Deep Temporal Backdrop Duplex Matting System
Hendrik Hachmann Bodo Rosenhahn
Deep learning-based alpha matting showed tremendous improvements in recent
years, yet, feature film production studios still rely on classical chroma
keying including costly post-production steps. This perceived discrepancy can
be explained by some missing links necessary for production which are currently
not adequately addressed in the alpha matting community, in particular
foreground color estimation or color spill compensation. We propose a neural
network-based temporal multi-backdrop production system that combines
beneficial features from chroma keying and alpha matting. Given two consecutive
frames with different background colors, our one-encoder-dual-decoder network
predicts foreground colors and alpha values using a patch-based overlap-blend
approach. The system is able to handle imprecise backdrops, dynamic cameras,
and dynamic foregrounds and has no restrictions on foreground colors. We
compare our method to state-of-the-art algorithms using benchmark datasets and
a video sequence captured by a demonstrator setup. We verify that a dual
backdrop input is superior to the usually applied trimap-based approach. In
addition, the proposed studio set is actor friendly, and produces high-quality,
temporal consistent alpha and color estimations that include a superior color
spill compensation.