Training-Free Neural Matte Extraction for Visual Effects
Sharif Elcott J. P. Lewis Nori Kanazawa Christoph Bregler
Alpha matting is widely used in video conferencing as well as in movies,
television, and social media sites. Deep learning approaches to the matte
extraction problem are well suited to video conferencing due to the consistent
subject matter (front-facing humans), however training-based approaches are
somewhat pointless for entertainment videos where varied subjects (spaceships,
monsters, etc.) may appear only a few times in a single movie -- if a method of
creating ground truth for training exists, just use that method to produce the
desired mattes. We introduce a training-free high quality neural matte
extraction approach that specifically targets the assumptions of visual effects
production. Our approach is based on the deep image prior, which optimizes a
deep neural network to fit a single image, thereby providing a deep encoding of
the particular image. We make use of the representations in the penultimate
layer to interpolate coarse and incomplete "trimap" constraints. Videos
processed with this approach are temporally consistent. The algorithm is both
very simple and surprisingly effective.