Lightweight Portrait Matting via Regional Attention and Refinement
Yatao Zhong Ilya Zharkov
We present a lightweight model for high resolution portrait matting. The
model does not use any auxiliary inputs such as trimaps or background captures
and achieves real time performance for HD videos and near real time for 4K. Our
model is built upon a two-stage framework with a low resolution network for
coarse alpha estimation followed by a refinement network for local region
improvement. However, a naive implementation of the two-stage model suffers
from poor matting quality if not utilizing any auxiliary inputs. We address the
performance gap by leveraging the vision transformer (ViT) as the backbone of
the low resolution network, motivated by the observation that the tokenization
step of ViT can reduce spatial resolution while retain as much pixel
information as possible. To inform local regions of the context, we propose a
novel cross region attention (CRA) module in the refinement network to
propagate the contextual information across the neighboring regions. We
demonstrate that our method achieves superior results and outperforms other
baselines on three benchmark datasets while only uses $1/20$ of the FLOPS
compared to the existing state-of-the-art model.