Lester: rotoscope animation through video object segmentation and
  tracking
Ruben Tous
This article introduces Lester, a novel method to automatically synthetise
retro-style 2D animations from videos. The method approaches the challenge
mainly as an object segmentation and tracking problem. Video frames are
processed with the Segment Anything Model (SAM) and the resulting masks are
tracked through subsequent frames with DeAOT, a method of hierarchical
propagation for semi-supervised video object segmentation. The geometry of the
masks' contours is simplified with the Douglas-Peucker algorithm. Finally,
facial traits, pixelation and a basic shadow effect can be optionally added.
The results show that the method exhibits an excellent temporal consistency and
can correctly process videos with different poses and appearances, dynamic
shots, partial shots and diverse backgrounds. The proposed method provides a
more simple and deterministic approach than diffusion models based
video-to-video translation pipelines, which suffer from temporal consistency
problems and do not cope well with pixelated and schematic outputs. The method
is also much most practical than techniques based on 3D human pose estimation,
which require custom handcrafted 3D models and are very limited with respect to
the type of scenes they can process.