Enhancing Bandwidth Efficiency for Video Motion Transfer Applications
  using Deep Learning Based Keypoint Prediction
Xue Bai Tasmiah Haque Sumit Mohan Yuliang Cai Byungheon Jeong Adam Halasz Srinjoy Das
We propose a deep learning based novel prediction framework for enhanced
bandwidth reduction in motion transfer enabled video applications such as video
conferencing, virtual reality gaming and privacy preservation for patient
health monitoring. To model complex motion, we use the First Order Motion Model
(FOMM) that represents dynamic objects using learned keypoints along with their
local affine transformations. Keypoints are extracted by a self-supervised
keypoint detector and organized in a time series corresponding to the video
frames. Prediction of keypoints, to enable transmission using lower frames per
second on the source device, is performed using a Variational Recurrent Neural
Network (VRNN). The predicted keypoints are then synthesized to video frames
using an optical flow estimator and a generator network. This efficacy of
leveraging keypoint based representations in conjunction with VRNN based
prediction for both video animation and reconstruction is demonstrated on three
diverse datasets. For real-time applications, our results show the
effectiveness of our proposed architecture by enabling up to 2x additional
bandwidth reduction over existing keypoint based video motion transfer
frameworks without significantly compromising video quality.