Sign Language Production with Latent Motion Transformer
Pan Xie Taiyi Peng Yao Du Qipeng Zhang
Sign Language Production (SLP) is the tough task of turning sign language
into sign videos. The main goal of SLP is to create these videos using a sign
gloss. In this research, we've developed a new method to make high-quality sign
videos without using human poses as a middle step. Our model works in two main
parts: first, it learns from a generator and the video's hidden features, and
next, it uses another model to understand the order of these hidden features.
To make this method even better for sign videos, we make several significant
improvements. (i) In the first stage, we take an improved 3D VQ-GAN to learn
downsampled latent representations. (ii) In the second stage, we introduce
sequence-to-sequence attention to better leverage conditional information.
(iii) The separated two-stage training discards the realistic visual semantic
of the latent codes in the second stage. To endow the latent sequences semantic
information, we extend the token-level autoregressive latent codes learning
with perceptual loss and reconstruction loss for the prior model with visual
perception. Compared with previous state-of-the-art approaches, our model
performs consistently better on two word-level sign language datasets, i.e.,
WLASL and NMFs-CSL.