SignDiff: Learning Diffusion Models for American Sign Language
  Production
Sen Fang Chunyu Sui Xuedong Zhang Yapeng Tian
The field of Sign Language Production (SLP) lacked a large-scale, pre-trained
model based on deep learning for continuous American Sign Language (ASL)
production in the past decade. This limitation hampers communication for all
individuals with disabilities relying on ASL. To address this issue, we
undertook the secondary development and utilization of How2Sign, one of the
largest publicly available ASL datasets. Despite its significance, prior
researchers in the field of sign language have not effectively employed this
corpus due to the intricacies involved in American Sign Language Production
(ASLP).
  To conduct large-scale ASLP, we propose SignDiff based on the latest work in
related fields, which is a dual-condition diffusion pre-training model that can
generate human sign language speakers from a skeleton pose. SignDiff has a
novel Frame Reinforcement Network called FR-Net, similar to dense human pose
estimation work, which enhances the correspondence between text lexical symbols
and sign language dense pose frames reduce the occurrence of multiple fingers
in the diffusion model. In addition, our ASLP method proposes two new improved
modules and a new loss function to improve the accuracy and quality of sign
language skeletal posture and enhance the ability of the model to train on
large-scale data.
  We propose the first baseline for ASL production and report the scores of
17.19 and 12.85 on BLEU-4 on the How2Sign dev/test sets. We also evaluated our
model on the previous mainstream dataset called PHOENIX14T, and the main
experiments achieved the results of SOTA. In addition, our image quality far
exceeds all previous results by 10 percentage points on the SSIM indicator.
Finally, we conducted ablation studies and qualitative evaluations for
discussion.