On-device Real-time Custom Hand Gesture Recognition
Esha Uboweja David Tian Qifei Wang Yi-Chun Kuo Joe Zou Lu Wang George Sung Matthias Grundmann
Most existing hand gesture recognition (HGR) systems are limited to a
predefined set of gestures. However, users and developers often want to
recognize new, unseen gestures. This is challenging due to the vast diversity
of all plausible hand shapes, e.g. it is impossible for developers to include
all hand gestures in a predefined list. In this paper, we present a
user-friendly framework that lets users easily customize and deploy their own
gesture recognition pipeline. Our framework provides a pre-trained single-hand
embedding model that can be fine-tuned for custom gesture recognition. Users
can perform gestures in front of a webcam to collect a small amount of images
per gesture. We also offer a low-code solution to train and deploy the custom
gesture recognition model. This makes it easy for users with limited ML
expertise to use our framework. We further provide a no-code web front-end for
users without any ML expertise. This makes it even easier to build and test the
end-to-end pipeline. The resulting custom HGR is then ready to be run on-device
for real-time scenarios. This can be done by calling a simple function in our
open-sourced model inference API, MediaPipe Tasks. This entire process only
takes a few minutes.