Combining Vision and EMG-Based Hand Tracking for Extended Reality
  Musical Instruments
Max Graf Mathieu Barthet
Hand tracking is a critical component of natural user interactions in
extended reality (XR) environments, including extended reality musical
instruments (XRMIs). However, self-occlusion remains a significant challenge
for vision-based hand tracking systems, leading to inaccurate results and
degraded user experiences. In this paper, we propose a multimodal hand tracking
system that combines vision-based hand tracking with surface electromyography
(sEMG) data for finger joint angle estimation. We validate the effectiveness of
our system through a series of hand pose tasks designed to cover a wide range
of gestures, including those prone to self-occlusion. By comparing the
performance of our multimodal system to a baseline vision-based tracking
method, we demonstrate that our multimodal approach significantly improves
tracking accuracy for several finger joints prone to self-occlusion. These
findings suggest that our system has the potential to enhance XR experiences by
providing more accurate and robust hand tracking, even in the presence of
self-occlusion.