Teaching Chinese Sign Language with Feedback in Mixed Reality
Hongli Wen Yang Xu Lin Li Xudong Ru
Traditional sign language teaching methods face challenges such as limited
feedback and diverse learning scenarios. Although 2D resources lack real-time
feedback, classroom teaching is constrained by a scarcity of teacher. Methods
based on VR and AR have relatively primitive interaction feedback mechanisms.
This study proposes an innovative teaching model that uses real-time monocular
vision and mixed reality technology. First, we introduce an improved
hand-posture reconstruction method to achieve sign language semantic retention
and real-time feedback. Second, a ternary system evaluation algorithm is
proposed for a comprehensive assessment, maintaining good consistency with
experts in sign language. Furthermore, we use mixed reality technology to
construct a scenario-based 3D sign language classroom and explore the user
experience of scenario teaching. Overall, this paper presents a novel teaching
method that provides an immersive learning experience, advanced posture
reconstruction, and precise feedback, achieving positive feedback on user
experience and learning effectiveness.