TRAVID: An End-to-End Video Translation Framework
Prottay Kumar Adhikary Bandaru Sugandhi Subhojit Ghimire Santanu Pal Partha Pakray
In today's globalized world, effective communication with people from diverse
linguistic backgrounds has become increasingly crucial. While traditional
methods of language translation, such as written text or voice-only
translations, can accomplish the task, they often fail to capture the complete
context and nuanced information conveyed through nonverbal cues like facial
expressions and lip movements. In this paper, we present an end-to-end video
translation system that not only translates spoken language but also
synchronizes the translated speech with the lip movements of the speaker. Our
system focuses on translating educational lectures in various Indian languages,
and it is designed to be effective even in low-resource system settings. By
incorporating lip movements that align with the target language and matching
them with the speaker's voice using voice cloning techniques, our application
offers an enhanced experience for students and users. This additional feature
creates a more immersive and realistic learning environment, ultimately making
the learning process more effective and engaging.