Towards Scene-Text to Scene-Text Translation
Onkar Susladkar Prajwal Gatti Anand Mishra
In this work, we study the task of ``visually" translating scene text from a
source language (e.g., English) to a target language (e.g., Chinese). Visual
translation involves not just the recognition and translation of scene text but
also the generation of the translated image that preserves visual features of
the text, such as font, size, and background. There are several challenges
associated with this task, such as interpolating font to unseen characters and
preserving text size and the background. To address these, we introduce VTNet,
a novel conditional diffusion-based method. To train the VTNet, we create a
synthetic cross-lingual dataset of 600K samples of scene text images in six
popular languages, including English, Hindi, Tamil, Chinese, Bengali, and
German. We evaluate the performance of VTnet through extensive experiments and
comparisons to related methods. Our model also surpasses the previous
state-of-the-art results on the conventional scene-text editing benchmarks.
Further, we present rigorous qualitative studies to understand the strengths
and shortcomings of our model. Results show that our approach generalizes well
to unseen words and fonts. We firmly believe our work can benefit real-world
applications, such as text translation using a phone camera and translating
educational materials. Code and data will be made publicly available.