Restoring Ancient Ideograph: A Multimodal Multitask Neural Network
  Approach
Siyu Duan Jun Wang Qi Su
Cultural heritage serves as the enduring record of human thought and history.
Despite significant efforts dedicated to the preservation of cultural relics,
many ancient artefacts have been ravaged irreversibly by natural deterioration
and human actions. Deep learning technology has emerged as a valuable tool for
restoring various kinds of cultural heritages, including ancient text
restoration. Previous research has approached ancient text restoration from
either visual or textual perspectives, often overlooking the potential of
synergizing multimodal information. This paper proposes a novel Multimodal
Multitask Restoring Model (MMRM) to restore ancient texts, particularly
emphasising the ideograph. This model combines context understanding with
residual visual information from damaged ancient artefacts, enabling it to
predict damaged characters and generate restored images simultaneously. We
tested the MMRM model through experiments conducted on both simulated datasets
and authentic ancient inscriptions. The results show that the proposed method
gives insightful restoration suggestions in both simulation experiments and
real-world scenarios. To the best of our knowledge, this work represents the
pioneering application of multimodal deep learning in ancient text restoration,
which will contribute to the understanding of ancient society and culture in
digital humanities fields.