Lost in the Source Language: How Large Language Models Evaluate the
  Quality of Machine Translation
Xu Huang Zhirui Zhang Xiang Geng Yichao Du Jiajun Chen Shujian Huang
Large Language Models (LLMs) have achieved remarkable results in the machine
translation evaluation task, yet there remains a gap in knowledge regarding how
they utilize the provided data to conduct evaluations. This study aims to
explore how LLMs leverage source and reference information in evaluating
translations, with the ultimate goal of better understanding the working
mechanism of LLMs. To this end, we design the controlled experiments across
various input modes and model types, and employ both coarse-grained and
fine-grained prompts to discern the utility of source versus reference
information. Surprisingly, we find that reference information significantly
enhances the evaluation accuracy, while source information sometimes is
counterproductive, indicating a lack of cross-lingual capability when using
LLMs to evaluate translations. We further conduct a meta-evaluation for
translation error detection of LLMs, observing a similar phenomenon. These
findings also suggest a potential research direction for LLMs that fully
exploits the cross-lingual capability of LLMs to achieve better performance in
machine translation evaluation tasks.