Concurrent Linguistic Error Detection (CLED) for Large Language Models
Jinhua Zhu Javier Conde Zhen Gao Pedro Reviriego Shanshan Liu Fabrizio Lombardi
The wide adoption of Large language models (LLMs) makes their dependability a
pressing concern. Detection of errors is the first step to mitigating their
impact on a system and thus, efficient error detection for LLMs is an important
issue. In many settings, the LLM is considered as a black box with no access to
the internal nodes; this prevents the use of many error detection schemes that
need access to the model's internal nodes. An interesting observation is that
the output of LLMs in error-free operation should be valid and normal text.
Therefore, when the text is not valid or differs significantly from normal
text, it is likely that there is an error. Based on this observation we propose
to perform Concurrent Linguistic Error Detection (CLED); this scheme extracts
some linguistic features of the text generated by the LLM and feeds them to a
concurrent classifier that detects errors. Since the proposed error detection
mechanism only relies on the outputs of the model, then it can be used on LLMs
in which there is no access to the internal nodes. The proposed CLED scheme has
been evaluated on the T5 model when used for news summarization and on the
OPUS-MT model when used for translation. In both cases, the same set of
linguistic features has been used for error detection to illustrate the
applicability of the proposed scheme beyond a specific case. The results show
that CLED can detect most of the errors at a low overhead penalty. The use of
the concurrent classifier also enables a trade-off between error detection
effectiveness and its associated overhead, so providing flexibility to a
designer.