NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition
  via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning
Kamer Ali Yuksel Thiago Ferreira Golara Javadi Mohamed El-Badrashiny Ahmet Gunduz
This paper introduces NoRefER, a novel referenceless quality metric for
automatic speech recognition (ASR) systems. Traditional reference-based metrics
for evaluating ASR systems require costly ground-truth transcripts. NoRefER
overcomes this limitation by fine-tuning a multilingual language model for
pair-wise ranking ASR hypotheses using contrastive learning with Siamese
network architecture. The self-supervised NoRefER exploits the known quality
relationships between hypotheses from multiple compression levels of an ASR for
learning to rank intra-sample hypotheses by quality, which is essential for
model comparisons. The semi-supervised version also uses a referenced dataset
to improve its inter-sample quality ranking, which is crucial for selecting
potentially erroneous samples. The results indicate that NoRefER correlates
highly with reference-based metrics and their intra-sample ranks, indicating a
high potential for referenceless ASR evaluation or a/b testing.