AIR-Bench: Benchmarking Large Audio-Language Models via Generative
  Comprehension
Qian Yang Jin Xu Wenrui Liu Yunfei Chu Ziyue Jiang Xiaohuan Zhou Yichong Leng Yuanjun Lv Zhou Zhao Chang Zhou Jingren Zhou
Recently, instruction-following audio-language models have received broad
attention for human-audio interaction. However, the absence of benchmarks
capable of evaluating audio-centric interaction capabilities has impeded
advancements in this field. Previous models primarily focus on assessing
different fundamental tasks, such as Automatic Speech Recognition (ASR), and
lack an assessment of the open-ended generative capabilities centered around
audio. Thus, it is challenging to track the progression in the Large
Audio-Language Models (LALMs) domain and to provide guidance for future
improvement. In this paper, we introduce AIR-Bench (\textbf{A}udio
\textbf{I}nst\textbf{R}uction \textbf{Bench}mark), the first benchmark designed
to evaluate the ability of LALMs to understand various types of audio signals
(including human speech, natural sounds, and music), and furthermore, to
interact with humans in the textual format. AIR-Bench encompasses two
dimensions: \textit{foundation} and \textit{chat} benchmarks. The former
consists of 19 tasks with approximately 19k single-choice questions, intending
to inspect the basic single-task ability of LALMs. The latter one contains 2k
instances of open-ended question-and-answer data, directly assessing the
comprehension of the model on complex audio and its capacity to follow
instructions. Both benchmarks require the model to generate hypotheses
directly. We design a unified framework that leverages advanced language
models, such as GPT-4, to evaluate the scores of generated hypotheses given the
meta-information of the audio. Experimental results demonstrate a high level of
consistency between GPT-4-based evaluation and human evaluation. By revealing
the limitations of existing LALMs through evaluation results, AIR-Bench can
provide insights into the direction of future research.