NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark
  Dataset for Generative Language Models in Norwegian
Peng Liu Lemei Zhang Terje Nissen Farup Even W. Lauvrak Jon Espen Ingvaldsen Simen Eide Jon Atle Gulla Zhirong Yang
Recent advancements in Generative Language Models (GLMs) have transformed
Natural Language Processing (NLP) by showcasing the effectiveness of the
"pre-train, prompt, and predict" paradigm in utilizing pre-trained GLM
knowledge for diverse applications. Despite their potential, these capabilities
lack adequate quantitative characterization due to the absence of comprehensive
benchmarks, particularly for low-resource languages. Existing low-resource
benchmarks focus on discriminative language models like BERT, neglecting the
evaluation of generative language models. Moreover, current benchmarks often
overlook measuring generalization performance across multiple tasks, a crucial
metric for GLMs.
  To bridge these gaps, we introduce NLEBench, a comprehensive benchmark
tailored for evaluating natural language generation capabilities in Norwegian,
a low-resource language. We use Norwegian as a case study to explore whether
current GLMs and benchmarks in mainstream languages like English can reveal the
unique characteristics of underrepresented languages. NLEBench encompasses a
suite of real-world NLP tasks ranging from news storytelling, summarization,
open-domain conversation, natural language understanding, instruction
fine-tuning, toxicity and bias evaluation, to self-curated Chain-of-Thought
investigation. It features two high-quality, human-annotated datasets: an
instruction dataset covering traditional Norwegian cultures, idioms, slang, and
special expressions, and a document-grounded multi-label dataset for topic
classification, question answering, and summarization. This paper also
introduces foundational Norwegian Generative Language Models (NorGLMs)
developed with diverse parameter scales and Transformer-based architectures.
Systematic evaluations on the proposed benchmark suite provide insights into
the capabilities and scalability of NorGLMs across various downstream tasks.