Assessing Large Language Models on Climate Information
Jannis Bulian Mike S. Schäfer Afra Amini Heidi Lam Massimiliano Ciaramita Ben Gaiarin Michelle Chen Huebscher Christian Buck Niels Mede Markus Leippold Nadine Strauss
Understanding how climate change affects us and learning about available
solutions are key steps toward empowering individuals and communities to
mitigate and adapt to it. As Large Language Models (LLMs) rise in popularity,
it is necessary to assess their capability in this domain. In this study, we
present a comprehensive evaluation framework, grounded in science communication
principles, to analyze LLM responses to climate change topics. Our framework
emphasizes both the presentational and epistemological adequacy of answers,
offering a fine-grained analysis of LLM generations. Spanning 8 dimensions, our
framework discerns up to 30 distinct issues in model outputs. The task is a
real-world example of a growing number of challenging problems where AI can
complement and lift human performance. We introduce a novel and practical
protocol for scalable oversight that uses AI Assistance and relies on raters
with relevant educational backgrounds. We evaluate several recent LLMs and
conduct a comprehensive analysis of the results, shedding light on both the
potential and the limitations of LLMs in the realm of climate communication.