BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
Odhran O'Donoghue Aleksandar Shtedritski John Ginger Ralph Abboud Ali Essa Ghareeb Justin Booth Samuel G Rodriques
The ability to automatically generate accurate protocols for scientific
experiments would represent a major step towards the automation of science.
Large Language Models (LLMs) have impressive capabilities on a wide range of
tasks, such as question answering and the generation of coherent text and code.
However, LLMs can struggle with multi-step problems and long-term planning,
which are crucial for designing scientific experiments. Moreover, evaluation of
the accuracy of scientific protocols is challenging, because experiments can be
described correctly in many different ways, require expert knowledge to
evaluate, and cannot usually be executed automatically. Here we present an
automatic evaluation framework for the task of planning experimental protocols,
and we introduce BioProt: a dataset of biology protocols with corresponding
pseudocode representations. To measure performance on generating scientific
protocols, we use an LLM to convert a natural language protocol into
pseudocode, and then evaluate an LLM's ability to reconstruct the pseudocode
from a high-level description and a list of admissible pseudocode functions. We
evaluate GPT-3 and GPT-4 on this task and explore their robustness. We
externally validate the utility of pseudocode representations of text by
generating accurate novel protocols using retrieved pseudocode, and we run a
generated protocol successfully in our biological laboratory. Our framework is
extensible to the evaluation and improvement of language model planning
abilities in other areas of science or other areas that lack automatic
evaluation.