Tasks People Prompt: A Taxonomy of LLM Downstream Tasks in Software
  Verification and Falsification Approaches
Víctor A. Braberman Flavia Bonomo-Braberman Yiannis Charalambous Juan G. Colonna Lucas C. Cordeiro Rosiane de Freitas
Prompting has become one of the main approaches to leverage emergent
capabilities of Large Language Models [Brown et al. NeurIPS 2020, Wei et al.
TMLR 2022, Wei et al. NeurIPS 2022]. During the last year, researchers and
practitioners have been playing with prompts to see how to make the most of
LLMs. By homogeneously dissecting 80 papers, we investigate in deep how
software testing and verification research communities have been abstractly
architecting their LLM-enabled solutions. More precisely, first, we want to
validate whether downstream tasks are an adequate concept to convey the
blueprint of prompt-based solutions. We also aim at identifying number and
nature of such tasks in solutions. For such goal, we develop a novel downstream
task taxonomy that enables pinpointing some engineering patterns in a rather
varied spectrum of Software Engineering problems that encompasses testing,
fuzzing, debugging, vulnerability detection, static analysis and program
verification approaches.