Revealing the Unwritten: Visual Investigation of Beam Search Trees to
  Address Language Model Prompting Challenges
Thilo Spinner Rebecca Kehlbeck Rita Sevastjanova Tobias Stähle Daniel A. Keim Oliver Deussen Andreas Spitz Mennatallah El-Assady
The growing popularity of generative language models has amplified interest
in interactive methods to guide model outputs. Prompt refinement is considered
one of the most effective means to influence output among these methods. We
identify several challenges associated with prompting large language models,
categorized into data- and model-specific, linguistic, and socio-linguistic
challenges. A comprehensive examination of model outputs, including runner-up
candidates and their corresponding probabilities, is needed to address these
issues. The beam search tree, the prevalent algorithm to sample model outputs,
can inherently supply this information. Consequently, we introduce an
interactive visual method for investigating the beam search tree, facilitating
analysis of the decisions made by the model during generation. We
quantitatively show the value of exposing the beam search tree and present five
detailed analysis scenarios addressing the identified challenges. Our
methodology validates existing results and offers additional insights.