Structured Thoughts Automaton: First Formalized Execution Model for
  Auto-Regressive Language Models
Tristan Vanderbruggen Chunhua Liao Peter Pirkelbauer Pei-Hung Lin
In recent months, Language Models (LMs) have become a part of daily
discourse, with focus on OpenAI and the potential of Artificial General
Intelligence (AGI). Furthermore, the leaking of LLama's weights to the public
has led to an influx of innovations demonstrating the impressive capabilities
of generative LMs. While we believe that AGI is still a distant goal, we
recognize the potential of LMs in solving tasks such as searching complex
documents, compiling reports with basic analysis, and providing assistance in
problem-solving. In this paper, we propose formalizing the execution model of
language models. We investigate current execution models, to find that this
formalism has received little attention, and present our contribution: the
first formalized execution model for LMs. We introduce a new algorithm for
sampling the predictions of LMs, which we use to build a reliable and
inspectable execution model. We introduce a low-level language to write
"cognitive program" for this execution model. We hope to shed light on the need
for execution models for LMs and encourage further research in this area.