Automated Statistical Model Discovery with Language Models
Michael Y. Li Emily B. Fox Noah D. Goodman
Statistical model discovery involves a challenging search over a vast space
of models subject to domain-specific modeling constraints. Efficiently
searching over this space requires human expertise in modeling and the problem
domain. Motivated by the domain knowledge and programming capabilities of large
language models (LMs), we introduce a method for language model driven
automated statistical model discovery. We cast our automated procedure within
the framework of Box's Loop: the LM iterates between proposing statistical
models represented as probabilistic programs, acting as a modeler, and
critiquing those models, acting as a domain expert. By leveraging LMs, we do
not have to define a domain-specific language of models or design a handcrafted
search procedure, key restrictions of previous systems. We evaluate our method
in three common settings in probabilistic modeling: searching within a
restricted space of models, searching over an open-ended space, and improving
classic models under natural language constraints (e.g., this model should be
interpretable to an ecologist). Our method matches the performance of previous
systems, identifies models on par with human expert designed models, and
extends classic models in interpretable ways. Our results highlight the promise
of LM driven model discovery.