AmadeusGPT: a natural language interface for interactive animal
  behavioral analysis
Shaokai Ye Jessy Lauer Mu Zhou Alexander Mathis Mackenzie W. Mathis
The process of quantifying and analyzing animal behavior involves translating
the naturally occurring descriptive language of their actions into
machine-readable code. Yet, codifying behavior analysis is often challenging
without deep understanding of animal behavior and technical machine learning
knowledge. To limit this gap, we introduce AmadeusGPT: a natural language
interface that turns natural language descriptions of behaviors into
machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4
allow for interactive language-based queries that are potentially well suited
for making interactive behavior analysis. However, the comprehension capability
of these LLMs is limited by the context window size, which prevents it from
remembering distant conversations. To overcome the context window limitation,
we implement a novel dual-memory mechanism to allow communication between
short-term and long-term memory using symbols as context pointers for retrieval
and saving. Concretely, users directly use language-based definitions of
behavior and our augmented GPT develops code based on the core AmadeusGPT API,
which contains machine learning, computer vision, spatio-temporal reasoning,
and visualization modules. Users then can interactively refine results, and
seamlessly add new behavioral modules as needed. We benchmark AmadeusGPT and
show we can produce state-of-the-art performance on the MABE 2022 behavior
challenge tasks. Note, an end-user would not need to write any code to achieve
this. Thus, collectively AmadeusGPT presents a novel way to merge deep
biological knowledge, large-language models, and core computer vision modules
into a more naturally intelligent system. Code and demos can be found at:
https://github.com/AdaptiveMotorControlLab/AmadeusGPT.