Machine Teaching for Building Modular AI Agents based on Zero-shot
  Learners
Karan Taneja Ashok Goel
The recent advances in large language models (LLMs) have led to the creation
of many modular AI agents. These agents employ LLMs as zero-shot learners to
perform sub-tasks in order to solve complex tasks set forth by human users. We
propose an approach to enhance the robustness and performance of modular AI
agents that utilize LLMs as zero-shot learners. Our iterative machine teaching
method offers an efficient way to teach AI agents over time with limited human
feedback, addressing the limit posed by the quality of zero-shot learning. We
advocate leveraging the data traces from initial deployments and outputs or
annotations from the zero-shot learners to train smaller and task-specific
substitute models which can reduce both the monetary costs and environmental
impact. Our machine teaching process avails human expertise to correct examples
with a high likelihood of misannotations. Results on three tasks, common to
conversational AI agents, show that close-to-oracle performance can be achieved
with supervision on 20-70% of the dataset depending upon the complexity of the
task and performance of zero-shot learners.