Alfred: A System for Prompted Weak Supervision
Peilin Yu Stephen H. Bach
Alfred is the first system for programmatic weak supervision (PWS) that
creates training data for machine learning by prompting. In contrast to typical
PWS systems where weak supervision sources are programs coded by experts,
Alfred enables users to encode their subject matter expertise via natural
language prompts for language and vision-language models. Alfred provides a
simple Python interface for the key steps of this emerging paradigm, with a
high-throughput backend for large-scale data labeling. Users can quickly
create, evaluate, and refine their prompt-based weak supervision sources; map
the results to weak labels; and resolve their disagreements with a label model.
Alfred enables a seamless local development experience backed by models served
from self-managed computing clusters. It automatically optimizes the execution
of prompts with optimized batching mechanisms. We find that this optimization
improves query throughput by 2.9x versus a naive approach. We present two
example use cases demonstrating Alfred on YouTube comment spam detection and
pet breeds classification. Alfred is open source, available at
https://github.com/BatsResearch/alfred.