Generating Pragmatic Examples to Train Neural Program Synthesizers
Saujas Vaduguru Daniel Fried Yewen Pu
Programming-by-example is the task of synthesizing a program that is
consistent with a set of user-provided input-output examples. As examples are
often an under-specification of one's intent, a good synthesizer must choose
the intended program from the many that are consistent with the given set of
examples. Prior work frames program synthesis as a cooperative game between a
listener (that synthesizes programs) and a speaker (a user choosing examples),
and shows that models of computational pragmatic inference are effective in
choosing the user intended programs. However, these models require
counterfactual reasoning over a large set of programs and examples, which is
infeasible in realistic program spaces. In this paper, we propose a novel way
to amortize this search with neural networks. We sample pairs of programs and
examples via self-play between listener and speaker models, and use pragmatic
inference to choose informative training examples from this sample.We then use
the informative dataset to train models to improve the synthesizer's ability to
disambiguate user-provided examples without human supervision. We validate our
method on the challenging task of synthesizing regular expressions from example
strings, and find that our method (1) outperforms models trained without
choosing pragmatic examples by 23% (a 51% relative increase) (2) matches the
performance of supervised learning on a dataset of pragmatic examples provided
by humans, despite using no human data in training.