Synapse: Learning Preferential Concepts from Visual Demonstrations
Sadanand Modak Noah Patton Isil Dillig Joydeep Biswas
This paper addresses the problem of preference learning, which aims to learn
user-specific preferences (e.g., "good parking spot", "convenient drop-off
location") from visual input. Despite its similarity to learning factual
concepts (e.g., "red cube"), preference learning is a fundamentally harder
problem due to its subjective nature and the paucity of person-specific
training data. We address this problem using a new framework called Synapse,
which is a neuro-symbolic approach designed to efficiently learn preferential
concepts from limited demonstrations. Synapse represents preferences as
neuro-symbolic programs in a domain-specific language (DSL) that operates over
images, and leverages a novel combination of visual parsing, large language
models, and program synthesis to learn programs representing individual
preferences. We evaluate Synapse through extensive experimentation including a
user case study focusing on mobility-related concepts in mobile robotics and
autonomous driving. Our evaluation demonstrates that Synapse significantly
outperforms existing baselines as well as its own ablations. The code and other
details can be found on the project website https://amrl.cs.utexas.edu/synapse .