Batch Active Learning of Reward Functions from Human Preferences
Erdem Bıyık Nima Anari Dorsa Sadigh
Data generation and labeling are often expensive in robot learning.
Preference-based learning is a concept that enables reliable labeling by
querying users with preference questions. Active querying methods are commonly
employed in preference-based learning to generate more informative data at the
expense of parallelization and computation time. In this paper, we develop a
set of novel algorithms, batch active preference-based learning methods, that
enable efficient learning of reward functions using as few data samples as
possible while still having short query generation times and also retaining
parallelizability. We introduce a method based on determinantal point processes
(DPP) for active batch generation and several heuristic-based alternatives.
Finally, we present our experimental results for a variety of robotics tasks in
simulation. Our results suggest that our batch active learning algorithm
requires only a few queries that are computed in a short amount of time. We
showcase one of our algorithms in a study to learn human users' preferences.