Pessimistic Off-Policy Multi-Objective Optimization
Shima Alizadeh Aniruddha Bhargava Karthick Gopalswamy Lalit Jain Branislav Kveton Ge Liu
Multi-objective optimization is a type of decision making problems where
multiple conflicting objectives are optimized. We study offline optimization of
multi-objective policies from data collected by an existing policy. We propose
a pessimistic estimator for the multi-objective policy values that can be
easily plugged into existing formulas for hypervolume computation and
optimized. The estimator is based on inverse propensity scores (IPS), and
improves upon a naive IPS estimator in both theory and experiments. Our
analysis is general, and applies beyond our IPS estimators and methods for
optimizing them. The pessimistic estimator can be optimized by policy gradients
and performs well in all of our experiments.