Adversarial Constrained Bidding via Minimax Regret Optimization with
  Causality-Aware Reinforcement Learning
Haozhe Wang Chao Du Panyan Fang Li He Liang Wang Bo Zheng
The proliferation of the Internet has led to the emergence of online
advertising, driven by the mechanics of online auctions. In these repeated
auctions, software agents participate on behalf of aggregated advertisers to
optimize for their long-term utility. To fulfill the diverse demands, bidding
strategies are employed to optimize advertising objectives subject to different
spending constraints. Existing approaches on constrained bidding typically rely
on i.i.d. train and test conditions, which contradicts the adversarial nature
of online ad markets where different parties possess potentially conflicting
objectives. In this regard, we explore the problem of constrained bidding in
adversarial bidding environments, which assumes no knowledge about the
adversarial factors. Instead of relying on the i.i.d. assumption, our insight
is to align the train distribution of environments with the potential test
distribution meanwhile minimizing policy regret. Based on this insight, we
propose a practical Minimax Regret Optimization (MiRO) approach that
interleaves between a teacher finding adversarial environments for tutoring and
a learner meta-learning its policy over the given distribution of environments.
In addition, we pioneer to incorporate expert demonstrations for learning
bidding strategies. Through a causality-aware policy design, we improve upon
MiRO by distilling knowledge from the experts. Extensive experiments on both
industrial data and synthetic data show that our method, MiRO with
Causality-aware reinforcement Learning (MiROCL), outperforms prior methods by
over 30%.