Adjustable Robust Reinforcement Learning for Online 3D Bin Packing
Yuxin Pan Yize Chen Fangzhen Lin
Designing effective policies for the online 3D bin packing problem (3D-BPP)
has been a long-standing challenge, primarily due to the unpredictable nature
of incoming box sequences and stringent physical constraints. While current
deep reinforcement learning (DRL) methods for online 3D-BPP have shown
promising results in optimizing average performance over an underlying box
sequence distribution, they often fail in real-world settings where some
worst-case scenarios can materialize. Standard robust DRL algorithms tend to
overly prioritize optimizing the worst-case performance at the expense of
performance under normal problem instance distribution. To address these
issues, we first introduce a permutation-based attacker to investigate the
practical robustness of both DRL-based and heuristic methods proposed for
solving online 3D-BPP. Then, we propose an adjustable robust reinforcement
learning (AR2L) framework that allows efficient adjustment of robustness
weights to achieve the desired balance of the policy's performance in average
and worst-case environments. Specifically, we formulate the objective function
as a weighted sum of expected and worst-case returns, and derive the lower
performance bound by relating to the return under a mixture dynamics. To
realize this lower bound, we adopt an iterative procedure that searches for the
associated mixture dynamics and improves the corresponding policy. We integrate
this procedure into two popular robust adversarial algorithms to develop the
exact and approximate AR2L algorithms. Experiments demonstrate that AR2L is
versatile in the sense that it improves policy robustness while maintaining an
acceptable level of performance for the nominal case.