Constrained Meta-Reinforcement Learning for Adaptable Safety Guarantee
  with Differentiable Convex Programming
Minjae Cho Chuangchuang Sun
Despite remarkable achievements in artificial intelligence, the deployability
of learning-enabled systems in high-stakes real-world environments still faces
persistent challenges. For example, in safety-critical domains like autonomous
driving, robotic manipulation, and healthcare, it is crucial not only to
achieve high performance but also to comply with given constraints.
Furthermore, adaptability becomes paramount in non-stationary domains, where
environmental parameters are subject to change. While safety and adaptability
are recognized as key qualities for the new generation of AI, current
approaches have not demonstrated effective adaptable performance in constrained
settings. Hence, this paper breaks new ground by studying the unique challenges
of ensuring safety in non-stationary environments by solving constrained
problems through the lens of the meta-learning approach (learning-to-learn).
While unconstrained meta-learning al-ready encounters complexities in
end-to-end differentiation of the loss due to the bi-level nature, its
constrained counterpart introduces an additional layer of difficulty, since the
constraints imposed on task-level updates complicate the differentiation
process. To address the issue, we first employ successive convex-constrained
policy updates across multiple tasks with differentiable convexprogramming,
which allows meta-learning in constrained scenarios by enabling end-to-end
differentiation. This approach empowers the agent to rapidly adapt to new tasks
under non-stationarity while ensuring compliance with safety constraints.