Towards Constituting Mathematical Structures for Learning to Optimize
Jialin Liu Xiaohan Chen Zhangyang Wang Wotao Yin HanQin Cai
Learning to Optimize (L2O), a technique that utilizes machine learning to
learn an optimization algorithm automatically from data, has gained arising
attention in recent years. A generic L2O approach parameterizes the iterative
update rule and learns the update direction as a black-box network. While the
generic approach is widely applicable, the learned model can overfit and may
not generalize well to out-of-distribution test sets. In this paper, we derive
the basic mathematical conditions that successful update rules commonly
satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired
structure that is broadly applicable and generalized well to
out-of-distribution problems. Numerical simulations validate our theoretical
findings and demonstrate the superior empirical performance of the proposed L2O
model.