SHOT: Suppressing the Hessian along the Optimization Trajectory for
  Gradient-Based Meta-Learning
JunHoo Lee Jayeon Yoo Nojun Kwak
In this paper, we hypothesize that gradient-based meta-learning (GBML)
implicitly suppresses the Hessian along the optimization trajectory in the
inner loop. Based on this hypothesis, we introduce an algorithm called SHOT
(Suppressing the Hessian along the Optimization Trajectory) that minimizes the
distance between the parameters of the target and reference models to suppress
the Hessian in the inner loop. Despite dealing with high-order terms, SHOT does
not increase the computational complexity of the baseline model much. It is
agnostic to both the algorithm and architecture used in GBML, making it highly
versatile and applicable to any GBML baseline. To validate the effectiveness of
SHOT, we conduct empirical tests on standard few-shot learning tasks and
qualitatively analyze its dynamics. We confirm our hypothesis empirically and
demonstrate that SHOT outperforms the corresponding baseline. Code is available
at: https://github.com/JunHoo-Lee/SHOT