Traversing Between Modes in Function Space for Fast Ensembling
EungGu Yun Hyungi Lee Giung Nam Juho Lee
Deep ensemble is a simple yet powerful way to improve the performance of deep
neural networks. Under this motivation, recent works on mode connectivity have
shown that parameters of ensembles are connected by low-loss subspaces, and one
can efficiently collect ensemble parameters in those subspaces. While this
provides a way to efficiently train ensembles, for inference, multiple forward
passes should still be executed using all the ensemble parameters, which often
becomes a serious bottleneck for real-world deployment. In this work, we
propose a novel framework to reduce such costs. Given a low-loss subspace
connecting two modes of a neural network, we build an additional neural network
that predicts the output of the original neural network evaluated at a certain
point in the low-loss subspace. The additional neural network, which we call a
"bridge", is a lightweight network that takes minimal features from the
original network and predicts outputs for the low-loss subspace without forward
passes through the original network. We empirically demonstrate that we can
indeed train such bridge networks and significantly reduce inference costs with
the help of bridge networks.