Geodesic Mode Connectivity
Charlie Tan Theodore Long Sarah Zhao Rudolf Laine
Mode connectivity is a phenomenon where trained models are connected by a
path of low loss. We reframe this in the context of Information Geometry, where
neural networks are studied as spaces of parameterized distributions with
curved geometry. We hypothesize that shortest paths in these spaces, known as
geodesics, correspond to mode-connecting paths in the loss landscape. We
propose an algorithm to approximate geodesics and demonstrate that they achieve
mode connectivity.