Rethink Model Re-Basin and the Linear Mode Connectivity
Xingyu Qu Samuel Horvath
Recent studies suggest that with sufficiently wide models, most SGD solutions
can, up to permutation, converge into the same basin. This phenomenon, known as
the model re-basin regime, has significant implications for model averaging.
However, current re-basin strategies are limited in effectiveness due to a lack
of comprehensive understanding of underlying mechanisms. Addressing this gap,
our work revisits standard practices and uncovers the frequent inadequacies of
existing matching algorithms, which we show can be mitigated through proper
re-normalization. By introducing a more direct analytical approach, we expose
the interaction between matching algorithms and re-normalization processes.
This perspective not only clarifies and refines previous findings but also
facilitates novel insights. For instance, it connects the linear mode
connectivity to pruning, motivating a lightweight yet effective post-pruning
plug-in that can be directly merged with any existing pruning techniques. Our
implementation is available at https://github.com/XingyuQu/rethink-re-basin.