Training-time Neuron Alignment through Permutation Subspace for
  Improving Linear Mode Connectivity and Model Fusion
Zexi Li Zhiqi Li Jie Lin Tao Shen Tao Lin Chao Wu
In deep learning, stochastic gradient descent often yields functionally
similar yet widely scattered solutions in the weight space even under the same
initialization, causing barriers in the Linear Mode Connectivity (LMC)
landscape. Overcoming these barriers is crucial for understanding deep learning
dynamics and enhancing model-fusion algorithms. Previous studies highlight the
role of permutation symmetry in reducing post-training barriers through network
permutation. However, these post-hoc methods, demanding extra computations, are
less effective for larger, complex models (e.g., ViT, LLM) due to numerous
permutation matrices. Thus, in this paper, we study training-time neuron
alignment. Our hypothesis suggests that training-time permutation subspace can
reduce LMC barriers for free. We find that pruning at initialization supports
this. Beyond pruning, we introduce TNA-PFN, a simple yet lossless algorithm
using a partial gradient mask during training. TNA-PFN is theoretically and
empirically validated for reducing LMC barriers. It excels in wide model fusion
applications, especially in federated learning, two algorithms based on TNA-FPN
that are proposed to show its prospects even under heterogeneous datasets.
Moreover, TNA-PFN can enhance the generalization of model soup for vision
transformers and ColD fusion for pretrained language models.