Harnessing Orthogonality to Train Low-Rank Neural Networks
Daniel Coquelin Katharina Flügel Marie Weiel Nicholas Kiefer Charlotte Debus Achim Streit Markus Götz
This study explores the learning dynamics of neural networks by analyzing the
singular value decomposition (SVD) of their weights throughout training. Our
investigation reveals that an orthogonal basis within each multidimensional
weight's SVD representation stabilizes during training. Building upon this, we
introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel
training method exploiting the intrinsic orthogonality of neural networks.
OIALR seamlessly integrates into existing training workflows with minimal
accuracy loss, as demonstrated by benchmarking on various datasets and
well-established network architectures. With appropriate hyperparameter tuning,
OIALR can surpass conventional training setups, including those of
state-of-the-art models.