PHYDI: Initializing Parameterized Hypercomplex Neural Networks as
  Identity Functions
Matteo Mancanelli Eleonora Grassucci Aurelio Uncini Danilo Comminiello
Neural models based on hypercomplex algebra systems are growing and
prolificating for a plethora of applications, ranging from computer vision to
natural language processing. Hand in hand with their adoption, parameterized
hypercomplex neural networks (PHNNs) are growing in size and no techniques have
been adopted so far to control their convergence at a large scale. In this
paper, we study PHNNs convergence and propose parameterized hypercomplex
identity initialization (PHYDI), a method to improve their convergence at
different scales, leading to more robust performance when the number of layers
scales up, while also reaching the same performance with fewer iterations. We
show the effectiveness of this approach in different benchmarks and with common
PHNNs with ResNets- and Transformer-based architecture. The code is available
at https://github.com/ispamm/PHYDI.