RandAlign: A Parameter-Free Method for Regularizing Graph Convolutional
  Networks
Haimin Zhang Min Xu
Studies continually find that message-passing graph convolutional networks
suffer from the over-smoothing issue. Basically, the issue of over-smoothing
refers to the phenomenon that the learned embeddings for all nodes can become
very similar to one another and therefore are uninformative after repeatedly
applying message passing iterations. Intuitively, we can expect the generated
embeddings become smooth asymptotically layerwisely, that is each layer of
graph convolution generates a smoothed version of embeddings as compared to
that generated by the previous layer. Based on this intuition, we propose
RandAlign, a stochastic regularization method for graph convolutional networks.
The idea of RandAlign is to randomly align the learned embedding for each node
with that of the previous layer using randomly interpolation in each graph
convolution layer. Through alignment, the smoothness of the generated
embeddings is explicitly reduced. To better maintain the benefit yielded by the
graph convolution, in the alignment step we introduce to first scale the
embedding of the previous layer to the same norm as the generated embedding and
then perform random interpolation for aligning the generated embedding.
RandAlign is a parameter-free method and can be directly applied without
introducing additional trainable weights or hyper-parameters. We experimentally
evaluate RandAlign on different graph domain tasks on seven benchmark datasets.
The experimental results show that RandAlign is a general method that improves
the generalization performance of various graph convolutional network models
and also improves the numerical stability of optimization, advancing the state
of the art performance for graph representation learning.