On the Initialization of Graph Neural Networks
Jiahang Li Yakun Song Xiang Song David Paul Wipf
Graph Neural Networks (GNNs) have displayed considerable promise in graph
representation learning across various applications. The core learning process
requires the initialization of model weight matrices within each GNN layer,
which is typically accomplished via classic initialization methods such as
Xavier initialization. However, these methods were originally motivated to
stabilize the variance of hidden embeddings and gradients across layers of
Feedforward Neural Networks (FNNs) and Convolutional Neural Networks (CNNs) to
avoid vanishing gradients and maintain steady information flow. In contrast,
within the GNN context classical initializations disregard the impact of the
input graph structure and message passing on variance. In this paper, we
analyze the variance of forward and backward propagation across GNN layers and
show that the variance instability of GNN initializations comes from the
combined effect of the activation function, hidden dimension, graph structure
and message passing. To better account for these influence factors, we propose
a new initialization method for Variance Instability Reduction within GNN
Optimization (Virgo), which naturally tends to equate forward and backward
variances across successive layers. We conduct comprehensive experiments on 15
datasets to show that Virgo can lead to superior model performance and more
stable variance at initialization on node classification, link prediction and
graph classification tasks. Codes are in
https://github.com/LspongebobJH/virgo_icml2023.