Masked Graph Autoencoder with Non-discrete Bandwidths
Ziwen Zhao Yuhua Li Yixiong Zou Jiliang Tang Ruixuan Li
Masked graph autoencoders have emerged as a powerful graph self-supervised
learning method that has yet to be fully explored. In this paper, we unveil
that the existing discrete edge masking and binary link reconstruction
strategies are insufficient to learn topologically informative representations,
from the perspective of message propagation on graph neural networks. These
limitations include blocking message flows, vulnerability to over-smoothness,
and suboptimal neighborhood discriminability. Inspired by these understandings,
we explore non-discrete edge masks, which are sampled from a continuous and
dispersive probability distribution instead of the discrete Bernoulli
distribution. These masks restrict the amount of output messages for each edge,
referred to as "bandwidths". We propose a novel, informative, and effective
topological masked graph autoencoder using bandwidth masking and a layer-wise
bandwidth prediction objective. We demonstrate its powerful graph topological
learning ability both theoretically and empirically. Our proposed framework
outperforms representative baselines in both self-supervised link prediction
(improving the discrete edge reconstructors by at most 20%) and node
classification on numerous datasets, solely with a structure-learning pretext.
Our implementation is available at https://github.com/Newiz430/Bandana.