Graph-level Representation Learning with Joint-Embedding Predictive
  Architectures
Geri Skenderi Hang Li Jiliang Tang Marco Cristani
Joint-Embedding Predictive Architectures (JEPAs) have recently emerged as a
novel and powerful technique for self-supervised representation learning. They
aim to learn an energy-based model by predicting the latent representation of a
target signal $y$ from a context signal $x$. JEPAs bypass the need for data
augmentation and negative samples, which are typically required by contrastive
learning, while avoiding the overfitting issues associated with
generative-based pretraining. In this paper, we show that graph-level
representations can be effectively modeled using this paradigm and propose
Graph-JEPA, the first JEPA for the graph domain. In particular, we employ
masked modeling to learn embeddings for different subgraphs of the input graph.
To endow the representations with the implicit hierarchy that is often present
in graph-level concepts, we devise an alternative training objective that
consists of predicting the coordinates of the encoded subgraphs on the unit
hyperbola in the 2D plane. Extensive validation shows that Graph-JEPA can learn
representations that are expressive and competitive in both graph
classification and regression problems.