ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and
  Text Embeddings
William Brannon Suyash Fulay Hang Jiang Wonjune Kang Brandon Roy Jad Kabbara Deb Roy
We propose ConGraT(Contrastive Graph-Text pretraining), a general,
self-supervised method for jointly learning separate representations of texts
and nodes in a parent (or ``supervening'') graph, where each text is associated
with one of the nodes. Datasets fitting this paradigm are common, from social
media (users and posts), to citation networks over articles, to link graphs
over web pages. We expand on prior work by providing a general,
self-supervised, joint pretraining method, one which does not depend on
particular dataset structure or a specific task. Our method uses two separate
encoders for graph nodes and texts, which are trained to align their
representations within a common latent space. Training uses a batch-wise
contrastive learning objective inspired by prior work on joint text and image
encoding. As graphs are more structured objects than images, we also extend the
training objective to incorporate information about node similarity and
plausible next guesses in matching nodes and texts. Experiments on various
datasets reveal that ConGraT outperforms strong baselines on various downstream
tasks, including node and text category classification and link prediction.
Code and certain datasets are available at
https://github.com/wwbrannon/congrat.