Patton: Language Model Pretraining on Text-Rich Networks
Bowen Jin Wentao Zhang Yu Zhang Yu Meng Xinyang Zhang Qi Zhu Jiawei Han
A real-world text corpus sometimes comprises not only text documents but also
semantic links between them (e.g., academic papers in a bibliographic network
are linked by citations and co-authorships). Text documents and semantic
connections form a text-rich network, which empowers a wide range of downstream
tasks such as classification and retrieval. However, pretraining methods for
such structures are still lacking, making it difficult to build one generic
model that can be adapted to various tasks on text-rich networks. Current
pretraining objectives, such as masked language modeling, purely model texts
and do not take inter-document structure information into consideration. To
this end, we propose our PretrAining on TexT-Rich NetwOrk framework Patton.
Patton includes two pretraining strategies: network-contextualized masked
language modeling and masked node prediction, to capture the inherent
dependency between textual attributes and network structure. We conduct
experiments on four downstream tasks in five datasets from both academic and
e-commerce domains, where Patton outperforms baselines significantly and
consistently.