Is Self-Supervised Pretraining Good for Extrapolation in Molecular
  Property Prediction?
Shun Takashige Masatoshi Hanai Toyotaro Suzumura Limin Wang Kenjiro Taura
The prediction of material properties plays a crucial role in the development
and discovery of materials in diverse applications, such as batteries,
semiconductors, catalysts, and pharmaceuticals. Recently, there has been a
growing interest in employing data-driven approaches by using machine learning
technologies, in combination with conventional theoretical calculations. In
material science, the prediction of unobserved values, commonly referred to as
extrapolation, is particularly critical for property prediction as it enables
researchers to gain insight into materials beyond the limits of available data.
However, even with the recent advancements in powerful machine learning models,
accurate extrapolation is still widely recognized as a significantly
challenging problem. On the other hand, self-supervised pretraining is a
machine learning technique where a model is first trained on unlabeled data
using relatively simple pretext tasks before being trained on labeled data for
target tasks. As self-supervised pretraining can effectively utilize material
data without observed property values, it has the potential to improve the
model's extrapolation ability. In this paper, we clarify how such
self-supervised pretraining can enhance extrapolation performance.We propose an
experimental framework for the demonstration and empirically reveal that while
models were unable to accurately extrapolate absolute property values,
self-supervised pretraining enables them to learn relative tendencies of
unobserved property values and improve extrapolation performance.