On Data Imbalance in Molecular Property Prediction with Pre-training
Limin Wang Masatoshi Hanai Toyotaro Suzumura Shun Takashige Kenjiro Taura
Revealing and analyzing the various properties of materials is an essential
and critical issue in the development of materials, including batteries,
semiconductors, catalysts, and pharmaceuticals. Traditionally, these properties
have been determined through theoretical calculations and simulations. However,
it is not practical to perform such calculations on every single candidate
material. Recently, a combination method of the theoretical calculation and
machine learning has emerged, that involves training machine learning models on
a subset of theoretical calculation results to construct a surrogate model that
can be applied to the remaining materials. On the other hand, a technique
called pre-training is used to improve the accuracy of machine learning models.
Pre-training involves training the model on pretext task, which is different
from the target task, before training the model on the target task. This
process aims to extract the input data features, stabilizing the learning
process and improving its accuracy. However, in the case of molecular property
prediction, there is a strong imbalance in the distribution of input data and
features, which may lead to biased learning towards frequently occurring data
during pre-training. In this study, we propose an effective pre-training method
that addresses the imbalance in input data. We aim to improve the final
accuracy by modifying the loss function of the existing representative
pre-training method, node masking, to compensate the imbalance. We have
investigated and assessed the impact of our proposed imbalance compensation on
pre-training and the final prediction accuracy through experiments and
evaluations using benchmark of molecular property prediction models.