Revealing data leakage in protein interaction benchmarks
Anton Bushuiev Roman Bushuiev Jiri Sedlar Tomas Pluskal Jiri Damborsky Stanislav Mazurenko Josef Sivic
In recent years, there has been remarkable progress in machine learning for
protein-protein interactions. However, prior work has predominantly focused on
improving learning algorithms, with less attention paid to evaluation
strategies and data preparation. Here, we demonstrate that further development
of machine learning methods may be hindered by the quality of existing
train-test splits. Specifically, we find that commonly used splitting
strategies for protein complexes, based on protein sequence or metadata
similarity, introduce major data leakage. This may result in overoptimistic
evaluation of generalization, as well as unfair benchmarking of the models,
biased towards assessing their overfitting capacity rather than practical
utility. To overcome the data leakage, we recommend constructing data splits
based on 3D structural similarity of protein-protein interfaces and suggest
corresponding algorithms. We believe that addressing the data leakage problem
is critical for further progress in this research area.