Crystalformer: Infinitely Connected Attention for Periodic Structure
  Encoding
Tatsunori Taniai Ryo Igarashi Yuta Suzuki Naoya Chiba Kotaro Saito Yoshitaka Ushiku Kanta Ono
Predicting physical properties of materials from their crystal structures is
a fundamental problem in materials science. In peripheral areas such as the
prediction of molecular properties, fully connected attention networks have
been shown to be successful. However, unlike these finite atom arrangements,
crystal structures are infinitely repeating, periodic arrangements of atoms,
whose fully connected attention results in infinitely connected attention. In
this work, we show that this infinitely connected attention can lead to a
computationally tractable formulation, interpreted as neural potential
summation, that performs infinite interatomic potential summations in a deeply
learned feature space. We then propose a simple yet effective Transformer-based
encoder architecture for crystal structures called Crystalformer. Compared to
an existing Transformer-based model, the proposed model requires only 29.4% of
the number of parameters, with minimal modifications to the original
Transformer architecture. Despite the architectural simplicity, the proposed
method outperforms state-of-the-art methods for various property regression
tasks on the Materials Project and JARVIS-DFT datasets.