AlloyBERT: Alloy Property Prediction with Large Language Models
Akshat Chaudhari Chakradhar Guntuboina Hongshuo Huang Amir Barati Farimani
The pursuit of novel alloys tailored to specific requirements poses
significant challenges for researchers in the field. This underscores the
importance of developing predictive techniques for essential physical
properties of alloys based on their chemical composition and processing
parameters. This study introduces AlloyBERT, a transformer encoder-based model
designed to predict properties such as elastic modulus and yield strength of
alloys using textual inputs. Leveraging the pre-trained RoBERTa encoder model
as its foundation, AlloyBERT employs self-attention mechanisms to establish
meaningful relationships between words, enabling it to interpret human-readable
input and predict target alloy properties. By combining a tokenizer trained on
our textual data and a RoBERTa encoder pre-trained and fine-tuned for this
specific task, we achieved a mean squared error (MSE) of 0.00015 on the Multi
Principal Elemental Alloys (MPEA) data set and 0.00611 on the Refractory Alloy
Yield Strength (RAYS) dataset. This surpasses the performance of shallow
models, which achieved a best-case MSE of 0.00025 and 0.0076 on the MPEA and
RAYS datasets respectively. Our results highlight the potential of language
models in material science and establish a foundational framework for
text-based prediction of alloy properties that does not rely on complex
underlying representations, calculations, or simulations.