Lessons in Reproducibility: Insights from NLP Studies in Materials
  Science
Xiangyun Lei Edward Kim Viktoriia Baibakova Shijing Sun
Natural Language Processing (NLP), a cornerstone field within artificial
intelligence, has been increasingly utilized in the field of materials science
literature. Our study conducts a reproducibility analysis of two pioneering
works within this domain: "Machine-learned and codified synthesis parameters of
oxide materials" by Kim et al., and "Unsupervised word embeddings capture
latent knowledge from materials science literature" by Tshitoyan et al. We aim
to comprehend these studies from a reproducibility perspective, acknowledging
their significant influence on the field of materials informatics, rather than
critiquing them. Our study indicates that both papers offered thorough
workflows, tidy and well-documented codebases, and clear guidance for model
evaluation. This makes it easier to replicate their results successfully and
partially reproduce their findings. In doing so, they set commendable standards
for future materials science publications to aspire to. However, our analysis
also highlights areas for improvement such as to provide access to training
data where copyright restrictions permit, more transparency on model
architecture and the training process, and specifications of software
dependency versions. We also cross-compare the word embedding models between
papers, and find that some key differences in reproducibility and
cross-compatibility are attributable to design choices outside the bounds of
the models themselves. In summary, our study appreciates the benchmark set by
these seminal papers while advocating for further enhancements in research
reproducibility practices in the field of NLP for materials science. This
balance of understanding and continuous improvement will ultimately propel the
intersecting domains of NLP and materials science literature into a future of
exciting discoveries.