An End-to-End System for Reproducibility Assessment of Source Code
  Repositories via Their Readmes
Eyüp Kaan Akdeniz Selma Tekir Malik Nizar Asad Al Hinnawi
Increased reproducibility of machine learning research has been a driving
force for dramatic improvements in learning performances. The scientific
community further fosters this effort by including reproducibility ratings in
reviewer forms and considering them as a crucial factor for the overall
evaluation of papers. Accompanying source code is not sufficient to make a work
reproducible. The shared codes should meet the ML reproducibility checklist as
well. This work aims to support reproducibility evaluations of papers with
source codes. We propose an end-to-end system that operates on the Readme file
of the source code repositories. The system checks the compliance of a given
Readme to a template proposed by a widely used platform for sharing source
codes of research. Our system generates scores based on a custom function to
combine section scores. We also train a hierarchical transformer model to
assign a class label to a given Readme. The experimental results show that the
section similarity-based system performs better than the hierarchical
transformer. Moreover, it has an advantage regarding explainability since one
can directly relate the score to the sections of Readme files.