Towards Enhancing the Reproducibility of Deep Learning Bugs: An
  Empirical Study
Mehil B. Shah Mohammad Masudur Rahman Foutse Khomh
Context: Deep learning has achieved remarkable progress in various domains.
However, like traditional software systems, deep learning systems contain bugs,
which can have severe impacts, as evidenced by crashes involving autonomous
vehicles. Despite substantial advancements in deep learning techniques, little
research has focused on reproducing deep learning bugs, which hinders resolving
them. Existing literature suggests that only 3% of deep learning bugs are
reproducible, underscoring the need for further research.
  Objective: This paper examines the reproducibility of deep learning bugs. We
identify edit actions and useful information that could improve deep learning
bug reproducibility.
  Method: First, we construct a dataset of 668 deep learning bugs from Stack
Overflow and Defects4ML across 3 frameworks and 22 architectures. Second, we
select 102 bugs using stratified sampling and try to determine their
reproducibility. While reproducing these bugs, we identify edit actions and
useful information necessary for their reproduction. Third, we used the Apriori
algorithm to identify useful information and edit actions required to reproduce
specific bug types. Finally, we conduct a user study with 22 developers to
assess the effectiveness of our findings in real-life settings.
  Results: We successfully reproduced 85 bugs and identified ten edit actions
and five useful information categories that can help us reproduce deep learning
bugs. Our findings improved bug reproducibility by 22.92% and reduced
reproduction time by 24.35% based on our user study.
  Conclusions: Our research addresses the critical issue of deep learning bug
reproducibility. Practitioners and researchers can leverage our findings to
improve deep learning bug reproducibility.