A Step-by-step Introduction to the Implementation of Automatic
  Differentiation
Yu-Hsueh Fang He-Zhe Lin Jie-Jyun Liu Chih-Jen Lin
Automatic differentiation is a key component in deep learning. This topic is
well studied and excellent surveys such as Baydin et al. (2018) have been
available to clearly describe the basic concepts. Further, sophisticated
implementations of automatic differentiation are now an important part of
popular deep learning frameworks. However, it is difficult, if not impossible,
to directly teach students the implementation of existing systems due to the
complexity. On the other hand, if the teaching stops at the basic concept,
students fail to sense the realization of an implementation. For example, we
often mention the computational graph in teaching automatic differentiation,
but students wonder how to implement and use it. In this document, we partially
fill the gap by giving a step by step introduction of implementing a simple
automatic differentiation system. We streamline the mathematical concepts and
the implementation. Further, we give the motivation behind each implementation
detail, so the whole setting becomes very natural.