The Elements of Differentiable Programming
Mathieu Blondel Vincent Roulet
Artificial intelligence has recently experienced remarkable advances, fueled
by large models, vast datasets, accelerated hardware, and, last but not least,
the transformative power of differentiable programming. This new programming
paradigm enables end-to-end differentiation of complex computer programs
(including those with control flows and data structures), making gradient-based
optimization of program parameters possible. As an emerging paradigm,
differentiable programming builds upon several areas of computer science and
applied mathematics, including automatic differentiation, graphical models,
optimization and statistics. This book presents a comprehensive review of the
fundamental concepts useful for differentiable programming. We adopt two main
perspectives, that of optimization and that of probability, with clear
analogies between the two. Differentiable programming is not merely the
differentiation of programs, but also the thoughtful design of programs
intended for differentiation. By making programs differentiable, we inherently
introduce probability distributions over their execution, providing a means to
quantify the uncertainty associated with program outputs.