Improved Order Analysis and Design of Exponential Integrator for
  Diffusion Models Sampling
Qinsheng Zhang Jiaming Song Yongxin Chen
Efficient differential equation solvers have significantly reduced the
sampling time of diffusion models (DMs) while retaining high sampling quality.
Among these solvers, exponential integrators (EI) have gained prominence by
demonstrating state-of-the-art performance. However, existing high-order
EI-based sampling algorithms rely on degenerate EI solvers, resulting in
inferior error bounds and reduced accuracy in contrast to the theoretically
anticipated results under optimal settings. This situation makes the sampling
quality extremely vulnerable to seemingly innocuous design choices such as
timestep schedules. For example, an inefficient timestep scheduler might
necessitate twice the number of steps to achieve a quality comparable to that
obtained through carefully optimized timesteps. To address this issue, we
reevaluate the design of high-order differential solvers for DMs. Through a
thorough order analysis, we reveal that the degeneration of existing high-order
EI solvers can be attributed to the absence of essential order conditions. By
reformulating the differential equations in DMs and capitalizing on the theory
of exponential integrators, we propose refined EI solvers that fulfill all the
order conditions, which we designate as Refined Exponential Solver (RES).
Utilizing these improved solvers, RES exhibits more favorable error bounds
theoretically and achieves superior sampling efficiency and stability in
practical applications. For instance, a simple switch from the single-step
DPM-Solver++ to our order-satisfied RES solver when Number of Function
Evaluations (NFE) $=9$, results in a reduction of numerical defects by $25.2\%$
and FID improvement of $25.4\%$ (16.77 vs 12.51) on a pre-trained ImageNet
diffusion model.