Overcoming Saturation in Density Ratio Estimation by Iterated
  Regularization
Lukas Gruber Markus Holzleitner Johannes Lehner Sepp Hochreiter Werner Zellinger
Estimating the ratio of two probability densities from finitely many samples,
is a central task in machine learning and statistics. In this work, we show
that a large class of kernel methods for density ratio estimation suffers from
error saturation, which prevents algorithms from achieving fast error
convergence rates on highly regular learning problems. To resolve saturation,
we introduce iterated regularization in density ratio estimation to achieve
fast error rates. Our methods outperform its non-iteratively regularized
versions on benchmarks for density ratio estimation as well as on large-scale
evaluations for importance-weighted ensembling of deep unsupervised domain
adaptation models.