IW-GAE: Importance weighted group accuracy estimation for improved
  calibration and model selection in unsupervised domain adaptation
Taejong Joo Diego Klabjan
Reasoning about a model's accuracy on a test sample from its confidence is a
central problem in machine learning, being connected to important applications
such as uncertainty representation, model selection, and exploration. While
these connections have been well-studied in the i.i.d. settings, distribution
shifts pose significant challenges to the traditional methods. Therefore, model
calibration and model selection remain challenging in the unsupervised domain
adaptation problem--a scenario where the goal is to perform well in a
distribution shifted domain without labels. In this work, we tackle
difficulties coming from distribution shifts by developing a novel importance
weighted group accuracy estimator. Specifically, we formulate an optimization
problem for finding an importance weight that leads to an accurate group
accuracy estimation in the distribution shifted domain with theoretical
analyses. Extensive experiments show the effectiveness of group accuracy
estimation on model calibration and model selection. Our results emphasize the
significance of group accuracy estimation for addressing challenges in
unsupervised domain adaptation, as an orthogonal improvement direction with
improving transferability of accuracy.