Optimal Sample Selection Through Uncertainty Estimation and Its
  Application in Deep Learning
Yong Lin Chen Liu Chenlu Ye Qing Lian Yuan Yao Tong Zhang
Modern deep learning heavily relies on large labeled datasets, which often
comse with high costs in terms of both manual labeling and computational
resources. To mitigate these challenges, researchers have explored the use of
informative subset selection techniques, including coreset selection and active
learning. Specifically, coreset selection involves sampling data with both
input ($\bx$) and output ($\by$), active learning focuses solely on the input
data ($\bx$).
  In this study, we present a theoretically optimal solution for addressing
both coreset selection and active learning within the context of linear softmax
regression. Our proposed method, COPS (unCertainty based OPtimal Sub-sampling),
is designed to minimize the expected loss of a model trained on subsampled
data. Unlike existing approaches that rely on explicit calculations of the
inverse covariance matrix, which are not easily applicable to deep learning
scenarios, COPS leverages the model's logits to estimate the sampling ratio.
This sampling ratio is closely associated with model uncertainty and can be
effectively applied to deep learning tasks. Furthermore, we address the
challenge of model sensitivity to misspecification by incorporating a
down-weighting approach for low-density samples, drawing inspiration from
previous works.
  To assess the effectiveness of our proposed method, we conducted extensive
empirical experiments using deep neural networks on benchmark datasets. The
results consistently showcase the superior performance of COPS compared to
baseline methods, reaffirming its efficacy.