Improving Uncertainty Sampling with Bell Curve Weight Function
Zan-Kai Chong Hiroyuki Ohsaki Bok-Min Goi
Typically, a supervised learning model is trained using passive learning by
randomly selecting unlabelled instances to annotate. This approach is effective
for learning a model, but can be costly in cases where acquiring labelled
instances is expensive. For example, it can be time-consuming to manually
identify spam mails (labelled instances) from thousands of emails (unlabelled
instances) flooding an inbox during initial data collection. Generally, we
answer the above scenario with uncertainty sampling, an active learning method
that improves the efficiency of supervised learning by using fewer labelled
instances than passive learning. Given an unlabelled data pool, uncertainty
sampling queries the labels of instances where the predicted probabilities, p,
fall into the uncertainty region, i.e., $p \approx 0.5$. The newly acquired
labels are then added to the existing labelled data pool to learn a new model.
Nonetheless, the performance of uncertainty sampling is susceptible to the area
of unpredictable responses (AUR) and the nature of the dataset. It is difficult
to determine whether to use passive learning or uncertainty sampling without
prior knowledge of a new dataset. To address this issue, we propose bell curve
sampling, which employs a bell curve weight function to acquire new labels.
With the bell curve centred at p=0.5, bell curve sampling selects instances
whose predicted values are in the uncertainty area most of the time without
neglecting the rest. Simulation results show that, most of the time bell curve
sampling outperforms uncertainty sampling and passive learning in datasets of
different natures and with AUR.