Automatic design optimization of preference-based subjective evaluation
  with online learning in crowdsourcing environment
Yusuke Yasuda Tomoki Toda
A preference-based subjective evaluation is a key method for evaluating
generative media reliably. However, its huge combinations of pairs prohibit it
from being applied to large-scale evaluation using crowdsourcing. To address
this issue, we propose an automatic optimization method for preference-based
subjective evaluation in terms of pair combination selections and allocation of
evaluation volumes with online learning in a crowdsourcing environment. We use
a preference-based online learning method based on a sorting algorithm to
identify the total order of evaluation targets with minimum sample volumes. Our
online learning algorithm supports parallel and asynchronous execution under
fixed-budget conditions required for crowdsourcing. Our experiment on
preference-based subjective evaluation of synthetic speech shows that our
method successfully optimizes the test by reducing pair combinations from 351
to 83 and allocating optimal evaluation volumes for each pair ranging from 30
to 663 without compromising evaluation accuracies and wasting budget
allocations.