Label Budget Allocation in Multi-Task Learning
Ximeng Sun Kihyuk Sohn Kate Saenko Clayton Mellina Xiao Bian
The cost of labeling data often limits the performance of machine learning
systems. In multi-task learning, related tasks provide information to each
other and improve overall performance, but the label cost can vary among tasks.
How should the label budget (i.e. the amount of money spent on labeling) be
allocated among different tasks to achieve optimal multi-task performance? We
are the first to propose and formally define the label budget allocation
problem in multi-task learning and to empirically show that different budget
allocation strategies make a big difference to its performance. We propose a
Task-Adaptive Budget Allocation algorithm to robustly generate the optimal
budget allocation adaptive to different multi-task learning settings.
Specifically, we estimate and then maximize the extent of new information
obtained from the allocated budget as a proxy for multi-task learning
performance. Experiments on PASCAL VOC and Taskonomy demonstrate the efficacy
of our approach over other widely used heuristic labeling strategies.