TBGC: Task-level Backbone-Oriented Gradient Clip for Multi-Task
  Foundation Model Learning
Zelun Zhang Xue Pan
The AllInOne training paradigm squeezes a wide range of tasks into a unified
model in a multi-task learning manner. However, optimization in multi-task
learning is more challenge than single-task learning, as the gradient norm from
different tasks may vary greatly, making the backbone overly biased towards one
specific task. To address this issue, we propose the task-level
backbone-oriented gradient clip paradigm, compared with the vanilla gradient
clip method, it has two points of emphasis:1) gradient clip is performed
independently for each task. 2) backbone gradients generated from each task are
rescaled to the same norm scale. Based on the experimental results, we argue
that the task-level backbone-oriented gradient clip paradigm can relieve the
gradient bias problem to some extent. We also propose a novel multi-branch data
augmentation strategy where conflict augmentations are placed in different
branches. Our approach has been shown to be effective and finally achieve 1st
place in the Leaderboard A and 2nd place in the Leaderboard B of the CVPR2023
Foundation Model Challenge. It's worth noting that instead of evaluating all
three tasks(detection, segmentation and fine-grained classification) in
Leaderboard A, the segmentation task is not evaluated in Leaderboard B, in
which our team has a huge advantage.