Rethinking of Feature Interaction for Multi-task Learning on Dense
  Prediction
Jingdong Zhang Jiayuan Fan Peng Ye Bo Zhang Hancheng Ye Baopu Li Yancheng Cai Tao Chen
Existing works generally adopt the encoder-decoder structure for Multi-task
Dense Prediction, where the encoder extracts the task-generic features, and
multiple decoders generate task-specific features for predictions. We observe
that low-level representations with rich details and high-level representations
with abundant task information are not both involved in the multi-task
interaction process. Additionally, low-quality and low-efficiency issues also
exist in current multi-task learning architectures. In this work, we propose to
learn a comprehensive intermediate feature globally from both task-generic and
task-specific features, we reveal an important fact that this intermediate
feature, namely the bridge feature, is a good solution to the above issues.
Based on this, we propose a novel Bridge-Feature-Centirc Interaction (BRFI)
method. A Bridge Feature Extractor (BFE) is designed for the generation of
strong bridge features and Task Pattern Propagation (TPP) is applied to ensure
high-quality task interaction participants. Then a Task-Feature Refiner (TFR)
is developed to refine final task predictions with the well-learned knowledge
from the bridge features. Extensive experiments are conducted on NYUD-v2 and
PASCAL Context benchmarks, and the superior performance shows the proposed
architecture is effective and powerful in promoting different dense prediction
tasks simultaneously.