HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from
  Multi-view Cameras
Zhongyu Xia ZhiWei Lin Xinhao Wang Yongtao Wang Yun Xing Shengxiang Qi Nan Dong Ming-Hsuan Yang
Three-dimensional perception from multi-view cameras is a crucial component
in autonomous driving systems, which involves multiple tasks like 3D object
detection and bird's-eye-view (BEV) semantic segmentation. To improve
perception precision, large image encoders, high-resolution images, and
long-term temporal inputs have been adopted in recent 3D perception models,
bringing remarkable performance gains. However, these techniques are often
incompatible in training and inference scenarios due to computational resource
constraints. Besides, modern autonomous driving systems prefer to adopt an
end-to-end framework for multi-task 3D perception, which can simplify the
overall system architecture and reduce the implementation complexity. However,
conflict between tasks often arises when optimizing multiple tasks jointly
within an end-to-end 3D perception model. To alleviate these issues, we present
an end-to-end framework named HENet for multi-task 3D perception in this paper.
Specifically, we propose a hybrid image encoding network, using a large image
encoder for short-term frames and a small image encoder for long-term temporal
frames. Then, we introduce a temporal feature integration module based on the
attention mechanism to fuse the features of different frames extracted by the
two aforementioned hybrid image encoders. Finally, according to the
characteristics of each perception task, we utilize BEV features of different
grid sizes, independent BEV encoders, and task decoders for different tasks.
Experimental results show that HENet achieves state-of-the-art end-to-end
multi-task 3D perception results on the nuScenes benchmark, including 3D object
detection and BEV semantic segmentation. The source code and models will be
released at https://github.com/VDIGPKU/HENet.