QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me
  Challenge
Hsi-Che Lin Chien-Yi Wang Min-Hung Chen Szu-Wei Fu Yu-Chiang Frank Wang
This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D
Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task
and the provided dataset, we propose to use two separate models to process the
input videos and audio. By doing so, we can utilize all the labeled training
data, including those without bounding box labels. Furthermore, we leverage the
face quality score from a facial landmark prediction model for filtering noisy
face input data. The face quality score is also employed in our proposed
quality-aware fusion for integrating the results from two branches. With the
simple architecture design, our model achieves 67.4% mean average precision
(mAP) on the test set, which ranks first on the leaderboard and outperforms the
baseline method by a large margin. Code is available at:
https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23