Object Detection Difficulty: Suppressing Over-aggregation for Faster and
  Better Video Object Detection
Bingqing Zhang Sen Wang Yifan Liu Brano Kusy Xue Li Jiajun Liu
Current video object detection (VOD) models often encounter issues with
over-aggregation due to redundant aggregation strategies, which perform feature
aggregation on every frame. This results in suboptimal performance and
increased computational complexity. In this work, we propose an image-level
Object Detection Difficulty (ODD) metric to quantify the difficulty of
detecting objects in a given image. The derived ODD scores can be used in the
VOD process to mitigate over-aggregation. Specifically, we train an ODD
predictor as an auxiliary head of a still-image object detector to compute the
ODD score for each image based on the discrepancies between detection results
and ground-truth bounding boxes. The ODD score enhances the VOD system in two
ways: 1) it enables the VOD system to select superior global reference frames,
thereby improving overall accuracy; and 2) it serves as an indicator in the
newly designed ODD Scheduler to eliminate the aggregation of frames that are
easy to detect, thus accelerating the VOD process. Comprehensive experiments
demonstrate that, when utilized for selecting global reference frames, ODD-VOD
consistently enhances the accuracy of Global-frame-based VOD models. When
employed for acceleration, ODD-VOD consistently improves the frames per second
(FPS) by an average of 73.3% across 8 different VOD models without sacrificing
accuracy. When combined, ODD-VOD attains state-of-the-art performance when
competing with many VOD methods in both accuracy and speed. Our work represents
a significant advancement towards making VOD more practical for real-world
applications.