Benchmarking Object Detectors with COCO: A New Path Forward
Shweta Singh Aayan Yadav Jitesh Jain Humphrey Shi Justin Johnson Karan Desai
The Common Objects in Context (COCO) dataset has been instrumental in
benchmarking object detectors over the past decade. Like every dataset, COCO
contains subtle errors and imperfections stemming from its annotation
procedure. With the advent of high-performing models, we ask whether these
errors of COCO are hindering its utility in reliably benchmarking further
progress. In search for an answer, we inspect thousands of masks from COCO
(2017 version) and uncover different types of errors such as imprecise mask
boundaries, non-exhaustively annotated instances, and mislabeled masks. Due to
the prevalence of COCO, we choose to correct these errors to maintain
continuity with prior research. We develop COCO-ReM (Refined Masks), a cleaner
set of annotations with visibly better mask quality than COCO-2017. We evaluate
fifty object detectors and find that models that predict visually sharper masks
score higher on COCO-ReM, affirming that they were being incorrectly penalized
due to errors in COCO-2017. Moreover, our models trained using COCO-ReM
converge faster and score higher than their larger variants trained using
COCO-2017, highlighting the importance of data quality in improving object
detectors. With these findings, we advocate using COCO-ReM for future object
detection research. Our dataset is available at https://cocorem.xyz