A Universal Protocol to Benchmark Camera Calibration for Sports
Floriane Magera Thomas Hoyoux Olivier Barnich Marc Van Droogenbroeck
Camera calibration is a crucial component in the realm of sports analytics,
as it serves as the foundation to extract 3D information out of the broadcast
images. Despite the significance of camera calibration research in sports
analytics, progress is impeded by outdated benchmarking criteria. Indeed, the
annotation data and evaluation metrics provided by most currently available
benchmarks strongly favor and incite the development of sports field
registration methods, i.e. methods estimating homographies that map the sports
field plane to the image plane. However, such homography-based methods are
doomed to overlook the broader capabilities of camera calibration in bridging
the 3D world to the image. In particular, real-world non-planar sports field
elements (such as goals, corner flags, baskets, ...) and image distortion
caused by broadcast camera lenses are out of the scope of sports field
registration methods. To overcome these limitations, we designed a new
benchmarking protocol, named ProCC, based on two principles: (1) the protocol
should be agnostic to the camera model chosen for a camera calibration method,
and (2) the protocol should fairly evaluate camera calibration methods using
the reprojection of arbitrary yet accurately known 3D objects. Indirectly, we
also provide insights into the metric used in SoccerNet-calibration, which
solely relies on image annotation data of viewed 3D objects as ground truth,
thus implementing our protocol. With experiments on the World Cup 2014, CARWC,
and SoccerNet datasets, we show that our benchmarking protocol provides fairer
evaluations of camera calibration methods. By defining our requirements for
proper benchmarking, we hope to pave the way for a new stage in camera
calibration for sports applications with high accuracy standards.