Weak Supervision for Label Efficient Visual Bug Detection
Farrukh Rahman
As video games evolve into expansive, detailed worlds, visual quality becomes
essential, yet increasingly challenging. Traditional testing methods, limited
by resources, face difficulties in addressing the plethora of potential bugs.
Machine learning offers scalable solutions; however, heavy reliance on large
labeled datasets remains a constraint. Addressing this challenge, we propose a
novel method, utilizing unlabeled gameplay and domain-specific augmentations to
generate datasets & self-supervised objectives used during pre-training or
multi-task settings for downstream visual bug detection. Our methodology uses
weak-supervision to scale datasets for the crafted objectives and facilitates
both autonomous and interactive weak-supervision, incorporating unsupervised
clustering and/or an interactive approach based on text and geometric prompts.
We demonstrate on first-person player clipping/collision bugs (FPPC) within the
expansive Giantmap game world, that our approach is very effective, improving
over a strong supervised baseline in a practical, very low-prevalence, low data
regime (0.336 $\rightarrow$ 0.550 F1 score). With just 5 labeled "good"
exemplars (i.e., 0 bugs), our self-supervised objective alone captures enough
signal to outperform the low-labeled supervised settings. Building on
large-pretrained vision models, our approach is adaptable across various visual
bugs. Our results suggest applicability in curating datasets for broader image
and video tasks within video games beyond visual bugs.