VoxWatch: An open-set speaker recognition benchmark on VoxCeleb
Raghuveer Peri Seyed Omid Sadjadi Daniel Garcia-Romero
Despite its broad practical applications such as in fraud prevention,
open-set speaker identification (OSI) has received less attention in the
speaker recognition community compared to speaker verification (SV). OSI deals
with determining if a test speech sample belongs to a speaker from a set of
pre-enrolled individuals (in-set) or if it is from an out-of-set speaker. In
addition to the typical challenges associated with speech variability, OSI is
prone to the "false-alarm problem"; as the size of the in-set speaker
population (a.k.a watchlist) grows, the out-of-set scores become larger,
leading to increased false alarm rates. This is in particular challenging for
applications in financial institutions and border security where the watchlist
size is typically of the order of several thousand speakers. Therefore, it is
important to systematically quantify the false-alarm problem, and develop
techniques that alleviate the impact of watchlist size on detection
performance. Prior studies on this problem are sparse, and lack a common
benchmark for systematic evaluations. In this paper, we present the first
public benchmark for OSI, developed using the VoxCeleb dataset. We quantify the
effect of the watchlist size and speech duration on the watchlist-based speaker
detection task using three strong neural network based systems. In contrast to
the findings from prior research, we show that the commonly adopted adaptive
score normalization is not guaranteed to improve the performance for this task.
On the other hand, we show that score calibration and score fusion, two other
commonly used techniques in SV, result in significant improvements in OSI
performance.