A Comparative Analysis of Word-Level Metric Differential Privacy:
  Benchmarking The Privacy-Utility Trade-off
Stephen Meisenbacher Nihildev Nandakumar Alexandra Klymenko Florian Matthes
The application of Differential Privacy to Natural Language Processing
techniques has emerged in relevance in recent years, with an increasing number
of studies published in established NLP outlets. In particular, the adaptation
of Differential Privacy for use in NLP tasks has first focused on the
$\textit{word-level}$, where calibrated noise is added to word embedding
vectors to achieve "noisy" representations. To this end, several
implementations have appeared in the literature, each presenting an alternative
method of achieving word-level Differential Privacy. Although each of these
includes its own evaluation, no comparative analysis has been performed to
investigate the performance of such methods relative to each other. In this
work, we conduct such an analysis, comparing seven different algorithms on two
NLP tasks with varying hyperparameters, including the $\textit{epsilon
($\varepsilon$)}$ parameter, or privacy budget. In addition, we provide an
in-depth analysis of the results with a focus on the privacy-utility trade-off,
as well as open-source our implementation code for further reproduction. As a
result of our analysis, we give insight into the benefits and challenges of
word-level Differential Privacy, and accordingly, we suggest concrete steps
forward for the research field.