Text generation for dataset augmentation in security classification
  tasks
Alexander P. Welsh Matthew Edwards
Security classifiers, designed to detect malicious content in computer
systems and communications, can underperform when provided with insufficient
training data. In the security domain, it is often easy to find samples of the
negative (benign) class, and challenging to find enough samples of the positive
(malicious) class to train an effective classifier. This study evaluates the
application of natural language text generators to fill this data gap in
multiple security-related text classification tasks. We describe a variety of
previously-unexamined language-model fine-tuning approaches for this purpose
and consider in particular the impact of disproportionate class-imbalances in
the training set. Across our evaluation using three state-of-the-art
classifiers designed for offensive language detection, review fraud detection,
and SMS spam detection, we find that models trained with GPT-3 data
augmentation strategies outperform both models trained without augmentation and
models trained using basic data augmentation strategies already in common
usage. In particular, we find substantial benefits for GPT-3 data augmentation
strategies in situations with severe limitations on known positive-class
samples.