Investigating Masking-based Data Generation in Language Models
Ed S. Ma
The current era of natural language processing (NLP) has been defined by the
prominence of pre-trained language models since the advent of BERT. A feature
of BERT and models with similar architecture is the objective of masked
language modeling, in which part of the input is intentionally masked and the
model is trained to predict this piece of masked information. Data augmentation
is a data-driven technique widely used in machine learning, including research
areas like computer vision and natural language processing, to improve model
performance by artificially augmenting the training data set by designated
techniques. Masked language models (MLM), an essential training feature of
BERT, have introduced a novel approach to perform effective pre-training on
Transformer based models in natural language processing tasks. Recent studies
have utilized masked language model to generate artificially augmented data for
NLP downstream tasks. The experimental results show that Mask based data
augmentation method provides a simple but efficient approach to improve the
model performance. In this paper, we explore and discuss the broader
utilization of these data augmentation methods based on MLM.