Controlled Randomness Improves the Performance of Transformer Models
Tobias Deußer Cong Zhao Wolfgang Krämer David Leonhard Christian Bauckhage Rafet Sifa
During the pre-training step of natural language models, the main objective
is to learn a general representation of the pre-training dataset, usually
requiring large amounts of textual data to capture the complexity and diversity
of natural language. Contrasting this, in most cases, the size of the data
available to solve the specific downstream task is often dwarfed by the
aforementioned pre-training dataset, especially in domains where data is
scarce. We introduce controlled randomness, i.e. noise, into the training
process to improve fine-tuning language models and explore the performance of
targeted noise in addition to the parameters of these models. We find that
adding such noise can improve the performance in our two downstream tasks of
joint named entity recognition and relation extraction and text summarization.