Studying the impacts of pre-training using ChatGPT-generated text on
  downstream tasks
Sarthak Anand
In recent times, significant advancements have been witnessed in the field of
language models, particularly with the emergence of Large Language Models
(LLMs) that are trained on vast amounts of data extracted from internet
archives. These LLMs, such as ChatGPT, have become widely accessible, allowing
users to generate text for various purposes including articles, essays, jokes,
and poetry. Given that LLMs are trained on a diverse range of text sources,
encompassing platforms like Reddit and Twitter, it is foreseeable that future
training datasets will also incorporate text generated by previous iterations
of the models themselves. In light of this development, our research aims to
investigate the influence of artificial text in the pre-training phase of
language models. Specifically, we conducted a comparative analysis between a
language model, RoBERTa, pre-trained using CNN/DailyMail news articles, and
ChatGPT, which employed the same articles for its training and evaluated their
performance on three downstream tasks as well as their potential gender bias,
using sentiment analysis as a metric. Through a series of experiments, we
demonstrate that the utilization of artificial text during pre-training does
not have a significant impact on either the performance of the models in
downstream tasks or their gender bias. In conclusion, our findings suggest that
the inclusion of text generated by LLMs in their own pre-training process does
not yield substantial effects on the subsequent performance of the models in
downstream tasks or their potential gender bias.