Is ChatGPT Transforming Academics' Writing Style?
Mingmeng Geng Roberto Trotta
Based on one million arXiv papers submitted from May 2018 to January 2024, we
assess the textual density of ChatGPT's writing style in their abstracts by
means of a statistical analysis of word frequency changes. Our model is
calibrated and validated on a mixture of real abstracts and ChatGPT-modified
abstracts (simulated data) after a careful noise analysis. We find that ChatGPT
is having an increasing impact on arXiv abstracts, especially in the field of
computer science, where the fraction of ChatGPT-revised abstracts is estimated
to be approximately 35%, if we take the output of one of the simplest prompts,
"revise the following sentences", as a baseline. We conclude with an analysis
of both positive and negative aspects of the penetration of ChatGPT into
academics' writing style.