How to Protect Copyright Data in Optimization of Large Language Models?
Timothy Chu Zhao Song Chiwun Yang
Large language models (LLMs) and generative AI have played a transformative
role in computer research and applications. Controversy has arisen as to
whether these models output copyrighted data, which can occur if the data the
models are trained on is copyrighted. LLMs are built on the transformer neural
network architecture, which in turn relies on a mathematical computation called
Attention that uses the softmax function.
  In this paper, we show that large language model training and optimization
can be seen as a softmax regression problem. We then establish a method of
efficiently performing softmax regression, in a way that prevents the
regression function from generating copyright data. This establishes a
theoretical method of training large language models in a way that avoids
generating copyright data.