Update README.md
Browse files
README.md
CHANGED
|
@@ -17,6 +17,10 @@ pipeline_tag: text2text-generation
|
|
| 17 |
|
| 18 |
# Goldfish Loss
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
We introduce goldfish loss, a new language modeling loss function that mitigates memorization of training data.
|
| 21 |
Specifically, goldfish loss pseudorandomly drops $1/k$ of total tokens seen (in the forward pass) during loss computation (i.e., it doesn't compute loss for these tokens), with k being a hyperparameter.
|
| 22 |
We show that the model finds it increasingly difficult to verbatim regurgitate training data even after 100 epochs. Please read our paper linked below for more details.
|
|
|
|
| 17 |
|
| 18 |
# Goldfish Loss
|
| 19 |
|
| 20 |
+
<div align="center">
|
| 21 |
+
<img src="https://raw.githubusercontent.com/ahans30/goldfish-loss/main/assets/goldfish-loss.jpg" width="300"/>
|
| 22 |
+
</div>
|
| 23 |
+
|
| 24 |
We introduce goldfish loss, a new language modeling loss function that mitigates memorization of training data.
|
| 25 |
Specifically, goldfish loss pseudorandomly drops $1/k$ of total tokens seen (in the forward pass) during loss computation (i.e., it doesn't compute loss for these tokens), with k being a hyperparameter.
|
| 26 |
We show that the model finds it increasingly difficult to verbatim regurgitate training data even after 100 epochs. Please read our paper linked below for more details.
|