Commit
·
b93b4b2
1
Parent(s):
81b1588
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,9 @@
|
|
| 1 |
This repo contains the fully trained ByT5 that was used to estimate per-character entropies. Using it, you can also recreate the illustration in the paper.
|
| 2 |
|
| 3 |
|
|
|
|
|
|
|
|
|
|
| 4 |
```python
|
| 5 |
from huggingface_hub import snapshot_download
|
| 6 |
data_folder = snapshot_download("fxtentacle/tevr-token-entropy-predictor-de")
|
|
|
|
| 1 |
This repo contains the fully trained ByT5 that was used to estimate per-character entropies. Using it, you can also recreate the illustration in the paper.
|
| 2 |
|
| 3 |
|
| 4 |
+
## Testing Tokenizer File
|
| 5 |
+
(copy of `TEVR Explanation.ipynb`)
|
| 6 |
+
|
| 7 |
```python
|
| 8 |
from huggingface_hub import snapshot_download
|
| 9 |
data_folder = snapshot_download("fxtentacle/tevr-token-entropy-predictor-de")
|