|
|
--- |
|
|
language: bg |
|
|
tags: |
|
|
- gpt2 |
|
|
- lora |
|
|
- bulgarian |
|
|
- causal-lm |
|
|
license: mit |
|
|
datasets: |
|
|
- cc100 |
|
|
model-index: |
|
|
- name: GPT-2 Bulgarian LoRA Adapter (Merged) |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
# 🤖 GPT-2 Bulgarian LoRA Adapter (Merged) |
|
|
|
|
|
**I will be training a much larger sample in the coming days (1k is small - but my computers bandwidth is smaller)** |
|
|
|
|
|
This model is a fine-tuned and merged version of `openai-community/gpt2-medium`, adapted to Bulgarian using the [LoRA](https://arxiv.org/abs/2106.09685) technique. Training was performed on a filtered sample of the Bulgarian subset of the CC100 dataset using [PEFT](https://github.com/huggingface/peft). |
|
|
|
|
|
## 🔧 Model Details |
|
|
|
|
|
- **Base Model**: `openai-community/gpt2-medium` |
|
|
- **LoRA Rank**: 8 |
|
|
- **Target Modules**: `c_attn` |
|
|
- **Dataset**: `cc100.bg` (1000 filtered samples) |
|
|
- **Max Seq Length**: 512 tokens |
|
|
- **Batch Size**: 2 (with gradient accumulation) |
|
|
- **Steps**: 1000 |
|
|
- **Merged Model**: Yes (LoRA weights fused into base model) |
|
|
|
|
|
## 💬 Example Usage |
|
|
|
|
|
This model is a fine-tuned and merged version of `openai-community/gpt2-medium`, adapted to Bulgarian using the [LoRA](https://arxiv.org/abs/2106.09685) technique. Training was performed on a filtered sample of the Bulgarian subset of the CC100 dataset using [PEFT](https://github.com/huggingface/peft). |
|
|
|
|
|
## 🔧 Model Details |
|
|
|
|
|
- **Base Model**: `openai-community/gpt2-medium` |
|
|
- **LoRA Rank**: 8 |
|
|
- **Target Modules**: `c_attn` |
|
|
- **Dataset**: `cc100.bg` (1000 filtered samples) |
|
|
- **Max Seq Length**: 512 tokens |
|
|
- **Batch Size**: 2 (with gradient accumulation) |
|
|
- **Steps**: 1000 |
|
|
- **Merged Model**: Yes (LoRA weights fused into base model) |
|
|
|
|
|
## 💬 Example Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
model = AutoModelForCausalLM.from_pretrained("your-username/gpt2-bulgarian-merged") |
|
|
tokenizer = AutoTokenizer.from_pretrained("your-username/gpt2-bulgarian-merged") |
|
|
inputs = tokenizer("България е известна със своите", return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## 📈 Intended Use |
|
|
|
|
|
For educational purposes, experimentation, and research on low-resource language modeling in Bulgarian. |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
- Trained on a small 1k sample. |
|
|
- No toxic content filtering or safety tuning. |
|
|
- Should not be used in production without further validation. |
|
|
|
|
|
## 👤 Author |
|
|
|
|
|
Developed by [Vanessa Beck](https://github.com/stochastic-sisyphus) on Google Colab using 🤗 Transformers + PEFT. |
