Hugging Face's logo

trollek
/
Llama-3.2-3B-Danoia

Text Generation
Transformers
Safetensors
Danish
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
Llama-3.2-3B-Danoia / README.md
trollek's picture
trollek
Update README.md
a7c9a37 verified
preview code
|
raw
history blame contribute delete
1.72 kB
metadata
license: llama3.2
datasets:
  - trollek/Danoia-v03
  - trollek/Danoia-v02
  - trollek/ProbingPanoia-v01
language:
  - da
base_model:
  - huihui-ai/Llama-3.2-3B-Instruct-abliterated
library_name: transformers

Llama-3.2-3B-Danoia

Der er ikke meget at sige andet end at den kan dansk.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 2
  • eval_batch_size: 1
  • seed: 222
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 16
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.01
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss
1.0895 0.2103 500 1.0488
1.0893 0.4205 1000 0.9952
0.864 0.6308 1500 0.9645
0.9665 0.8411 2000 0.9406
0.9387 1.0514 2500 0.9242
0.7996 1.2617 3000 0.9126
0.7904 1.4720 3500 0.9005
0.9745 1.6822 4000 0.8926
1.0152 1.8925 4500 0.8859
0.7676 2.1028 5000 0.8821
0.8127 2.3131 5500 0.8791
0.9498 2.5234 6000 0.8770
0.795 2.7336 6500 0.8758
0.8029 2.9439 7000 0.8758

Framework versions

  • PEFT 0.11.1
  • Transformers 4.46.1
  • Pytorch 2.5.1
  • Datasets 2.20.0
  • Tokenizers 0.20.3