Mallard74/bert-xcode-tags-classification

bert-base-uncased

This model is a fine-tuned version of bert-base-uncased on the xCodeEval dataset, more precisely on the multi-tag classification task.

It achieves the following results on the evaluation set:

• Loss: 0.2880
• F1 Macro: 0.3305
• F1 Micro: 0.6076
• Roc Auc: 0.8857
• Accuracy: 0.4314
• Hamming Loss: 0.1064

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Macro	F1 Micro	Roc Auc	Accuracy	Hamming Loss
No log	1.0	287	0.3286	0.1706	0.4810	0.8514	0.3137	0.1270
0.3527	2.0	574	0.2958	0.3283	0.6029	0.8760	0.4196	0.1059
0.3527	3.0	861	0.2880	0.3305	0.6076	0.8857	0.4314	0.1064

Framework versions

• Transformers 4.44.2
• Pytorch 2.2.2
• Datasets 3.0.0
• Tokenizers 0.19.1