superbean/distilbert-base-cased-finer-finetuned

Model Card

Model description

This model is a fine-tuned version of distilbert/distilbert-base-cased on a subset of the finer-139 dataset.

It achieves the following results on the test set (selected subset, selection process is documented below).

	precision	recall	f1-score	support
DebtInstrumentBasisSpreadOnVariableRate1	0.35	1.00	0.51	9
DebtInstrumentFaceAmount	0.06	0.46	0.10	13
DebtInstrumentInterestRateStatedPercentage	0.11	1.00	0.19	8
LineOfCreditFacilityMaximumBorrowingCapacity	0.11	0.62	0.19	13

Training and evaluation data

A subset of the original finer-139 dataset is used for building the model.

The following steps have been taken for getting the subset:

1. Instead of using the original set of 139 entities, only a subset of 4 entities are covered in this data, namely:

   • DebtInstrumentInterestRateStatedPercentage,
   • LineOfCreditFacilityMaximumBorrowingCapacity
   • DebtInstrumentBasisSpreadOnVariableRate1
   • DebtInstrumentFaceAmount

   These 4 entities above are picked up because they are the most common ones from the original dataset. Any other entity from the original dataset will be considered as "O".

2. Any record in the dataset with more than 200 tokens(words) are removed. (What is left is already covering majority of the cases.)

3. Any record without any entity in it is removed.

All the three steps haven been executed with both "train" and "validation" part of the finer-139 dataset. For the "test" set, however, step 3 is not run because we still want to see how the fine-tuned model can cope with more generalized cases.

However, for speed reason, we randomly pick 1000 records to make a smaller test set in the experiment.

Training procedure

Training hyperparameteres

The following hyperparameters were used during training:

• seed: 42
• learning_rate: 2e-5
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• num_train_epochs: 3
• weight_decay: 0.01

Training results

Epoch	Training Loss	Validation Loss	Precision	Recall	F1	Accuracy
1	No log	0.056148	0.715729	0.796148	0.753799	0.980751
2	0.093300	0.059250	0.781487	0.826645	0.803432	0.982387
3	0.017500	0.064857	0.785185	0.850722	0.816641	0.983058

Libraries versions (during model development)

• jupyterlab: 4.3.5
• transformers: 4.48.3
• torch: 2.6.0
• datasets: 3.3.2
• pandas: 2.2.3
• matplotlib: 3.10.1
• seaborn: 0.13.2
• seqeval: 1.2.2
• evaluate: 0.4.3
• accelerate: 1.5.1
• scikit-learn: 1.6.1
• onnxruntime: 1.21.0
• onnx: 1.17.0
• optimum['exporters']: 1.24.0

How to use the Model

Original PyTorch Model

Example usage

from transformers import pipeline

ner_pipeline = pipeline('token-classification', model='superbean/distilbert-base-cased-finer-finetuned')

test_text = "( 3 ) In February 2020 , the committed amount under the facility was temporarily increased $ 75.0 million to $ 150.0 million , which expires on May 29 , 2020 ."

result = ner_pipeline(test_text)

ONNX Model

Example usage

from optimum.pipelines import pipeline
from optimum.onnxruntime import ORTModelForTokenClassification

ort_model = ORTModelForTokenClassification.from_pretrained('superbean/distilbert-base-cased-finer-finetuned', subfolder='onnx')
ort_ner_pipeline = pipeline('token-classification', model=ort_model, accelerator='ort')

test_text = "( 3 ) In February 2020 , the committed amount under the facility was temporarily increased $ 75.0 million to $ 150.0 million , which expires on May 29 , 2020 ."

result = ort_ner_pipeline(test_text)

For both usage example, you can expectt output result being somethign like the following:

# [
#     {
#         'entity': 'B-LineOfCreditFacilityMaximumBorrowingCapacity',
#         'score': np.float32(0.5120859),
#         'index': 18,
#         'word': '75',
#         'start': 93,
#         'end': 95
#     }, {
#         'entity': 'I-LineOfCreditFacilityMaximumBorrowingCapacity',
#         'score': np.float32(0.68770176),
#         'index': 19,
#         'word': '.',
#         'start': 95,
#         'end': 96
#     }, ..., {
#         'entity': 'I-LineOfCreditFacilityMaximumBorrowingCapacity',
#         'score': np.float32(0.9414747),
#         'index': 26,
#         'word': '0',
#         'start': 115,
#         'end': 116
#     }
# ]