Hugging Face's logo

waris-gill
/
ModernBert-Medical-v1

Sentence Similarity
sentence-transformers
Safetensors
modernbert
feature-extraction
Generated from Trainer
dataset_size:2438
loss:MatryoshkaLoss
loss:OnlineContrastiveLoss
Eval Results
Model card Files Community
ModernBert-Medical-v1 / README.md
waris-gill's picture
waris-gill
Update README.md
24f1156 verified
preview code
|
raw
history blame contribute delete
13.6 kB
metadata
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:2438
  - loss:MatryoshkaLoss
  - loss:OnlineContrastiveLoss
base_model: Alibaba-NLP/gte-modernbert-base
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - cosine_mcc
model-index:
  - name: SentenceTransformer based on Alibaba-NLP/gte-modernbert-base
    results:
      - task:
          type: my-binary-classification
          name: My Binary Classification
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy
            value: 0.9159836065573771
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8090976476669312
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.9216061185468452
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.8090976476669312
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.9305019305019305
            name: Cosine Precision
          - type: cosine_recall
            value: 0.9128787878787878
            name: Cosine Recall
          - type: cosine_ap
            value: 0.974188222191262
            name: Cosine Ap
          - type: cosine_mcc
            value: 0.8312925398469787
            name: Cosine Mcc

SentenceTransformer based on Alibaba-NLP/gte-modernbert-base

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: Alibaba-NLP/gte-modernbert-base
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • csv

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("waris-gill/ModernBert-Medical-v1")
# Run inference
sentences = [
    'My rheumatologist said \'if a patient has lupus then prednisone doesn\'t work." why is that?',
    "I have lupus,my rheumatologist told me that prednisone doesn't work in my case. Could you educate me why? What are my chances? ",
    'Hello doctor, my grandmother has 3rd degree bed sore. What can be done to help?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

My Binary Classification

  • Evaluated with scache.train.MyBinaryClassificationEvaluator
Metric Value
cosine_accuracy 0.916
cosine_accuracy_threshold 0.8091
cosine_f1 0.9216
cosine_f1_threshold 0.8091
cosine_precision 0.9305
cosine_recall 0.9129
cosine_ap 0.9742
cosine_mcc 0.8313

Training Details

Training Dataset

csv

  • Dataset: csv

  • Size: 2,438 training samples

  • Columns: question_1, question_2, and label

  • Approximate statistics based on the first 1000 samples:

  • Loss: MatryoshkaLoss with these parameters:

    {
    "loss": "OnlineContrastiveLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Evaluation Dataset

csv

  • Dataset: csv
  • Size: 2,438 evaluation samples

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 256
  • learning_rate: 6.5383156211679e-05
  • max_grad_norm: 0.5
  • num_train_epochs: 1
  • lr_scheduler_type: constant
  • load_best_model_at_end: True
  • torch_compile: True
  • torch_compile_backend: inductor
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 256
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 6.5383156211679e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 0.5
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: constant
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: True
  • torch_compile_backend: inductor
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss Validation Loss cosine_ap
0.0323 1 4.4977 - -
0.0645 2 4.9952 - -
0.0968 3 2.9984 - -
0.1290 4 4.8052 - -
0.1613 5 4.0031 - -
0.1935 6 3.7682 - -
0.2258 7 4.0361 - -
0.2581 8 3.4003 - -
0.2903 9 1.1674 - -
0.3226 10 2.3826 14.3756 0.9742
0.3548 11 3.8777 - -
0.3871 12 2.6367 - -
0.4194 13 2.5763 - -
0.4516 14 3.5591 - -
0.4839 15 2.3568 - -
0.5161 16 2.9432 - -
0.5484 17 2.746 - -
0.5806 18 3.647 - -
0.6129 19 3.0907 - -
0.6452 20 3.9776 12.4766 0.9771
0.6774 21 3.4131 - -
0.7097 22 3.0084 - -
0.7419 23 2.7182 - -
0.7742 24 1.5211 - -
0.8065 25 1.8332 - -
0.8387 26 3.4883 - -
0.8710 27 2.0585 - -
0.9032 28 2.775 - -
0.9355 29 2.9137 - -
0.9677 30 2.4238 12.4805 0.9769
1.0 31 1.2115 14.3756 0.9742
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.11.11
  • Sentence Transformers: 3.4.1
  • Transformers: 4.49.0
  • PyTorch: 2.5.1+cu124
  • Accelerate: 1.4.0
  • Datasets: 3.3.2
  • Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}