README.md

---
base_model: EuroBERT/EuroBERT-210m
language:
- ar
library_name: sentence-transformers
license: mit
metrics:
- pearson_cosine
- spearman_cosine
pipeline_tag: feature-extraction
tags:
- sentence-transformers
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
- EuroBert
- Arabic
widget:
- source_sentence: امرأة شقراء تطل على مشهد (سياتل سبيس نيدل)
  sentences:
  - رجل يستمتع بمناظر جسر البوابة الذهبية
  - فتاة بالخارج تلعب في الثلج
  - شخص ما يأخذ في نظرة إبرة الفضاء.
- source_sentence: سوق الشرق الأوسط
  sentences:
  - مسرح أمريكي
  - متجر في الشرق الأوسط
  - البالغون صغار
- source_sentence: رجلين يتنافسان في ملابس فنون الدفاع عن النفس
  sentences:
  - هناك العديد من الناس الحاضرين.
  - الكلب الأبيض على الشاطئ
  - هناك شخص واحد فقط موجود.\
- source_sentence: مجموعة من الناس تمشي بجانب شاحنة.
  sentences:
  - الناس يقفون
  - بعض الناس بالخارج
  - بعض الرجال يقودون على الطريق
- source_sentence: لاعبة كرة ناعمة ترمي الكرة إلى زميلتها في الفريق
  sentences:
  - شخصان يلعبان كرة البيسبول
  - الرجل ينظف
  - لاعبين لكرة البيسبول يجلسان على مقعد
model-index:
- name: SentenceTransformer based on EuroBERT/EuroBERT-210m
  results:
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 768
      type: sts-dev-768
    metrics:
    - type: pearson_cosine
      value: 0.8111988062913815
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.8100586279907306
      name: Spearman Cosine
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 512
      type: sts-dev-512
    metrics:
    - type: pearson_cosine
      value: 0.8092891955563192
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.8087644228771842
      name: Spearman Cosine
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 256
      type: sts-dev-256
    metrics:
    - type: pearson_cosine
      value: 0.8076510620939634
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.8080588277305082
      name: Spearman Cosine
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 128
      type: sts-dev-128
    metrics:
    - type: pearson_cosine
      value: 0.8028710019029521
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.8054855987917489
      name: Spearman Cosine
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 64
      type: sts-dev-64
    metrics:
    - type: pearson_cosine
      value: 0.7923252906438638
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.7975941111911333
      name: Spearman Cosine
---

# Ara-EuroBERT: Arabic-optimized Sentence Transformer

<img src="https://i.ibb.co/d4svDscP/Clear-Familiar-situations-that-you-already-have-best-practices-for-4.png" width="100" align="left"/>

Ara-EuroBERT is a [sentence-transformers](https://www.SBERT.net) model fine-tuned from [EuroBERT/EuroBERT-210m](https://huggingface.co/EuroBERT/EuroBERT-210m) specifically optimized for **Semantic Arabic text embeddings**.

This model maps sentences and paragraphs to a **768-dimensional dense vector space**  and  **Maximum Sequence Length:** 8,192 tokens.

Paper: [](https://huggingface.co/papers/2503.05500)

You can find more information on the base model at https://huggingface.co/EuroBERT/EuroBERT-210m

![image/png](https://cdn-uploads.huggingface.co/production/uploads/628f7a71dd993507cfcbe587/gKbhM-U-RsAoIa8pkDQX4.png)

Our fine-tuned model shows remarkable improvements over the base models, achieving a 73.5% relative improvement on STS17 and a 21.6% relative improvement on STS22.v2 compared to the base EuroBERT-210M.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer with Matryoshka Embeddings
- **Base model:** [EuroBERT/EuroBERT-210m](https://huggingface.co/EuroBERT/EuroBERT-210m)
- **Maximum Sequence Length:** 8,192 tokens
- **Output Dimensionality:** Matryoshka embeddings with dimensions [768, 512, 256, 128, 64]
- **Similarity Function:** Cosine Similarity
- **Languages:** Optimized for Arabic
- **License:** Same as EuroBERT (MIT)

### Matryoshka Embeddings

This model is trained with Matryoshka Representation Learning, allowing for flexible embedding dimensionality without retraining. You can use smaller dimensions (64, 128, 256, 512) for efficiency or the full 768 dimensions for maximum performance. The model maintains strong performance even at reduced dimensions:

| Dimension | Spearman Correlation (STS Dev) |
|:---------:|:------------------------------:|
| 768       | 0.8101                         |
| 512       | 0.8088                         |
| 256       | 0.8081                         |
| 128       | 0.8055                         |
| 64        | 0.7976                         |

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: EuroBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```

## Use Cases

This model excels at various Arabic NLP tasks:
- Semantic textual similarity
- Semantic search and information retrieval
- Document clustering and classification
- Question answering
- Paraphrase detection
- Zero-shot classification

## Usage

### Installation

```bash
pip install -U sentence-transformers
```

### Basic Usage

```python
from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("Omartificial-Intelligence-Space/AraEuroBert-210M")

# Encode Arabic sentences
sentences = [
    'التقدم العلمي في مجال الذكاء الاصطناعي يتسارع بشكل ملحوظ في السنوات الأخيرة',
    'تطوير نماذج لغوية متقدمة يساهم في تحسين فهم اللغة العربية آليًا',
    'أصبحت تقنيات معالجة اللغات الطبيعية جزءًا أساسيًا من التطبيقات الحديثة',
    'يعاني الشرق الأوسط من تحديات مناخية متزايدة تهدد الأمن المائي والغذائي',
    'تراث الأدب العربي غني بالقصائد والروايات التي تعكس تاريخ وثقافة المنطقة',
]
embeddings = model.encode(sentences)
print(embeddings.shape)  # [3, 768]

# Get the similarity scores
from sentence_transformers import util
similarities = util.cos_sim(embeddings, embeddings)
print(similarities)
```

### Using Matryoshka Embeddings

```python
# Get embeddings with different dimensions
embeddings_768 = model.encode(sentences)  # Default: full 768 dimensions
embeddings_256 = model.encode(sentences, truncate_dim=256)  # Use only 256 dimensions
embeddings_64 = model.encode(sentences, truncate_dim=64)  # Use only 64 dimensions
```

### Training Method

- **Loss Function:** MatryoshkaLoss with MultipleNegativesRankingLoss
- **Matryoshka Dimensions:** [768, 512, 256, 128, 64]
- **Batch Size:** 32
- **Epochs:** 1 (with early stopping)
- **Optimizer:** AdamW
- **Learning Rate:** 5e-05 with linear scheduler and 10% warmup
- **Hardware:** Multiple NVIDIA GPUs with mixed precision (fp16)

## Base Model: EuroBERT

EuroBERT is a new family of multilingual encoder models designed specifically for European and widely spoken global languages. It offers several advantages over traditional multilingual encoders:

- **Extensive Multilingual Coverage:** Trained on a 5 trillion-token dataset across 15 languages
- **Advanced Architecture:** Uses grouped query attention, rotary position embeddings, and RMS normalization
- **Long Context Support:** Natively processes up to 8,192 tokens
- **Specialized Knowledge:** Includes math and programming language data for improved reasoning

## Limitations and Recommendations

- The model is primarily optimized for Arabic text and may not perform optimally on other languages
- Performance may vary for specialized domains not well-represented in the training data
- For short texts (<5 words), consider augmenting with context for better representations
- For extremely long documents, consider splitting into meaningful chunks before encoding

## Citation

If you use this model in your research, please cite the following works:

```bibtex
@misc{boizard2025eurobertscalingmultilingualencoders,
      title={EuroBERT: Scaling Multilingual Encoders for European Languages}, 
      author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo},\
      year={2025},\
      eprint={2503.05500},\
      archivePrefix={arXiv},\
      primaryClass={cs.CL},\
      url={https://arxiv.org/abs/2503.05500}, 
}
```

```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},\
    eprint={2205.13147},\
    archivePrefix={arXiv},\
    primaryClass={cs.LG}\
}
```