|
--- |
|
library_name: transformers |
|
license: apache-2.0 |
|
language: |
|
- ja |
|
--- |
|
|
|
# llm-jp-modernbert-base |
|
|
|
This model is based on the [modernBERT-base](https://arxiv.org/abs/2412.13663) architecture with [llm-jp-tokenizer](https://github.com/llm-jp/llm-jp-tokenizer). |
|
It was trained using the Japanese subset (3.4TB) of the llm-jp-corpus v4 and supports a max sequence length of 8192. |
|
|
|
For detailed information on the training methods, evaluation, and analysis results, please visit at [llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length](https://arxiv.org/abs/2504.15544) |
|
|
|
## Usage |
|
|
|
Please install the transformers library. |
|
```bash |
|
pip install "transformers>=4.48.0" |
|
``` |
|
|
|
If your GPU supports flash-attn 2, it is recommended to install flash-attn. |
|
``` |
|
pip install flash-attn --no-build-isolation |
|
``` |
|
|
|
Using AutoModelForMaskedLM: |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForMaskedLM |
|
|
|
model_id = "llm-jp/llm-jp-modernbert-base" |
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForMaskedLM.from_pretrained(model_id) |
|
|
|
text = "日本の首都は<MASK|LLM-jp>です。" |
|
inputs = tokenizer(text, return_tensors="pt") |
|
outputs = model(**inputs) |
|
|
|
# To get predictions for the mask: |
|
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id) |
|
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1) |
|
predicted_token = tokenizer.decode(predicted_token_id) |
|
print("Predicted token:", predicted_token) |
|
# Predicted token: 東京 |
|
``` |
|
|
|
|
|
## Training |
|
|
|
This model was trained with a max_seq_len of 1024 in stage 1, and then with a max_seq_len of 8192 in stage 2. |
|
|
|
Training code can be found at https://github.com/llm-jp/llm-jp-modernbert |
|
|
|
| Model | stage 1 | stage 2 | |
|
|:------------------ |----------------:|----------------:| |
|
| max_seq_len | 1024 | 8192 | |
|
| max_steps | 500,000 | 200,000 | |
|
| Total batch size | 3328 | 384 | |
|
| Peak LR | 5e-4 | 5e-5 | |
|
| warmup step | 24,000 | | |
|
| LR schedule | Linear decay | | |
|
| Adam beta 1 | 0.9 | | |
|
| Adam beta 2 | 0.98 | | |
|
| Adam eps | 1e-6 | | |
|
| MLM prob | 0.30 | | |
|
| Gradient clipping | 1.0 | | |
|
| weight decay | 1e-5 | | |
|
| line_by_line | True | | |
|
|
|
The blank in stage 2 indicate the same value as in stage 1. |
|
|
|
## Evaluation |
|
|
|
JSTS, JNLI, and JCoLA from [JGLUE](https://aclanthology.org/2022.lrec-1.317/) were used. |
|
Evaluation code can be found at https://github.com/llm-jp/llm-jp-modernbert |
|
|
|
| Model | JSTS (pearson) | JNLI (accuracy) | JCoLA (accuracy) | Avg | |
|
|-------------------------------------------------------|--------|--------|---------|--------------| |
|
| tohoku-nlp/bert-base-japanese-v3 | 0.920 | 0.912 | 0.880 | 0.904 | |
|
| sbintuitions/modernbert-ja-130m | 0.916 | 0.927 | 0.868 | 0.904 | |
|
| sbintuitions/modernbert-ja-310m | **0.932** | **0.933** | **0.883** | **0.916** | |
|
| **llm-jp/llm-jp-modernbert-base** | 0.918 | 0.913 | 0.844 | 0.892 | |
|
|
|
## LICENSE |
|
|
|
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
|
|
|
## Citation |
|
|
|
``` |
|
@misc{sugiura2025llmjpmodernbertmodernbertmodeltrained, |
|
title={llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length}, |
|
author={Issa Sugiura and Kouta Nakayama and Yusuke Oda}, |
|
year={2025}, |
|
eprint={2504.15544}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2504.15544}, |
|
} |
|
``` |
