metadata
license: apache-2.0
datasets:
- bigcode/the-stack
- bigcode/the-stack-v2
- bigcode/starcoderdata
- bigcode/commitpack
library_name: transformers
tags:
- code
base_model:
- JetBrains/Mellum-4b-base
model-index:
- name: Mellum-4b-sft-python
results:
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python)
metrics:
- name: EM
type: exact_match
value: 0.2837
verified: false
- name: EM ≤ 8k
type: exact_match
value: 0.2987
verified: false
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python, 2k)
metrics:
- name: EM
type: exact_match
value: 0.2924
verified: false
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python, 4k)
metrics:
- name: EM
type: exact_match
value: 0.306
verified: false
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python, 8k)
metrics:
- name: EM
type: exact_match
value: 0.2977
verified: false
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python, 12k)
metrics:
- name: EM
type: exact_match
value: 0.268
verified: false
- task:
type: text-generation
dataset:
type: tianyang/repobench_python_v1.1
name: RepoBench 1.1 (Python, 16k)
metrics:
- name: EM
type: exact_match
value: 0.2543
verified: false
- task:
type: text-generation
dataset:
type: gonglinyuan/safim
name: SAFIM
metrics:
- name: pass@1
type: pass@1
value: 0.4212
verified: false
- task:
type: text-generation
dataset:
type: gonglinyuan/safim
name: SAFIM (Algorithmic)
metrics:
- name: pass@1
type: pass@1
value: 0.3316
verified: false
- task:
type: text-generation
dataset:
type: gonglinyuan/safim
name: SAFIM (Control)
metrics:
- name: pass@1
type: pass@1
value: 0.3611
verified: false
- task:
type: text-generation
dataset:
type: gonglinyuan/safim
name: SAFIM (API)
metrics:
- name: pass@1
type: pass@1
value: 0.571
verified: false
- task:
type: text-generation
dataset:
type: loubnabnl/humaneval_infilling
name: HumanEval Infilling (Single-Line)
metrics:
- name: pass@1
type: pass@1
value: 0.8045
verified: false
- task:
type: text-generation
dataset:
type: loubnabnl/humaneval_infilling
name: HumanEval Infilling (Multi-Line)
metrics:
- name: pass@1
type: pass@1
value: 0.4819
verified: false
- task:
type: text-generation
dataset:
type: loubnabnl/humaneval_infilling
name: HumanEval Infilling (Random Span)
metrics:
- name: pass@1
type: pass@1
value: 0.3768
verified: false
Model Description
Mellum-4b-sft-python is a fine-tuned version of JetBrains' first open-source large language model (LLM) optimized for code-related tasks.
Pre-trained on over 4 trillion tokens with a context window of 8192 tokens across multiple programming languages, and then fine-tuned, Mellum-4b-sft-python is tailored specifically for code completion in Python. The model follows a LLaMA-style architecture with 4 billion parameters, making it efficient for both cloud inference (e.g., via vLLM) and local deployment (e.g., using llama.cpp or Ollama).
Mellum was trained using Automatic Mixed Precision (AMP) with bf16 precision. The uploaded version on Hugging Face retains the bf16 format for public use.
Designed for integration into professional developer tooling (e.g., intelligent code suggestions in IDEs), AI-powered coding assistants, and research on code understanding and generation, Mellum is also well-suited for educational applications and fine-tuning experiments.
Limitations
- Biases: May reflect biases present in public codebases. For example it will likely produce code which is similar in style to the open-source repositories.
- Security: Code suggestions should not be assumed to be secure or free of vulnerabilities.
Sample Usage
Here are examples of how to run and sample from the model.
Generic generaion
import json
from transformers import AutoTokenizer, AutoModelForCausalLM
example = """
import sys
import os
import time
sys.path.append(os.getcwd())
from cluster.prepare_data import get_headers_pairs_list, write_dist_matrix
from cluster.token_edit_distance import get_distance_matrix
if len(sys.argv) < 3:
print(
"Too few arguments. You should provide: \n1. dataset_filename" +
"\n2. output_data_filename"
)
sys.exit()
start = time.perf_counter()
dataset_filename_ = sys.argv[1]
output_data_filename_ = sys.argv[2]
headers_pairs = get_headers_pairs_list(dataset_filename_, verbose=True)
dist_matrix, max_dist = get_distance_matrix(
list(map(lambda x: x[1], headers_pairs)),
verbose=True
)
write_dist_matrix(dist_matrix, max_dist, output_data_filename_, verbose=True)
end = time.perf_counter()
"""
tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-base')
model = AutoModelForCausalLM.from_pretrained('JetBrains/Mellum-4b-base')
encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
input_len = len(encoded_input["input_ids"][0])
out = model.generate(
**encoded_input,
max_new_tokens=100,
)
print("### Context")
print(tokenizer.decode(out[0][:input_len]))
print("### Prediction")
print(tokenizer.decode(out[0][input_len:]))
Fill in the middle generation
prefix = """
def fibonacci(n: int) -> int:
"""
suffix = """
if __name__ == "__main__":
print(fibonacci(10))
"""
encoded_input = tokenizer(f"<fim_suffix>{suffix}<fim_prefix>{prefix}<fim_middle>", return_tensors='pt', return_token_type_ids=False)
out = model.generate(
**encoded_input,
max_new_tokens=100,
)
Citation
If you use this model, please cite:
@misc{mellum-base-4b,
title={Mellum base 4B},
author={Nikita Pavlichenko, Iurii Nazarov, Ivan Dolgov, Julia Reshetnikova, Ekaterina Garanina, Karol Lasocki, Sergei Boitsov, Dariia Karaeva, Ivan Bondyrev, Maksim Sheptyakov, Dmitry Ustalov, Nikita Abramov, Olga Kolomyttseva, Kseniia Lysaniuk, Ilia Zavidnyi, Anton Semenkin, Uladzislau Sazanovich},
year={2025},
}
Contact
For questions, collaborations and requests reach us out via [email protected]