Hugging Face's logo

JetBrains
/
Mellum-4b-sft-python

Text Generation
Transformers
Safetensors
llama
code
Eval Results
text-generation-inference
Model card Files Community
Mellum-4b-sft-python / README.md
topshik's picture
topshik
Update README.md
3b8d629 verified
preview code
|
raw
history blame contribute delete
7.03 kB
metadata
license: apache-2.0
datasets:
  - bigcode/the-stack
  - bigcode/the-stack-v2
  - bigcode/starcoderdata
  - bigcode/commitpack
library_name: transformers
tags:
  - code
base_model:
  - JetBrains/Mellum-4b-base
model-index:
  - name: Mellum-4b-sft-python
    results:
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python)
        metrics:
          - name: EM
            type: exact_match
            value: 0.2837
            verified: false
          - name: EM ≤ 8k
            type: exact_match
            value: 0.2987
            verified: false
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python, 2k)
        metrics:
          - name: EM
            type: exact_match
            value: 0.2924
            verified: false
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python, 4k)
        metrics:
          - name: EM
            type: exact_match
            value: 0.306
            verified: false
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python, 8k)
        metrics:
          - name: EM
            type: exact_match
            value: 0.2977
            verified: false
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python, 12k)
        metrics:
          - name: EM
            type: exact_match
            value: 0.268
            verified: false
      - task:
          type: text-generation
        dataset:
          type: tianyang/repobench_python_v1.1
          name: RepoBench 1.1 (Python, 16k)
        metrics:
          - name: EM
            type: exact_match
            value: 0.2543
            verified: false
      - task:
          type: text-generation
        dataset:
          type: gonglinyuan/safim
          name: SAFIM
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.4212
            verified: false
      - task:
          type: text-generation
        dataset:
          type: gonglinyuan/safim
          name: SAFIM (Algorithmic)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.3316
            verified: false
      - task:
          type: text-generation
        dataset:
          type: gonglinyuan/safim
          name: SAFIM (Control)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.3611
            verified: false
      - task:
          type: text-generation
        dataset:
          type: gonglinyuan/safim
          name: SAFIM (API)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.571
            verified: false
      - task:
          type: text-generation
        dataset:
          type: loubnabnl/humaneval_infilling
          name: HumanEval Infilling (Single-Line)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.8045
            verified: false
      - task:
          type: text-generation
        dataset:
          type: loubnabnl/humaneval_infilling
          name: HumanEval Infilling (Multi-Line)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.4819
            verified: false
      - task:
          type: text-generation
        dataset:
          type: loubnabnl/humaneval_infilling
          name: HumanEval Infilling (Random Span)
        metrics:
          - name: pass@1
            type: pass@1
            value: 0.3768
            verified: false

Model Description

Mellum-4b-sft-python is a fine-tuned version of JetBrains' first open-source large language model (LLM) optimized for code-related tasks.

Pre-trained on over 4 trillion tokens with a context window of 8192 tokens across multiple programming languages, and then fine-tuned, Mellum-4b-sft-python is tailored specifically for code completion in Python. The model follows a LLaMA-style architecture with 4 billion parameters, making it efficient for both cloud inference (e.g., via vLLM) and local deployment (e.g., using llama.cpp or Ollama).

Mellum was trained using Automatic Mixed Precision (AMP) with bf16 precision. The uploaded version on Hugging Face retains the bf16 format for public use.

Designed for integration into professional developer tooling (e.g., intelligent code suggestions in IDEs), AI-powered coding assistants, and research on code understanding and generation, Mellum is also well-suited for educational applications and fine-tuning experiments.

Limitations

  • Biases: May reflect biases present in public codebases. For example it will likely produce code which is similar in style to the open-source repositories.
  • Security: Code suggestions should not be assumed to be secure or free of vulnerabilities.

Sample Usage

Here are examples of how to run and sample from the model.

Generic generaion 

import json
from transformers import AutoTokenizer, AutoModelForCausalLM

example = """
import sys
import os
import time

sys.path.append(os.getcwd())

from cluster.prepare_data import get_headers_pairs_list, write_dist_matrix
from cluster.token_edit_distance import get_distance_matrix

if len(sys.argv) < 3:
    print(
        "Too few arguments. You should provide: \n1. dataset_filename" +
        "\n2. output_data_filename"
    )
    sys.exit()

start = time.perf_counter()
dataset_filename_ = sys.argv[1]
output_data_filename_ = sys.argv[2]

headers_pairs = get_headers_pairs_list(dataset_filename_, verbose=True)

dist_matrix, max_dist = get_distance_matrix(
    list(map(lambda x: x[1], headers_pairs)),
    verbose=True
)

write_dist_matrix(dist_matrix, max_dist, output_data_filename_, verbose=True)

end = time.perf_counter()
"""

tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-sft-python')
model = AutoModelForCausalLM.from_pretrained('JetBrains/Mellum-4b-sft-python')
encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
input_len = len(encoded_input["input_ids"][0])
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)
print("### Context")
print(tokenizer.decode(out[0][:input_len]))
print("### Prediction")
print(tokenizer.decode(out[0][input_len:]))

Fill in the middle with additional files as context generation 

example = """<filename>utils.py
def multiply(x, y):
    return x * y
<filename>config.py
DEBUG = True
MAX_VALUE = 100
<filename>example.py
<fim_suffix> 

# Test the function
result = calculate_sum(5, 10)
print(result)<fim_prefix>def calculate_sum(a, b):
<fim_middle>"""

encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)

Citation

If you use this model, please cite:

@misc{Mellum-4b-base,
  title     = {Mellum-4b-base},
  author    = {Pavlichenko, Nikita and Nazarov, Iurii and Dolgov, Ivan and Garanina, Ekaterina and Lasocki, Karol and Reshetnikova, Julia and Boitsov, Sergei and Bondyrev, Ivan and Karaeva, Dariia and Sheptyakov, Maksim and Ustalov, Dmitry and Abramov, Nikita and Kolomyttseva, Olga and Lysaniuk, Kseniia and Zavidnyi, Ilia and Semenkin, Anton and Tankov, Vladislav and Sazanovich, Uladzislau},
  year      = {2025},
}

Contact

For questions, collaborations and requests reach us out via [email protected]