Mellum-4b-base API & Inference Endpoint

RepoBench 1.1

Type: single-line
Languages: Python and Java
Metric: Exact Match (EM), %

Since Mellum has a maximum context window of 8k, we report here both the average performance across all evaluated context lengths (2k, 4k, 8k, 12k, and 16k) and the average over context lengths within its supported range (≤ 8k).

Python Subset

Table with columns: Model, 2k, 4k, 8k, 12k, 16k, Avg, Avg ≤ 8k
Model	2k	4k	8k	12k	16k	Avg	Avg ≤ 8k
Mellum-4b-sft-python	29.24%	30.60%	29.77%	26.80%	25.43%	28.37%	29.87%
Mellum-4b-base	28.20%	27.95%	27.77%	24.53%	21.10%	25.91%	27.97%

Java Subset

Table with columns: Model, 2k, 4k, 8k, 12k, 16k, Avg, Avg ≤ 8k
Model	2k	4k	8k	12k	16k	Avg	Avg ≤ 8k
Mellum-4b-base	32.02%	32.12%	29.10%	24.92%	24.74%	28.58%	31.08%

Syntax-Aware Fill-in-the-Middle (SAFIM)

Type: mix of multi-line and single-line
Languages: multi-language
Metric: pass@1, %

Table with columns: Model, Algorithmic, Control, API, Average
Model	Algorithmic	Control	API	Average
Mellum-4b-sft-python	33.16%	36.11%	57.10%	42.12%
Mellum-4b-base	25.30%	38.39%	50.65%	38.11%

HumanEval Infilling

Type: single-line and multi-line
Languages: Python
Metric: pass@1, %

Table with columns: Model, Single-Line, Multi-Line, Random Span
Model	Single-Line	Multi-Line	Random Span
Mellum-4b-sft-python	80.45%	48.19%	37.68%
Mellum-4b-base	66.21%	38.52%	29.70%

We continue to work on model improvements and will share the next iteration soon.

Limitations

Biases: May reflect biases present in public codebases. For example it will likely produce code which is similar in style to the open-source repositories.
Security: Code suggestions should not be assumed to be secure or free of vulnerabilities.

Sample Usage

Here are examples of how to run and sample from the model.

Generic generaion

python
from transformers import AutoTokenizer, AutoModelForCausalLM

example = """
import sys
import os
import time

sys.path.append(os.getcwd())

from cluster.prepare_data import get_headers_pairs_list, write_dist_matrix
from cluster.token_edit_distance import get_distance_matrix

if len(sys.argv) < 3:
    print(
        "Too few arguments. You should provide: \n1. dataset_filename" +
        "\n2. output_data_filename"
    )
    sys.exit()

start = time.perf_counter()
dataset_filename_ = sys.argv[1]
output_data_filename_ = sys.argv[2]

headers_pairs = get_headers_pairs_list(dataset_filename_, verbose=True)

dist_matrix, max_dist = get_distance_matrix(
    list(map(lambda x: x[1], headers_pairs)),
    verbose=True
)

write_dist_matrix(dist_matrix, max_dist, output_data_filename_, verbose=True)

end = time.perf_counter()
"""

tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-base')
model = AutoModelForCausalLM.from_pretrained('JetBrains/Mellum-4b-base')
encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
input_len = len(encoded_input["input_ids"][0])
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)
print("### Context")
print(tokenizer.decode(out[0][:input_len]))
print("### Prediction")
print(tokenizer.decode(out[0][input_len:]))

Fill in the middle with additional files as context generation

python
example = """<filename>utils.py
def multiply(x, y):
    return x * y
<filename>config.py
DEBUG = True
MAX_VALUE = 100
<filename>example.py
<fim_suffix> 

# Test the function
result = calculate_sum(5, 10)
print(result)<fim_prefix>def calculate_sum(a, b):
<fim_middle>"""

encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)

Citation

If you use this model, please cite:

bibtex
@misc{Mellum-4b-base,
  title     = {Mellum-4b-base},
  author    = {Pavlichenko, Nikita and Nazarov, Iurii and Dolgov, Ivan and Garanina, Ekaterina and Lasocki, Karol and Reshetnikova, Julia and Boitsov, Sergei and Bondyrev, Ivan and Karaeva, Dariia and Sheptyakov, Maksim and Ustalov, Dmitry and Mukhin, Artem and Proshev, Semyon and Abramov, Nikita and Kolomyttseva, Olga and Lysaniuk, Kseniia and Zavidnyi, Ilia and Semenkin, Anton and Tankov, Vladislav and Sazanovich, Uladzislau},
  year      = {2025},
}

Contact

For questions, collaborations and requests reach us out via mellum@jetbrains.com

Model

12k

16k

Avg

Avg ≤ 8k

Mellum-4b-sft-python

29.24%

30.60%

29.77%

26.80%

25.43%

28.37%

29.87%

Mellum-4b-base

28.20%

27.95%

27.77%

24.53%

21.10%

25.91%

27.97%

Model

12k

16k

Avg

Avg ≤ 8k

Mellum-4b-base

32.02%

32.12%

29.10%

24.92%

24.74%

28.58%

31.08%

Model

Algorithmic

Control

API

Average

Mellum-4b-sft-python

33.16%

36.11%

57.10%

42.12%

Mellum-4b-base

25.30%

38.39%

50.65%

38.11%

Model

Single-Line

Multi-Line

Random Span

Mellum-4b-sft-python

80.45%

48.19%

37.68%

Mellum-4b-base

66.21%

38.52%

29.70%

Sample Usage

Here are examples of how to run and sample from the model.

Generic generaion

python

from transformers import AutoTokenizer, AutoModelForCausalLM

example = """
import sys
import os
import time

sys.path.append(os.getcwd())

from cluster.prepare_data import get_headers_pairs_list, write_dist_matrix
from cluster.token_edit_distance import get_distance_matrix

if len(sys.argv) < 3:
    print(
        "Too few arguments. You should provide: \n1. dataset_filename" +
        "\n2. output_data_filename"
    )
    sys.exit()

start = time.perf_counter()
dataset_filename_ = sys.argv[1]
output_data_filename_ = sys.argv[2]

headers_pairs = get_headers_pairs_list(dataset_filename_, verbose=True)

dist_matrix, max_dist = get_distance_matrix(
    list(map(lambda x: x[1], headers_pairs)),
    verbose=True
)

write_dist_matrix(dist_matrix, max_dist, output_data_filename_, verbose=True)

end = time.perf_counter()
"""

tokenizer = AutoTokenizer.from_pretrained('JetBrains/Mellum-4b-base')
model = AutoModelForCausalLM.from_pretrained('JetBrains/Mellum-4b-base')
encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
input_len = len(encoded_input["input_ids"][0])
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)
print("### Context")
print(tokenizer.decode(out[0][:input_len]))
print("### Prediction")
print(tokenizer.decode(out[0][input_len:]))

Fill in the middle with additional files as context generation

python

example = """<filename>utils.py
def multiply(x, y):
    return x * y
<filename>config.py
DEBUG = True
MAX_VALUE = 100
<filename>example.py
<fim_suffix> 

# Test the function
result = calculate_sum(5, 10)
print(result)<fim_prefix>def calculate_sum(a, b):
<fim_middle>"""

encoded_input = tokenizer(example, return_tensors='pt', return_token_type_ids=False)
out = model.generate(
    **encoded_input,
    max_new_tokens=100,
)

Citation

If you use this model, please cite:

bibtex

@misc{Mellum-4b-base,
  title     = {Mellum-4b-base},
  author    = {Pavlichenko, Nikita and Nazarov, Iurii and Dolgov, Ivan and Garanina, Ekaterina and Lasocki, Karol and Reshetnikova, Julia and Boitsov, Sergei and Bondyrev, Ivan and Karaeva, Dariia and Sheptyakov, Maksim and Ustalov, Dmitry and Mukhin, Artem and Proshev, Semyon and Abramov, Nikita and Kolomyttseva, Olga and Lysaniuk, Kseniia and Zavidnyi, Ilia and Semenkin, Anton and Tankov, Vladislav and Sazanovich, Uladzislau},
  year      = {2025},
}

Mellum-4b-base

README

RepoBench 1.1

Python Subset

Java Subset

Syntax-Aware Fill-in-the-Middle (SAFIM)

HumanEval Infilling

Limitations

Sample Usage

Generic generaion

Fill in the middle with additional files as context generation

Citation

Contact

Explore FriendliAI today

README

RepoBench 1.1

Python Subset

Java Subset

Syntax-Aware Fill-in-the-Middle (SAFIM)

HumanEval Infilling

Limitations

Sample Usage

Generic generaion

Fill in the middle with additional files as context generation

Citation

Contact