GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
maylore-ai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
maylore-ai
Model Tree
Qwen/Qwen2.5-Coder-1.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
HumanEval-RS and MBPP-RS show pass@1 / pass@10, %. pass@1 is the percentage of tasks solved on the first attempt, pass@10 the percentage solved in at least one of 10 attempts.
| model | HumanEval-RS | MBPP-RS | Temper Bench v2 (greedy pass@1) |
|---|---|---|---|
| Qwen3.5-9B | 76.0 / 90.2 | 65.6 / 79.2 | 29.3 |
| Qwen2.5-Coder-7B-Instruct | 72.4 / 82.6 | 60.5 / 70.0 | 26.0 |
| Tessa-Rust-T1-7B | 71.5 / 79.0 | 57.8 / 68.9 | 29.3 |
| Qwen3.5-4B | 66.2 / 81.4 | 57.7 / 73.4 | 20.7 |
| Temper-2-Axon | 64.2 / 73.8 | 54.7 / 61.7 | 21.3 |
| Qwen3-4B | 59.8 / 64.8 | 55.6 / 60.6 | 18.7 |
| Qwen2.5-Coder-3B-Instruct | 56.5 / 71.9 | 44.8 / 60.0 | 14.7 |
| Temper-1-1.5B | 54.1 / 68.8 | 48.6 / 59.2 | 16.7 |
| Qwen2.5-Coder-1.5B-Instruct | 39.0 / 56.1 | 38.3 / 52.1 | 7.3 |
Models are sorted by HumanEval-RS pass@1, bold marks the best result among models up to 1.5B. All models were measured on the same server. HumanEval-RS (156 tasks) and MBPP-RS (354 tasks) from MultiPL-E at revision 28441b6 go to the model as a chat message with answers up to 1000 tokens, temperature 0.2, top_p 0.95 and 50 answers per task. vLLM took the remaining sampling parameters, top_k and repetition_penalty among them, from the generation_config.json of each model. pass@k comes from the unbiased estimator in the Codex paper. Because the first closed code block of the answer is compiled with the tests, the numbers differ from leaderboards that score code completion. mbpp_67_bell_number has no correct answer in Rust and counts as failed for every model. Every model ran without a system message, using the default of its own chat template, with Qwen2.5-Coder models in their Instruct versions and Qwen3 and Qwen3.5 models without thinking mode. The scripts are open in maylore-ai/rust-evals.
Temper Bench v2 is our own set of 150 Rust tasks at three difficulty levels, scored greedy pass@1.
The base is Qwen2.5-Coder-1.5B-Instruct. Like Temper-2-Ion, the model was fine-tuned on Rust tasks with solutions that passed the compiler and tests, along with dialogs where it talks about itself, writes code over several turns or keeps up a conversation. The requests vary from a detailed specification to a phrase of a few words. The fine-tuned model was merged by weights with the original Instruct to bring back part of the request understanding it loses during fine-tuning.
Reinforcement learning on tasks with tests came last, with a reward only for an answer that passed all tests.
python
from transformers import AutoModelForCausalLM, AutoTokenizer name = "maylore-ai/Temper-2-Axon"tokenizer = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto") messages = [{"role": "user", "content": "Write a Rust function that returns the n-th Fibonacci number."}]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1000)print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))Without a system message the chat template adds the Temper system prompt. The default generation config samples with temperature 0.2, top_p 0.8, top_k 20 and repetition_penalty 1.1. The results above used the same temperature with top_p 0.95, top_k 20 and repetition_penalty 1.1 from this config.
A GGUF version for llama.cpp, LM Studio and Ollama is in maylore-ai/Temper-2-Axon-GGUF.
Axon trails larger models, with Qwen2.5-Coder-7B solving 8.2 points more HumanEval-RS tasks on the first try and Qwen3.5-9B 11.8 more. The model rarely solves tasks with macros, lifetimes or async code. Having been trained on chat requests without fill-in-the-middle, it does not work as a drop-in model for inline completion in an editor.
Apache 2.0, same as Qwen2.5-Coder-1.5B-Instruct.
from transformers import AutoModelForCausalLM, AutoTokenizer name = "maylore-ai/Temper-2-Axon"tokenizer = AutoTokenizer.from_pretrained(name)model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto") messages = [{"role": "user", "content": "Write a Rust function that returns the n-th Fibonacci number."}]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=1000)print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))