GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
mayankbhaskarr
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
mayankbhaskarr
Model Tree
unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit
Input Modalities
Output Modalities
Supported Functionality
qwen3-4b-coder-ft is a QLoRA fine-tune of the Qwen3-4B-Instruct-2507 family, adapted for coding and code-instruction tasks. The repo contains the full BF16 weights (two-shard safetensors, ~8.0 GB) plus the Qwen3 chat template (chat_template.jinja). A quantized Q8_0 GGUF build is available in the companion repo mayankbhaskarr/qwen3-4b-coder-ft-gguf, and the model was validated with a live runtime benchmark (38 executed tasks + throughput profiling) — full results below.
The model was evaluated with a live runtime benchmark: a 38-item executed suite (12 easy + 12 medium + 6 hard Python coding tasks scored by running generated code against held-out tests, plus 8 instruction-following items with objective rubric checks) and standardized llama-bench throughput profiling. Inference ran on CPU-only hardware (2 vCPU, ~3.7 GB usable RAM) with llama.cpp v0.7.0, greedy decoding (temperature 0.0), context 1280, 2 threads.
Because the BF16 (~8.0 GB) and Q8_0 (4.28 GB) artifacts exceed the host's usable RAM, the benchmark quantized the uploaded Q8_0 GGUF (SHA-256 verified across downloads) into Q5_K_M and Q4_K_M derivatives with llama-quantize --allow-requantize, then ran the full suite on both. Q8_0 is a documented near-lossless format (+0.0004% relative perplexity, k-quants reference data), so these results represent the fine-tuned weights faithfully, with a slight conservative bias.
| Tier | Q5_K_M | Q4_K_M |
|---|---|---|
| Easy coding (12) | 12/12 (100%) | 12/12 (100%) |
| Medium coding (12) | 12/12 (100%) | 12/12 (100%) |
| Hard coding (6) | 3/6 (50.0%) | 2/6 (33.3%) |
| Instruction following (8) | 6/8 (75.0%) | 7/8 (87.5%) |
| Overall (38) | 33/38 (86.8%) | 33/38 (86.8%) |
| Config | Q5_K_M | Q4_K_M |
|---|---|---|
| Prompt eval, t=1 | 28.88 tok/s | 28.30 tok/s |
| Prompt eval, t=2 | 40.54 tok/s | 41.93 tok/s |
| Generation, t=1 | 2.58 tok/s | 3.32 tok/s |
| Generation, t=2 | 3.33 tok/s | 4.58 tok/s |
Server cold-load: 8.55 s (Q5_K_M) / 6.3 s (Q4_K_M) to first healthy response; peak RSS ~3.8 GB including the 1,280-token KV cache.
python
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "mayankbhaskarr/qwen3-4b-coder-ft"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto") messages = [{"role": "user", "content": "Implement binary search in Python."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(inputs, max_new_tokens=512, do_sample=False)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))For CPU / low-memory deployment, use the Q8_0 GGUF from the companion repo with llama.cpp (llama-server -hf mayankbhaskarr/qwen3-4b-coder-ft-gguf --jinja).
bash
# enginesgit clone --depth 1 --branch v0.7.0 https://github.com/ggml-org/llama.cpp && cd llama.cppcmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build -j --target llama-server llama-bench # speed./build/bin/llama-bench -m <model>.gguf -p 512 -n 128 -r 3 -t 1,2 # serve (chat template via --jinja)./build/bin/llama-server -m <model>.gguf -c 1280 -t 2 --jinja --temp 0.0 --top-p 1.0Raw per-item outputs: benchmark/results/ · suite definition: benchmark/harness/bench_suite.py · runner: benchmark/harness/run_bench.py
bibtex
@misc{bhaskar2026qwen3coderft, title = {Comparative Benchmark Evaluation of a QLoRA Fine-Tuned Qwen3-4B Coder Model and its GGUF Quantization}, author = {Bhaskar, Mayank}, year = {2026}, month = {October}, url = {https://huggingface.co/mayankbhaskarr}}from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "mayankbhaskarr/qwen3-4b-coder-ft"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto") messages = [{"role": "user", "content": "Implement binary search in Python."}]inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(inputs, max_new_tokens=512, do_sample=False)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))# enginesgit clone --depth 1 --branch v0.7.0 https://github.com/ggml-org/llama.cpp && cd llama.cppcmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build -j --target llama-server llama-bench # speed./build/bin/llama-bench -m <model>.gguf -p 512 -n 128 -r 3 -t 1,2 # serve (chat template via --jinja)./build/bin/llama-server -m <model>.gguf -c 1280 -t 2 --jinja --temp 0.0 --top-p 1.0@misc{bhaskar2026qwen3coderft, title = {Comparative Benchmark Evaluation of a QLoRA Fine-Tuned Qwen3-4B Coder Model and its GGUF Quantization}, author = {Bhaskar, Mayank}, year = {2026}, month = {October}, url = {https://huggingface.co/mayankbhaskarr}}