GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
abhishek085
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
abhishek085
Model Tree
Qwen/Qwen3-4B
Input Modalities
Output Modalities
Supported Functionality
v5 narrows scope on purpose: training and evaluation are now Jev-style decisions only — agent-harness control (tool-call verification,
context pruning, response-quality checks, delegation, error recovery, plan alignment), guardrails (prompt-injection and message-manipulation
gating, PII/output-leak checks), moderation and ticket routing, retrieval/re-ranking, and structured records decisions (entity resolution, fraud
and risk scoring, invoice/insurance/SOC routing). General-purpose text classification is no longer a training goal for this release. Training
data grew from v3's 967 rows to 11,792 rows / 19,568 examples across 49 task packs, drawn from os-datagen (LLM-written scenarios,
independently verified) plus new code-only rule packs where both the label and the surface text come from deterministic rules, no LLM in the
loop. Recipe: LoRA r16 (merged), lr 5e-5, 1 epoch (lower learning rate and fewer epochs than v3), cross-entropy plus a Brier regulariser.
The state is rendered once; each question is appended as a menu (A. ..., B. ..., option definitions in the prompt) followed by the Qwen3
non-thinking assistant header. The logits of the first answer token are restricted to the option-letter tokens (A..Z), divided by a
per-type temperature and softmaxed. That distribution is the answer. The weights are an ordinary Qwen3 causal LM; the readout is done by the
open-spark-Jev repository (https://github.com/abhishek085/open-spark-jev) code (open_spark_jev.model.MenuScorer, open_spark_jev/serve), which
also serves the POST /v1/decide API. calibration.json holds the temperatures, fitted for every question type this release: choice 2.026,
score 1.149, noul 1.318.
| split | accuracy | ECE (calibrated) |
|---|---|---|
| calibration | 0.886 | 0.016 |
| locked test | 0.867 | 0.019 |
| challenge | 0.861 | 0.026 |
| source | accuracy | ECE |
|---|---|---|
| Agent tool-call risk (60-row diagnostic set) | 0.917 | 0.062 |
| Jev-directory (70 questions) | 0.800 | 0.140 |
| Prompt injection, with deployment context | 0.894 | 0.105 |
| Prompt injection, no context | 0.903 | 0.098 |
| Kev decision-v1 (out-of-domain classification, external) | 0.755 | 0.184 |
Accuracy on Benchmark Heaven's JevBench v1.2, public items only (231 of 534 total; the judge tier and part of each other tier are private and were not run):
| tier | n | accuracy |
|---|---|---|
| easy | 48 | 1.000 |
| standard | 72 | 0.847 |
| hard | 111 | 0.523 |
This is not the official JevBench Score (that also needs the private item set plus Calibration, Speed and Cost axes measured under JevBench's own protocol); it is a same-methodology comparison point across our own releases.
Batch size 1, idle GPU, one NVIDIA DGX Spark (GB10):
| serving stack | precision | p50 | decisions/s |
|---|---|---|---|
| Hugging Face Transformers (in-process) | bf16 | 65.9 ms | 15.1 |
| vLLM | bf16 | 61.6 ms | 16.2 |
| vLLM | NVFP4, MLP layers only | 37.1 ms | 27.1 |
The HF Transformers bf16 number is carried over from the v3 release on the same backbone and prompt set (LoRA-merged weights don't change
inference cost); the vLLM numbers were measured fresh on this checkpoint. NVFP4 quantization (weights of the MLP projections only; attention and
the LM head stay bf16, since the option-letter readout depends directly on the LM head's logits) gives a real 1.66x speedup on the DGX
Spark's Blackwell tensor cores. Unlike on the 1.7B checkpoint (where the same recipe cost 9-13 accuracy points and was not shipped), the 4B
tolerates it much better — see the accuracy table below. The NVFP4 checkpoint is published separately at
abhishek085/spark-s1-4b-v5-nvfp4 (4.0 GB, needs a Blackwell-generation GPU and
vLLM/TensorRT-LLM to serve); it can also be reproduced with scripts/quant/ptq_nvfp4.py --cfg NVFP4_MLP_ONLY_CFG in the source repository.
| tier | bf16 | NVFP4-MLP | Δ |
|---|---|---|---|
| easy | 1.000 | 1.000 | 0.0 |
| standard | 0.847 | 0.847 | 0.0 |
| hard | 0.523 | 0.477 | -4.6 |
Unlike the 1.7B (where the same MLP-only recipe cost 9-13 points across every external set), the 4B holds its easy/standard-tier accuracy exactly and only gives up 4.6 points on the hard tier. This is the only accuracy check run on the NVFP4 variant so far (JevBench public tiers); it has not yet been scored on the own-splits/external-set battery the bf16 numbers above use, so treat the accuracy cost as smaller than the 1.7B's but not fully characterized outside JevBench.
Code: github.com/abhishek085/open-spark-jev. Training: configs/train/sft_v5.yaml
(python -m open_spark_jev.train.sft --config configs/train/sft_v5.yaml --set model=Qwen/Qwen3-4B output_dir=checkpoints/v5-4b). Evaluation:
python -m open_spark_jev.eval.osdg, python -m open_spark_jev.eval.external. Single seed; no confidence intervals reported yet.
Apache-2.0 (the Qwen/Qwen3-4B backbone is Apache-2.0). The training data was generated and verified with Gemma-4-26B-A4B, Qwen3.6-35B-A3B, Qwen3.6-27B and NVIDIA-Nemotron-3-Super-120B-A12B (used as a writer for a subset of the harness packs); check those terms for commercial use.