GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
CocaKova
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
CocaKova
Model Tree
Agnes-AI/Agnes-3.0-Flash
Input Modalities
Output Modalities
Supported Functionality
parallel_ffn, 2048 wide) whose output is added to the main MLP. Adding two SwiGLUs on the same input is one wider SwiGLU, so gate/up were concatenated along the output dim, down along the input dim, and intermediate_size became 17408 + 2048 = 19456. The rename (delta_attn→linear_attn, global_attn→self_attn) is the same one the model's bundled SGLang patch applies at load time. Result: Qwen3_5ForConditionalGeneration, no trust_remote_code.modeling_agnes.py (transformers 5.12.1) vs the folded checkpoint through stock Qwen3_5ForConditionalGeneration (transformers 5.10.1): mean KL 5.0e-3, argmax agreement 99.2%, perplexity 164.3 vs 163.0. Same fold the bundled SGLang patch applies at load time.NVFP4 (weights + activations FP4, per-group-16 FP8 scales), 32 × 4096-token calibration samples from ultrachat_200k. Kept in BF16: lm_head, the vision tower, the delta-rule conv1d, and the MTP head (mtp.*, re-attached after quantization and listed in quantization_config.ignore). The MTP layer has no parallel branch in the original, so its SwiGLU was zero-padded from 17408 to 19456 (exact) so vLLM's draft model matches the folded width.Teacher-forced perplexity. Assistant-token row = the bf16 model's own greedy answers to three prompts (2,001 assistant tokens), scored under each model — the only positions an SFT model is trained on.
| text | bf16 (folded, vLLM) | NVFP4 (this build) |
|---|---|---|
| assistant tokens (bf16 greedy answers, 2,001 tok) | 1.131 | 1.259 |
| raw prose (facts, 58 tok) | 2.836 | 2.984 |
| Apache license paragraph (87 tok) | 1.099 | 1.149 |
| counting 1..30 (108 tok) | 1.108 | 1.131 |
Greedy answers from the two models diverge early (as 4-bit builds do) but both reach the same conclusions (e.g. π(200) = 46, correct merge function with tests). Tool calling (qwen3_xml parser) and reasoning (<think>) verified end-to-end.
Do not score system/user tokens: SFT never trains on them, both models are near-random there, and the numbers mean nothing (bf16 read 160 ppl on a user prompt, NVFP4 68 — and neither is "better").
bash
vllm serve CocaKova/Agnes-3.0-Flash-Preview-NVFP4 \ --served-model-name agnes-3.0-flash \ --max-model-len 131072 --kv-cache-dtype bfloat16 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser deepseek_r1Measured on a DGX Spark (GB10, sm_121a):
| single stream, thinking off | |
|---|---|
| decode, no speculation | 10.6 tok/s (memory-bandwidth floor for ~20 GB of weights) |
| decode, MTP k=3, prose | 15 tok/s (acceptance length ≈ 3.4) |
| decode, MTP k=3, code | 21 tok/s |
| decode, MTP k=3, highly predictable text | 26 tok/s (acceptance ≈ 4.0) |
| cold prefill, 31k-token prompt | 1,650 tok/s |
KV cache at --gpu-memory-utilization 0.35 | 206k tokens (bf16 KV) — the rest of the 128 GB stays free |
vLLM 0.20.2-dev tree built for sm_121a, FlashInferCutlassNvFp4LinearKernel, Triton/FLA GDN kernels, flashinfer attention.
The chat template is Agnes's own: chat_template_kwargs: {"reasoning_effort": "low"|"medium"|"xhigh"} (default xhigh; set it lower for agent work) and enable_thinking: false to disable reasoning.
vllm serve CocaKova/Agnes-3.0-Flash-Preview-NVFP4 \ --served-model-name agnes-3.0-flash \ --max-model-len 131072 --kv-cache-dtype bfloat16 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser deepseek_r1