GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
capyctl
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
capyctl
Model Tree
microsoft/FrogNano-4B-2609
Input Modalities
Output Modalities
Supported Functionality
| Layers | Format |
|---|---|
mlp.{gate,up,down}_proj (32 layers) and lm_head | NVFP4: FP4 (E2M1) weights with an FP8 (E4M3) scale per 16 values and an FP32 scale per tensor; NVFP4 input activations with a static input scale (W4A4) |
self_attn.{q,k,v,o}_proj (8 full-attention layers), linear_attn.{in_proj_qkv,in_proj_z,out_proj} (24 GatedDeltaNet layers) | FP8 (E4M3) weights and static FP8 input activations, one scale per tensor (W8A8) |
embed_tokens, linear_attn.in_proj_a / in_proj_b, conv1d, A_log, dt_bias, all norms, vision tower, MTP layer | BF16, bit-identical to the original |
| KV cache | Not quantized (kv_cache_quant_algo: null) |
| Output layer | Untied. The original ties lm_head to embed_tokens. Here lm_head starts as an exact copy of embed_tokens and is then quantized to NVFP4, while the embedding stays BF16. tie_word_embeddings: false. TensorFold's NVFP4 path reads a separate lm_head. |
| Architecture | Unchanged: Qwen3_5ForConditionalGeneration. The BF16 vision tower and MTP layer are kept so the checkpoint keeps the original's architecture name. The vision tower was not tested with this checkpoint. |
nvidia-modelopt 0.47.0 with transformers 5.14.1 and torch 2.13.0, using ModelOpt's examples/hf_ptq/hf_ptq.py from the 0.47.0 tag.huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast, changed in two ways to match the NVIDIA checkpoint: NVFP4 input quantizers on the MLP and lm_head (W4A4), and no KV-cache cast. Calibration algorithm max.test_sft split.The commands below, with these flags, were run against a local copy of this checkpoint on an RTX 4090 Laptop GPU (16 GB, SM 8.9). That GPU has no FP4 tensor cores, so every engine ran the NVFP4 layers as FP4 weights with BF16 activations (W4A16). GPUs with native FP4 (SM 10.x, SM 12.x) run them as FP4 x FP4.
vLLM 0.30.0:
bash
vllm serve capyctl/FrogNano-4B-2609-NVFP4 --max-model-len 16384 --gpu-memory-utilization 0.75 \ --tool-call-parser qwen3_coder --reasoning-parser qwen3 --enable-auto-tool-choicevLLM detects modelopt_mixed on its own. On SM 8.9 it picks its Marlin NVFP4 kernel and warns that the GPU has no native FP4.
SGLang 0.5.21:
bash
SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1 python -m sglang.launch_server \ --model-path capyctl/FrogNano-4B-2609-NVFP4 --context-length 16384 --mem-fraction-static 0.7 \ --fp4-gemm-backend marlin --reasoning-parser qwen3 --tool-call-parser qwen3_coderThe environment variable and --fp4-gemm-backend marlin are what made it start on SM 8.9: without the variable, SGLang's fused
SiLU and FP4-quantize kernel has no SM 8.9 build and CUDA graph capture fails. On GPUs with native FP4, these two settings
should not be needed, but that has not been tested yet.
TensorFold 0.6.3, CUDA:
bash
tensorfold serve capyctl/FrogNano-4B-2609-NVFP4 --no-drafts --context 16384 --backend cuda --snapshot-dir none--no-drafts is required: there is no drafter for this model. TensorFold skips the vision tower and MTP layer for NVFP4 checkpoints.
On SM 8.9 the first request after a fresh install waits a few minutes while its CUDA kernels compile.
A small greedy comparison against the original BF16 weights on vLLM 0.30.0, thinking off, max_tokens 1024.
Generated code was run against unit tests in a sandbox. This is a spot check, not SWE-bench or a perplexity measurement.
All runs on the RTX 4090 Laptop GPU, so the NVFP4 layers ran as W4A16.
| Prompts | This checkpoint, vLLM 0.30.0 | This checkpoint, TensorFold 0.6.3 | Original BF16, vLLM 0.30.0 |
|---|---|---|---|
| Python coding, 30 | 26 | 27 | 26 |
| Tool calls, 30 (right tool and arguments) | 29 | 29 | 28 |
| Reasoning and math, 20 | 20 | 20 | 20 |
| Total, 80 | 75 (94%) | 76 (95%) | 74 (93%) |
The differences from BF16 are one or two prompts, in both directions, which is within noise for a set this size. Every tool call was valid JSON, and 29 of 30 tool calls were identical to the BF16 ones. Every reasoning answer matched BF16's. The wording of longer answers drifts from BF16 after about 20 tokens.
With thinking on (10 prompts, max_tokens 4096) this checkpoint passed 9 on vLLM and 8 on TensorFold; BF16 passed 9.
On TensorFold, 2 of the 10 replies kept re-checking their answer until the token limit and never gave one;
on vLLM one similar reply finished just under the limit. BF16 did not do this. Set a max_tokens limit when thinking is on.
Speed and quality with native FP4 x FP4 (NVIDIA GB10) will be added after they are measured.
The original repository is tagged MIT, and its card's license row names Apache 2.0. This derived checkpoint keeps the original's terms and attribution; check the original repository for them. The calibration data (UltraChat 200k and Magicoder OSS-Instruct 75K) is MIT licensed and is not included in this repository.
| Tokenizer, chat template |
| The original files, unchanged. |
generation_config.json | Added, with eos_token_id: [248046, 248044] (<|im_end|>, <|endoftext|>), so replies stop at the end of a turn. |