GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
JongYeop
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
JongYeop
Model Tree
Base
Qwen/Qwen3-30B-A3B
Quantized
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
lm_head, MoE router (re:.*mlp.gate$)HuggingFaceH4/ultrachat_200k
(train_sft), max sequence length 2048NVFP4 fused-MoE inference requires a runtime kernel for your GPU. The cutlass
NVFP4 grouped-GEMM MoE kernel (get_cutlass_moe_mm_data in vLLM) is currently
compiled only for CUDA compute capability 9.0 (Hopper) and 10.0
(datacenter Blackwell, B200/GB200).
On SM120 GPUs (e.g. RTX PRO 6000 Blackwell, compute capability 12.0), the
stock vLLM build used to produce this checkpoint does not yet ship a compiled
NVFP4 MoE kernel, so vLLM serving fails with
No compiled get_cutlass_moe_mm_data: ... capability 120. Required capability: 90 or 100.
This is a runtime kernel limitation, not a problem with the checkpoint — the
quantized weights were verified (correct NVFP4 packing; router and lm_head left
in full precision). To serve this model, use a GPU/kernel combination that
provides an NVFP4 fused-MoE kernel (Hopper or B200), or a vLLM build with NVFP4
MoE kernels compiled for your architecture.
python
from vllm import LLM, SamplingParamsllm = LLM(model="JongYeop/Qwen3-30B-A3B-NVFP4-W4A4")out = llm.generate(["Explain mixture-of-experts in one sentence."],SamplingParams(temperature=0.7, max_tokens=128),)print(out[0].outputs[0].text)
yaml
quant_stage:quant_modifiers:QuantizationModifier:ignore: ["lm_head", "re:.*mlp.gate$"]scheme: "NVFP4"targets: ["Linear"]