GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
sakamakismile
Model Tree
Base
llm-jp/llm-jp-4-32b-a3b-thinking
Quantized
this model
Input Modalities
Text
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
Container
The base model's moe_intermediate_size is 960, which is not accepted by vLLM's fast NVFP4 MoE kernels (FLASHINFER_CUTLASS/MARLIN require 128-aligned, and 960/TP=480 is rejected).
This checkpoint therefore has each expert's intermediate dimension zero-padded from 960 to 1024 before quantization, and config.json reports moe_intermediate_size: 1024. The padding is mathematically lossless (the 64 extra units have zero gate/up output rows and zero down-projection columns, contributing exactly 0), but the tensor shapes differ from the base model — keep this in mind if you diff against llm-jp/llm-jp-4-32b-a3b-thinking.
bash
vllm serve sakamakismile/llm-jp-4-32b-a3b-thinking-NVFP4 \--trust-remote-code \--tensor-parallel-size 2 \ # or 4--max-model-len 16384 \--kv-cache-dtype fp8 \--gpu-memory-utilization 0.90# multi-GPU without NVLink/P2P: add --disable-custom-all-reduce and env NCCL_P2P_DISABLE=1
compressed-tensors quantization is auto-detected; FLASHINFER_CUTLASS NVFP4 MoE backend is selected automatically on Blackwell.analysis → final). vLLM 0.22's openai_gptoss reasoning parser raises NotImplementedError in non-streaming mode, so run without --reasoning-parser — the harmony text appears in content and the user-facing answer follows the final channel marker. (Use streaming if you want the parser to split reasoning_content.)llmjp4_tokenizer.py / llmjp4_harmony.py) ships with the model and loads via --trust-remote-code.On 2× / 4× NVIDIA RTX PRO 2000 Blackwell (16 GB, 288 GB/s, no NVLink), kv fp8, CUDA graphs:
| single-stream | aggregate | |
|---|---|---|
| TP=2 | ~119 tok/s | ~295 tok/s (4-way) |
| TP=4 | ~151–185 tok/s | ~465 (4-way) / ~750 (8-way) tok/s |
QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"]), calibrated on neuralmagic/calibration (LLM split). MoE experts are linearized for per-expert calibration (calibrate_all_experts).
Apache-2.0, inherited from the base model llm-jp/llm-jp-4-32b-a3b-thinking by LLM-jp. Please cite and follow the base model's terms. This is a community quantization; no affiliation with LLM-jp.