What differs from the main repository
Converted with convert_vllm.py (also in the main repo, inference/):
- FP8 attention / shared-expert / dense-MLP weights → BF16 (vLLM expects block-wise FP8 for attention and its FP8 linear kernel needs an NVRTC JIT).
- Per expert,
gate_proj / up_proj NVFP4 global scales equalized to min(Gg, Gu) with the block scales of the other projection rescaled (vLLM fuses them into one tensor).
input_activations removed from the quantization config — the checkpoint is weight-only; declaring activation quantization makes vLLM take the W4A4 path and produce NaN logits.
Serve (vLLM 0.31, Python 3.12)
pip install uv && uv venv --python-preference only-managed -p 3.12 /tmp/vvuv pip install --python /tmp/vv/bin/python vllm instanttensorV=$(/tmp/vv/bin/python -c "import flashinfer;print(flashinfer.__version__)")uv pip install --python /tmp/vv/bin/python "flashinfer-jit-cache==$V" --extra-index-url https://flashinfer.ai/whl/cu130# DeepGEMM JIT needs CUDA headers matching nvcc: uv pip install --python /tmp/vv/bin/python "nvidia-cuda-cccl==<nvcc version>.*"# and export CPATH=<site-packages>/nvidia/cu13/include/cccl NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK /tmp/vv/bin/vllm serve Asilarkness/GLM-5.3-Flash-Uncensored-NVFP4-Pruned-vLLM --served-model-name glm --max-model-len 65536 --gpu-memory-utilization 0.90 \ --max-num-seqs 32 --load-format instanttensor --no-enable-flashinfer-autotune --trust-remote-code \ --limit-mm-per-prompt '{"image":0,"video":0}' --enable-auto-tool-choice --tool-call-parser glm47
Measured on one RTX PRO 6000 Blackwell: ~47 tokens/s single stream, ~175 tokens/s with 8 parallel requests; agent tasks and tool calls verified (see the main repo).
The base model is abliterated (refusal-removed): outputs are unfiltered, use at your own responsibility.