Qwen3.8-27B-NVFP4 API & Inference Endpoint

Calibration

Post-training quantization with 1024 samples (max. 512 tokens each), 256 each from four openly available datasets: cnn_dailymail, Magpie-Align/Magpie-Pro-MT-300K-v0.1, nvidia/OpenCodeReasoning, nvidia/OpenMathReasoning. Attention implementation during calibration: SDPA.

Serving with vLLM

bash
vllm serve <this-repo> \
  --quantization modelopt \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

NVFP4 runs natively on Blackwell GPUs; the checkpoint loads on earlier architectures via vLLM's ModelOpt support with reduced benefit.

Notes

Community quantization by A2Genesis — not affiliated with or endorsed by the Qwen team or NVIDIA.
The MTP (multi-token prediction) tensors are preserved in BF16, so speculative decoding remains available.
License: Apache 2.0, inherited from the base model. All credit for the model itself belongs to the Qwen team.

Calibration

Serving with vLLM

bash
vllm serve <this-repo> \
  --quantization modelopt \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

NVFP4 runs natively on Blackwell GPUs; the checkpoint loads on earlier architectures via vLLM's ModelOpt support with reduced benefit.

Notes

Community quantization by A2Genesis — not affiliated with or endorsed by the Qwen team or NVIDIA.
The MTP (multi-token prediction) tensors are preserved in BF16, so speculative decoding remains available.
License: Apache 2.0, inherited from the base model. All credit for the model itself belongs to the Qwen team.

Qwen3.8-27B-NVFP4

README

Calibration

Serving with vLLM

Notes

Explore FriendliAI today

README

Calibration

Serving with vLLM

Notes