Contents
Table with columns: File, Content, Precision| File | Content | Precision |
|---|
model.safetensors | language model (64 layers) | NVFP4, lm_head + embeddings BF16 |
model-visual.safetensors | vision tower + merger (333 tensors) | BF16, bit-identical to bytkim/Qwen3.8-27B-pi |
model_mtp.safetensors | MTP head (speculative decoding) | BF16 |
Vision fix: the first quantized export was loaded via AutoModelForCausalLM, which dropped the
vision tower. vLLM then initialised the ViT randomly and every image prompt degenerated into an endless
!!!! loop. In this repo the original vision weights have been restored (verified byte-for-byte against the source).
Quantization
- Scheme: NVFP4, targets
Linear
- Ignored:
lm_head, re:.*embed.*, re:.*visual.*, re:.*vision.*, re:.*merger.*
- Calibration: 512 samples of
HuggingFaceH4/ultrachat_200k, max_seq_length 2048
Serving (vLLM 0.30.0, TP=2)
vllm serve grandslam77/Qwen3.8-27B-pi-NVFP4 \ --served-model-name qwen3.8-27b --tensor-parallel-size 2 --trust-remote-code \ --kv-cache-dtype turboquant_k8v4 --max-model-len 150000 --max-num-seqs 2 \ --gpu-memory-utilization 0.95 --max-num-batched-tokens 16384 \ --enable-prefix-caching --enable-chunked-prefill \ --mm-encoder-tp-mode data --mm-encoder-attn-backend TRITON_ATTN \ --mamba-cache-dtype bfloat16 --mamba-ssm-cache-dtype bfloat16 \ --limit-mm-per-prompt '{"image": 4, "video": 0}' \ --mm-processor-kwargs '{"min_pixels": 65536, "max_pixels": 1003520}' \ --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice \ --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true, "reasoning_effort": "medium"}'
Recommended sampling (thinking / agent mode): temperature 0.7, top_p 0.95, top_k 20, min_p 0.05, presence_penalty 0, repetition_penalty 1.0.
License
Apache-2.0, inherited from bytkim/Qwen3.8-27B-pi and Qwen/Qwen3.8-27B.