GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
acyildirimer
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
acyildirimer
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
| Component | Precision |
|---|---|
| Language projections | 309 W4A16 NVFP4 modules + 92 FP8 W8A8 modules |
lm_head | W4A16 NVFP4, group size 16 |
| KV cache | FP8 scaling metadata; intended for an FP8 runtime cache |
| Vision tower | BF16 |
| MTP head | BF16 |
| Embeddings, norms, and control components | BF16 |
Quantization used NVIDIA ModelOpt 0.46.0. The 5.0-bit target applies to the
compatible quantized language weights, not the BF16 components or runtime KV
cache. Exact module assignments are recorded in hf_quant_config.json and
config.json.
The complete repository occupies approximately 20.39 GB (18.99 GiB).
Calibration used 1,024 fully populated 512-token samples from the
deterministic version-6 corpus (seed 42). Its token distribution was:
| Calibration source | Token share |
|---|---|
| Nemotron SFT Instruction Following Chat v2 | 20% |
| Nemotron Science v1 | 8% |
| Nemotron Competitive Programming v1 | 12% |
| Nemotron SFT Agentic v2 — function calling | 20% |
| Nemotron SFT Agentic v2 — web search |
The first row contains author-reported results for this checkpoint. The comparison rows use the benchmark table published for acyildirimer/Qwen3.8-27B-NVFP4.
Thinking was disabled during these quality benchmark runs to keep the evaluation conditions consistent and avoid mixing reasoning-mode differences into the checkpoint comparison.
| Checkpoint | Quantization | Backend | MMLU | ARC-C | HellaSwag | GSM8K | HumanEval | 4-task mean |
|---|---|---|---|---|---|---|---|---|
| acyildirimer/Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit | ModelOpt NVFP4 + FP8, 5-bit target | vLLM | 84.43 | 96.25 | 94.07 | 96.82 | 93.29 |
4-task mean is the arithmetic mean of MMLU, ARC-Challenge, HellaSwag, and
GSM8K. HumanEval is reported separately and is not included in that aggregate.
The first-row MMLU and HellaSwag results use their complete evaluation sets:
MMLU is 5-shot accuracy over 14,042 questions (11,856 correct), and
HellaSwag is 10-shot accuracy over 10,042 questions (9,447 correct). Both
were evaluated through vLLM chat completions with thinking disabled,
temperature 0, and strict generated-letter scoring. These are generative
accuracy results, not log-likelihood leaderboard scores.
The first-row results are author-reported. The remaining comparison results
originate from the public
witcheer/rtx-5090-benchmarks
dataset. The serving stacks differ, so the comparison is indicative rather than
a controlled quantization ablation.
Evaluated with tool-eval-bench v2.2.0 through vLLM at temperature 0, seed
42, across all 69 scenarios:
| Points | Normalized score | Outcomes |
|---|---|---|
| 127 / 138 | 92 / 100 | 60 passed, 7 partial, 2 failed |
One failure involved prompt-injection resistance. Tool-enabled deployments should enforce independent authorization and defenses for untrusted tool, file, and web content.
This checkpoint was designed for headless deployment on a dual-GPU system with 32 GB of total VRAM, while retaining the full native context window and vision support. Both memory-conscious configurations below are set to serve one request at a time.
A vLLM runtime with ModelOpt mixed NVFP4/FP8 support is required. This checkpoint was smoke-tested with vLLM 0.28.0.
bash
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \OMP_NUM_THREADS=1 \CUDA_DEVICE_ORDER=PCI_BUS_ID \CUDA_VISIBLE_DEVICES=0,1 \NCCL_P2P_DISABLE=1 \vllm serve acyildirimer/Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit \--host 0.0.0.0 \--port 8000 \--quantization modelopt \--tensor-parallel-size 2 \--safetensors-load-strategy lazy \--kv-cache-dtype fp8 \--max-model-len auto \--disable-custom-all-reduce \--gpu-memory-utilization 0.97 \--max-num-seqs 1 \--max-num-batched-tokens 2048 \--enable-chunked-prefill \--enable-prefix-caching \--performance-mode interactivity \--attention-backend auto \--reasoning-parser qwen3 \--enable-auto-tool-choice \--generation-config auto \--override-generation-config \'{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \--tool-call-parser qwen3_coder \--default-chat-template-kwargs \'{"preserve_thinking":true,"reasoning_effort":"xhigh"}'
Vision support is enabled by leaving out --language-model-only. This example
also disables NCCL P2P communication and vLLM's custom all-reduce path for a
dual-GPU system without reliable peer-to-peer connectivity.
The SGLang configuration below provides an equivalent multimodal setup.
The equivalent SGLang configuration is:
bash
CUDA_DEVICE_ORDER=PCI_BUS_ID \CUDA_VISIBLE_DEVICES=0,1 \NCCL_P2P_DISABLE=1 \OMP_NUM_THREADS=1 \PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \sglang serve \--trust-remote-code \--model-path acyildirimer/Qwen3.8-27B-AutoQuant-NVFP4-FP8-5bit \--tp 2 \--disable-custom-all-reduce \--enable-multimodal \--mem-fraction-static 0.93 \--attention-backend flashinfer \--kv-cache-dtype fp8_e4m3 \--chunked-prefill-size 2048 \--max-running-requests 1 \--cuda-graph-bs-decode 1 \--disable-prefill-cuda-graph \--mamba-radix-cache-strategy extra_buffer_lazy \--max-mamba-cache-size 4 \--mamba-ssm-dtype float32 \--reasoning-parser qwen3 \--tool-call-parser qwen3_coder \--default-chat-template-kwargs \'{"enable_thinking":true,"preserve_thinking":true,"reasoning_effort":"xhigh"}' \--host 0.0.0.0 \--port 8000
This configuration intentionally disables NCCL P2P communication and SGLang's custom all-reduce path for dual-GPU systems without reliable peer-to-peer connectivity.
This is a lossy post-training quantization. Results can vary with the evaluation protocol, runtime, GPU architecture, context length, and workload. The model also inherits the capabilities and limitations of the upstream checkpoint.
Refer to the Qwen3.8-27B model card for the authoritative architecture description, generation settings, vision preprocessing, and upstream limitations.
This checkpoint retains the upstream Apache 2.0 license and was quantized with NVIDIA ModelOpt.
| 5% |
| Nemotron Math v2 | 15% |
| Nemotron SFT SWE v2 — agentless SWE | 10% |
| Nemotron SFT SWE v2 — OpenHands SWE | 10% |
| acyildirimer/Qwen3.8-27B-NVFP4 | ModelOpt NVFP4 + FP8 | vLLM | 85.1 | 96.8 | 94.1 | 96.9 | 92.1 | 93.2 |
| Qwen/Qwen3.8-27B | BF16 | llama.cpp (partial offload) | 85.3 | 96.8 | 94.3 | 97.4 | 93.9 | 93.5 |
| unsloth/Qwen3.8-27B-NVFP4 | Dynamic V3 NVFP4 | vLLM | 84.3 | 96.9 | 94.3 | 97.1 | 89.6 | 93.2 |
| unsloth/Qwen3.8-27B-GGUF | Q8_0 | llama.cpp | 85.2 | 96.8 | 94.4 | 97.4 | 94.5 | 93.5 |
| unsloth/Qwen3.8-27B-GGUF | Q6_K | llama.cpp | 85.3 | 96.7 | 94.3 | 97.5 | 94.5 | 93.5 |
| unsloth/Qwen3.8-27B-GGUF | UD-Q4_K_XL | llama.cpp | 85.1 | 96.6 | 94.4 | 97.3 | 93.9 | 93.4 |
| unsloth/Qwen3.8-27B-GGUF | Q4_K_M | llama.cpp | 85.0 | 96.8 | 94.3 | 97.1 | 92.7 | 93.3 |