GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
ShadowsInTheNetwork
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ShadowsInTheNetwork
Model Tree
Qwen/Qwen3-Next-80B-A3B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This is a 4-bit floating-point (NVFP4) quantization of Qwen/Qwen3-Next-80B-A3B-Instruct, the non-thinking (instruct) model of the Qwen3-Next series: an 80B-parameter, high-sparsity Mixture-of-Experts model with about 3B active parameters per token and a hybrid Gated DeltaNet / Gated Attention architecture. Please see the original Qwen model page for the model's capabilities, benchmarks, intended use and limitations. All credit for the model itself goes to the Qwen team.
The weights were converted from the original BF16 checkpoint with llm-compressor to shrink the model to roughly 46 GB (43 GiB) on disk so it fits on a single NVIDIA GB10 (Blackwell, unified memory) system. The architecture, tokenizer and chat template are unchanged. This is a community conversion, not an official Qwen release, and no accuracy benchmarks are published here; compare against the BF16 model on your own workload before relying on it.
compressed-tensors, nvfp4-pack-quantized (4-bit float weights, group size 16, FP8 E4M3 scales)QuantizationModifier, scheme: nvfp4, targets: [Linear] (see recipe.yaml)lm_head, mlp.shared_expert_gate, linear_attn.in_proj_ba, and the other modules listed under quantization_config.ignore in config.jsonThis is a sanitized copy of the compose file used to serve the model. It targets an NVIDIA GB10 system with nvcr.io/nvidia/vllm:26.08-py3 (vLLM 0.27.1). vLLM detects the quantization from config.json, so no --quantization flag is needed. Replace the host paths with your own and adjust the memory settings to your hardware.
yaml
services: vllm: image: nvcr.io/nvidia/vllm:26.08-py3 container_name: qwen3-next-80b-a3b-instruct-nvfp4 environment: - NVIDIA_VISIBLE_DEVICES=all - NVIDIA_DRIVER_CAPABILITIES=compute,utility - VLLM_NVFP4_GEMM_BACKEND=marlin - VLLM_USE_FLASHINFER=0 volumes: - /path/to/Qwen3-Next-80B-A3B-Instruct-NVFP4:/model - /path/to/vllm-cache:/root/.cache/vllm ports: - "8000:8000" runtime: nvidia restart: "no" entrypoint: ["vllm"] command: ["serve", "/model", "--served-model-name", "qwen-Qwen3-Next-80B-A3B-Instruct-NVFP4", "--host", "0.0.0.0", "--port", "8000", "--trust-remote-code", "--max-model-len", "131072", "--max-num-batched-tokens", "24576", "--gpu-memory-utilization", "0.30", "--kv-cache-dtype", "fp8", "--kv-cache-memory-bytes", "22548578304", "--enable-prefix-caching", "--enable-chunked-prefill", "--max-num-seqs", "16", "--enable-auto-tool-choice", "--tool-call-parser", "hermes", "--no-enable-log-requests"]Notes on the settings:
VLLM_NVFP4_GEMM_BACKEND=marlin and VLLM_USE_FLASHINFER=0 are what this GB10 deployment used; other hardware or vLLM versions may not need them.--kv-cache-memory-bytes fixes the KV cache at 21 GiB (fp8), which vLLM reported as roughly 13x concurrency at the full 131072-token context on this deployment. When it is set, it takes precedence over --gpu-memory-utilization.--reasoning-parser is configured.Apache-2.0, inherited from the base model.
services: vllm: image: nvcr.io/nvidia/vllm:26.08-py3 container_name: qwen3-next-80b-a3b-instruct-nvfp4 environment: - NVIDIA_VISIBLE_DEVICES=all - NVIDIA_DRIVER_CAPABILITIES=compute,utility - VLLM_NVFP4_GEMM_BACKEND=marlin - VLLM_USE_FLASHINFER=0 volumes: - /path/to/Qwen3-Next-80B-A3B-Instruct-NVFP4:/model - /path/to/vllm-cache:/root/.cache/vllm ports: - "8000:8000" runtime: nvidia restart: "no" entrypoint: ["vllm"] command: ["serve", "/model", "--served-model-name", "qwen-Qwen3-Next-80B-A3B-Instruct-NVFP4", "--host", "0.0.0.0", "--port", "8000", "--trust-remote-code", "--max-model-len", "131072", "--max-num-batched-tokens", "24576", "--gpu-memory-utilization", "0.30", "--kv-cache-dtype", "fp8", "--kv-cache-memory-bytes", "22548578304", "--enable-prefix-caching", "--enable-chunked-prefill", "--max-num-seqs", "16", "--enable-auto-tool-choice", "--tool-call-parser", "hermes", "--no-enable-log-requests"]