GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
raoashish10
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
raoashish10
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Derived directly from Qwen3-8B-NVFP4 by:
quant_algo from NVFP4 to W4A16_NVFP4 in both
config.json and hf_quant_config.json, and removing the
input_activations entry from config.json's
quantization_config.config_groups.group_0 (there's no activation
quantization here, so nothing should claim there is).input_scale tensors from
model.safetensors — these were real, calibrated activation-scale
factors from the source checkpoint's W4A4 quantization, but they're
inert in a weight-only serving path and would misrepresent the
checkpoint if left in place next to a W4A16_NVFP4 label.The 4-bit weight tensors themselves (.weight, .weight_scale,
.weight_scale_2) are untouched — byte-identical to the source repo.
NVIDIA TensorRT-Model-Optimizer's weight quantizer is calibrated
identically regardless of whether activation quantization is also
enabled (both the W4A4 and W4A16 presets import the same weight-quantizer
config), so this is a subtractive-only change: nothing was re-quantized,
nothing about the weights changed, only the now-inapplicable
activation-quantization metadata and tensors were removed.
No metadata patch needed — this repo is correctly labeled, so vLLM's
modelopt_fp4 loader picks the weight-only (Marlin) path on its own:
bash
vllm serve raoashish10/Qwen3-8B-NVFP4-W4A16 --quantization modelopt_fp4
Verified on an RTX 5070 Ti (SM120) with vLLM 0.28.0: loads via
MarlinNvFp4LinearKernel and produces coherent output, no manual
quant_algo edit required. This checkpoint ships with Qwen3's "thinking
mode" on by default — pass
markdown
chat_template_kwargs={"enable_thinking": false}
Apache-2.0, inherited from the base Qwen/Qwen3-8B
model, same as the source Qwen3-8B-NVFP4 repo.