GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
NDIJayant
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
NDIJayant
Model Tree
Qwen/Qwen3.5-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.5-35B-A3B is a hybrid linear-attention + softmax-attention MoE
vision-language model: 40 layers, 256 routed experts at top-8, a SigLIP-style
vision tower, and an MTP head. A single blanket format does not suit it, so this
uses Model Optimizer's architecture-specific recipe
huggingface/qwen3_5_moe/ptq/nvfp4_experts_mse-fp8_rest-kv_fp8.
| Component | Format |
|---|---|
256 routed experts (gate_proj, up_proj, down_proj) — 93 % of parameters | NVFP4 W4A4, block size 16, static weight scales from an MSE FP8-scale sweep, dynamic input scales |
| Self-attention projections | FP8 W8A8, per-tensor, max-calibrated |
Linear-attention in_proj_qkv, in_proj_z, out_proj | FP8 W8A8 |
| KV cache | FP8 |
| Shared expert | FP8 W8A8 |
lm_head | BF16 |
| Vision tower (333 tensors) | BF16 |
| MTP head (785 tensors) | BF16, listed in quantization_config.ignore |
Routers/gates, conv1d, linear_attn.in_proj_a/b | BF16 |
Calibration: 512 samples × 512 tokens from cnn_dailymail. Only the static FP8
activation scales and the FP8 KV cache are data-fitted — the experts' NVFP4 input
scales are dynamic, and their weight scales derive from the weights themselves.
--moe-runner-backend flashinfer_cutlass is required. SGLang's default
flashinfer_trtllm MoE runner does not implement NVFP4 grouped GEMM and aborts
on the first forward with
NotImplementedError: Unsupported moe_runner_backend for NVFP4 MoE.
bash
python -m sglang.launch_server \--model-path NDIJayant/Qwen3.5-35B-A3B-NVFP4 \--moe-runner-backend flashinfer_cutlass \--attention-backend flashinfer \--reasoning-parser qwen3 \--trust-remote-code \--port 8000
On SM120 (RTX PRO 6000 Blackwell) this hybrid-GDN architecture is restricted to
the triton, trtllm_mha, and flashinfer attention backends; fa4 asserts at
startup.
No special flags needed — vLLM 0.29.0 loads and serves this checkpoint directly.
bash
vllm serve NDIJayant/Qwen3.5-35B-A3B-NVFP4 \--max-model-len 8192 \--trust-remote-code \--port 8000
Tested on 1× NVIDIA RTX PRO 6000 Blackwell (SM120, 96 GB).
Checkpoint structure
U8, half the
unpacked width) with both weight_scale and weight_scale_2 presentquantization_config.ignoreSGLang 0.5.6 — loads and serves; 17*23→391, capital of Japan→Tokyo,
144/12→12; coherent 1159-token open-ended generation.
vLLM 0.29.0 — loads and serves with no extra flags; same three prompts correct (3/3); coherent open-ended generation.
TensorRT-LLM has not been tested.
No formal benchmark suite (MMLU, GSM8K, etc.) has been run. If you need accuracy guarantees for a production deployment, evaluate on your own task set first.
One known behaviour, present on both engines: asked to spell a word backwards, the model derives the correct letter sequence but loops while transcribing it. This is a character-level tokenization pathology common to reasoning models and was not compared against the unquantized base model, so it is not established as a quantization artifact.
nvidia/Qwen3.6-35B-A3B-NVFP4 (a different model release, closest official
reference), this checkpoint is more conservative in two places: the shared
expert is FP8 rather than NVFP4, and lm_head stays BF16 rather than NVFP4.
Both follow Model Optimizer's shipped recipe for qwen3_5_moe.Full scripts, including the environment fixes this host needed, are in the conversion tooling this was built with. The essential command:
bash
python hf_ptq.py \--pyt_ckpt_path <local snapshot dir of Qwen/Qwen3.5-35B-A3B> \--recipe huggingface/qwen3_5_moe/ptq/nvfp4_experts_mse-fp8_rest-kv_fp8 \--export_path ./Qwen3.5-35B-A3B-NVFP4 \--export_fmt hf --dataset cnn_dailymail \--calib_size 512 --calib_seq 512 \--use_seq_device_map --gpu_max_mem_percentage 0.85 --trust_remote_code
--pyt_ckpt_path must be a local directory, not a repo id. Model Optimizer
recovers the MTP head by reading model.safetensors.index.json from that path on
the local filesystem; given a repo id it silently finds nothing and drops all 785
MTP tensors while still exiting 0.
Apache License 2.0 — the full text is in LICENSE, inherited from
Qwen/Qwen3.5-35B-A3B.
Anyone may use this model, including commercially. Apache-2.0 grants you a perpetual, worldwide, royalty-free licence to use, copy, modify, merge, publish, distribute, sublicense, and sell this model and works derived from it. No permission request and no fee is required.
What the licence asks of you in return:
markdown
Copyright 2026 Alibaba Cloud
The model is provided "AS IS", without warranties or conditions of any kind. See the licence for the full disclaimer and limitation of liability.
Per Apache-2.0 §4(b), the change relative to the upstream model: the weights have been post-training quantized to NVFP4/FP8 with NVIDIA Model Optimizer 0.46.1 as described above. No architecture change, no fine-tuning, no training data added.
Qwen/Qwen3.5-35B-A3B,
© 2026 Alibaba Cloud, Apache-2.0