GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
will1208
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
will1208
Model Tree
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive
Input Modalities
Output Modalities
Supported Functionality
python
recipe = QuantizationModifier( targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$", "re:.*linear_attn.*", "re:^mtp.*"],)oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=1024, num_calibration_samples=128, moe_calibrate_all_experts=True, pipeline="basic")This repository has been smoke-tested locally on an RTX 5090 with vllm/vllm-openai:v0.21.0-cu130-local, compressed-tensors, NVFP4 Marlin GEMM, FP8 KV cache, and a real image chat.completions request.
bash
VLLM_USE_FLASHINFER_MOE_FP4=0 \VLLM_NVFP4_GEMM_BACKEND=marlin \vllm serve ./Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-NVFP4 \ --served-model-name qwen36-35b-a3b-hauhaucs-nvfp4 \ --quantization compressed-tensors \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.90 \ --max-model-len 4096 \ --max-num-seqs 1 \ --max-num-batched-tokens 1024 \ --trust-remote-codeFor short non-thinking answers, pass chat_template_kwargs at the top level of the OpenAI-compatible request:
json
{ "chat_template_kwargs": {"enable_thinking": false}}bash
VLLM_USE_FLASHINFER_MOE_FP4=0 \VLLM_NVFP4_GEMM_BACKEND=marlin \vllm serve ./Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-NVFP4 \ --quantization compressed-tensors \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.95 \ --max-model-len 100000 \ --max-num-seqs 1 \ --reasoning-parser qwen3 \ --language-model-only \ --trust-remote-codeConverted using li-yifei/gguf-to-nvfp4:
text
Q8_K_P GGUF → step1_convert_qwen36_moe.py → HF bf16 → step2_quantize_qwen36_moe.py → NVFP4recipe = QuantizationModifier( targets="Linear", scheme="NVFP4", ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$", "re:.*linear_attn.*", "re:^mtp.*"],)oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=1024, num_calibration_samples=128, moe_calibrate_all_experts=True, pipeline="basic")VLLM_USE_FLASHINFER_MOE_FP4=0 \VLLM_NVFP4_GEMM_BACKEND=marlin \vllm serve ./Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-NVFP4 \ --served-model-name qwen36-35b-a3b-hauhaucs-nvfp4 \ --quantization compressed-tensors \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.90 \ --max-model-len 4096 \ --max-num-seqs 1 \ --max-num-batched-tokens 1024 \ --trust-remote-code{ "chat_template_kwargs": {"enable_thinking": false}}VLLM_USE_FLASHINFER_MOE_FP4=0 \VLLM_NVFP4_GEMM_BACKEND=marlin \vllm serve ./Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-NVFP4 \ --quantization compressed-tensors \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.95 \ --max-model-len 100000 \ --max-num-seqs 1 \ --reasoning-parser qwen3 \ --language-model-only \ --trust-remote-code