GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sakamakismile
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sakamakismile
Model Tree
huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
Input Modalities
Output Modalities
Supported Functionality
All the power of Qwen3.6-35B-A3B with abliteration — no refusals for local agent workflows:
| Base model | huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated |
| Architecture | Qwen3.5 MoE — 35B total, 3B active, 256 experts (8 routed + 1 shared) |
| Quantization | NVFP4 W4A4 (weights FP4, activations FP4, scales FP8) |
| Format | compressed-tensors (native vLLM support) |
| Tool | vllm-project/llm-compressor (main) |
| Calibration | 512 samples, ultrachat_200k, seq_len=2048, moe_calibrate_all_experts=True |
| Size | 21.9 GB |
| Max context | 262,144 tokens (native) |
| Requires | NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130) |
bash
vllm serve Lna-Lab/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 \--max-model-len 32768 \--reasoning-parser qwen3 \--kv-cache-dtype fp8
bash
vllm serve Lna-Lab/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 \--max-model-len 32768 \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--kv-cache-dtype fp8
bash
docker run --gpus '"device=0"' -p 8016:8016 \-v /path/to/model:/models/current:ro \--shm-size 16gb \vllm/vllm-openai:cu130-nightly \vllm serve /models/current --port 8016 --max-model-len 32768 \--reasoning-parser qwen3 --kv-cache-dtype fp8
Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).
| Test | Speed | Tokens | Result |
|---|---|---|---|
| English (CAP theorem) | 160 tok/s | 256 | PASS |
| Code (async scheduler) | 161 tok/s | 512 | PASS |
| Math (Bayes' theorem) | 161 tok/s | 512 | PASS |
| Japanese (quantum computing) | 159 tok/s | 256 | PASS |
Sustained: ~168 tok/s (single GPU, container).
python
recipe = QuantizationModifier(targets="Linear",scheme="NVFP4",ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],)
moe_calibrate_all_experts=True — ensures all 256 experts receive calibration datapython
from transformers import Qwen3_5MoeForConditionalGeneration, AutoProcessor, AutoTokenizerfrom datasets import load_datasetfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierMODEL_ID = "huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated"model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)recipe = QuantizationModifier(targets="Linear", scheme="NVFP4",ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],)ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:512]")ds = ds.shuffle(seed=42)def preprocess(example):return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}ds = ds.map(preprocess)def tokenize(sample):return tokenizer(sample["text"], padding=False, max_length=2048,truncation=True, add_special_tokens=False)ds = ds.map(tokenize, remove_columns=ds.column_names)oneshot(model=model, dataset=ds, recipe=recipe,max_seq_length=2048, num_calibration_samples=512,moe_calibrate_all_experts=True)model.save_pretrained("Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4", save_compressed=True)processor.save_pretrained("Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4")tokenizer.save_pretrained("Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4")
| Package | Version |
|---|---|
| torch | 2.11.0+cu130 |
| transformers | 5.5.4 |
| llmcompressor | 0.1.dev5 (main) |
| compressed-tensors | 0.15.1a20260414 |
| CUDA | 13.0 |
--kv-cache-dtype fp8 for 2x KV capacity at no quality cost.bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge| Container burst (×3) | 168 tok/s | 512 | PASS — stable |