Huihui-Qwen3.6-35B-A3B-abliterated-int4-AutoRound

English

INT4 AutoRound quantization of huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated, optimized for NVIDIA DGX Spark (GB10 SM121) with Marlin INT4 kernel acceleration.

Model Details

Quantization Details

Layers Preserved in BF16

The following layers are not quantized to preserve model quality:

Performance

Tested on a single NVIDIA DGX Spark (GB10, 128GB LPDDR5X, SM121):

Speculative Decoding

This model supports two speculative decoding methods:

DFlash (requires separate drafter model):

bash
Copy code
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}'

Note: The DFlash drafter was trained on the original Qwen3.6-35B-A3B. Acceptance rate on the abliterated variant may be lower than on the original model.

MTP (uses built-in weights, no extra model needed):

bash
Copy code
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Serving with vLLM

bash
Copy code
vllm serve /path/to/model \
    --quantization moe_wna16 \
    --served-model-name qwen3.6-35b-a3b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

• Use --quantization moe_wna16 for Marlin INT4 kernel (SM121 compatible via SM120 binary compat)
• FP8 KV cache is not compatible with GDN non-causal attention layers; use --kv-cache-dtype auto
• NVFP4 falls back to Marlin W4A16 on SM121 (missing cvt.e2m1x2 PTX instruction)
• Runtime FP8 (--quantization fp8) is not compatible with DFlash (drafter inherits FP8 config and crashes)
• --language-model-only skips vision encoder profiling for text-only inference
• --performance-mode throughput enables CUDA graphs and kernels for throughput optimization
• Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Safety Warning

This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.

Credits

• Original Model: Qwen/Qwen3.6-35B-A3B by Alibaba Qwen Team
• Abliteration: huihui-ai
• INT4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
• Quantization Tool: Intel AutoRound

---

繁體中文

huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated 的 INT4 AutoRound 量化版本，針對 NVIDIA DGX Spark (GB10 SM121) 最佳化，使用 Marlin INT4 kernel 加速。

模型資訊

量化詳情

保留 BF16 的層

以下層未被量化以保持模型品質：

效能表現

在單台 NVIDIA DGX Spark (GB10, 128GB LPDDR5X, SM121) 上實測：

投機解碼

本模型支援兩種投機解碼方式：

DFlash（需額外下載 drafter 模型）：

bash
Copy code
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}'

注意：DFlash drafter 是以原版 Qwen3.6-35B-A3B 訓練的，在 abliterated 版本上的接受率可能較原版低。

MTP（使用內建權重，不需額外模型）：

bash
Copy code
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

使用 vLLM 部署

bash
Copy code
vllm serve /path/to/model \
    --quantization moe_wna16 \
    --served-model-name qwen3.6-35b-a3b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 65536 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) 相容性說明

• 使用 --quantization moe_wna16 啟用 Marlin INT4 kernel（SM121 透過 SM120 二進制相容性支援）
• FP8 KV cache 與 GDN non-causal attention 不相容，請使用 --kv-cache-dtype auto
• NVFP4 在 SM121 上會 fallback 到 Marlin W4A16（缺少 cvt.e2m1x2 PTX 指令）
• Runtime FP8（--quantization fp8）與 DFlash 不相容（drafter 繼承 FP8 config 導致 crash）
• --language-model-only 跳過視覺編碼器 profiling，加速純文字推理啟動
• --performance-mode throughput 啟用吞吐量最佳化的 CUDA graphs 和 kernel
• UMA 架構啟動前請先清除 page cache：sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

安全警告

此模型已移除安全過濾機制（abliterated），可能產生敏感、爭議性或不當內容。使用者須自行承擔所有風險與法律責任，並確保使用方式符合當地法規與倫理標準。不適用於公開或生產環境。

致謝

• 原始模型：Qwen/Qwen3.6-35B-A3B，Alibaba Qwen 團隊
• 去審查：huihui-ai
• INT4 量化：YuYu1015，於 NVIDIA DGX Spark (GB10) 上完成
• 量化工具：Intel AutoRound