🌟 Model Overview
Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-NVFP4-QAT is a native NVFP4 (W4A4) + FP8 KV Cache Quantization-Aware Trained (Scale-QAT) build of huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated (commit 14c13fcf593d692a5b96f29afecca6fe2fc030e6), itself an abliterated (uncensored) version of nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2.
Rather than relying on one-shot Post-Training Quantization (PTQ) alone, this checkpoint was trained with NVIDIA ModelOpt 0.48 (09a14b9ae89522363ffca9b0ac2245cbbb073fdd) using the learned-scale QAT recipe general/qad/nvfp4_lsq-mse_init-fp8_kv across 22,000 execution-filtered nvidia/OpenCodeInstruct tasks (average_test_score=1 with all-pass unit test execution records).
- Significant Loss Reduction over Initial Quantization: Held-out 64-example test loss dropped from
0.265642 (initial MSE + FP8 scale sweep) down to 0.158151 after Scale-QAT (-40.5% test loss, validation loss 0.241453 -> 0.136581 at best step 100).
- Full Composite Architecture Preserved: All 1,199 source weight keys are preserved, including 348 BF16 Vision Tower (27 ViT blocks) & Multi-Token Prediction (MTP) tensors alongside 400 packed NVFP4 language projection weights, packaged into 3 safetensors shards (~20.59 GB).
- Hardware-Verified on NVIDIA GB10 Grace Blackwell: Passed full-model offline vLLM
0.33.0rc1 (CUDA 13.4) smoke verification using the native modelopt_fp4 backend.
⚙️ Mandatory Thinking & Sampling Parameters
[!IMPORTANT]
Preserve Reasoning Traces: To obtain full reasoning performance and prevent premature cutoff, ensure your client or UI preserves <think> ... </think> tokens and exposes them in reasoning_content.
Table with columns: Parameter, Recommended Value, Purpose & Notes| Parameter | Recommended Value | Purpose & Notes |
|---|
temperature | 1.0 | Mandatory for Qwen3.8 EfficientThink reasoning exploration dynamics. |
top_p | 0.95 | Nucleus sampling threshold. |
top_k | 20 | Stabilizes token choices during multi-step reasoning. |
min_p | 0.0 | Disabled (avoids clipping low-probability reasoning branches). |
presence_penalty | 0.0 | Neutral presence penalty. |
repetition_penalty | 1.0 | Disabled (prevents damaging code indentation and syntax structures). |
reasoning_effort | xhigh | Activates deep multi-phase chain-of-thought exploration. |
max_tokens | 524,288 (512k) | Huge output budget for full repository solutions and exhaustive reasoning. |
context_window | 1,048,576 (1M) | 1 Million token context window supported with YaRN RoPE (262,144 native base). |
preserve_reasoning | true | Keep internal <think> traces intact. |
🧠 Transparent Architecture: Exactly How I Trained & Quantized It
[huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated (BF16, rev 14c13fcf)] │ ▼ [MSE + FP8 Scale-Sweep Initialization (512 Calibration Samples)] │ ▼ [NVIDIA ModelOpt NVFP4 Learned-Scale QAT (general/qad/nvfp4_lsq-mse_init-fp8_kv)] (1,520,435,200 trainable block-scale parameters, frozen BF16 weights, 22k OpenCodeInstruct) │ ▼ [Exact Tied-LSQ Scale Restoration & 3-Shard NVFP4 Safetensors Export] (400 packed W4A4 language projections + 348 BF16 Vision/MTP tensors, 1,199 keys preserved) │ ▼ [vLLM 0.33.0rc1 Verification on NVIDIA GB10 (CUDA 13.4, modelopt_fp4)] (Passed arithmetic, structured JSON, and Python code generation smoke suite)
1. The Huihui Coder390 EfficientThink Foundation
- Source Checkpoint:
huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated (revision 14c13fcf593d692a5b96f29afecca6fe2fc030e6).
- Lineage: Uncensored (layers 2–52 abliterated) derivative of
nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2, combining efficient chain-of-thought reasoning with strong software engineering synthesis.
2. Execution-Verified Coding Calibration & QAT Dataset
- Dataset:
nvidia/OpenCodeInstruct (revision 8f3ba5bafe4d6e8db46082cf7ae6741bc370604d, CC-BY-4.0).
- Strict Filtering: Examined 62,121 samples and selected 22,000 unique prompts (deduplicated by SHA-256) requiring
average_test_score=1, unit_tests present, and nonempty all-pass recorded execution statuses (seed=42).
- Split:
19,800 training / 1,100 validation / 1,100 test rows, with fixed 64-example evaluation and test subsets and a 4,096-token sequence cap with dynamic right-padding trimming (rounded to 128-token multiples). All tokenized rows retain assistant supervision.
3. NVFP4 W4A4 Learned-Scale QAT (general/qad/nvfp4_lsq-mse_init-fp8_kv)
- Why Scale-QAT? Full-weight Adam QAT on a 27B hybrid model exceeds single-workstation memory. Scale-QAT freezes the BF16 base weights and directly optimizes all 1,520,435,200 tied NVFP4 block-scale parameters (
group_size=16) initialized via MSE and an FP8 scale sweep over 512 calibration samples.
- Training Configuration: Official
QATTrainer, learning rate 1e-4, micro-batch size 1 with gradient accumulation 4, up to 200 optimizer steps (20,750.6s total wall time, 74.59 GiB peak CUDA allocated on GB10), evaluating every 50 steps with plateau stopping.
- Measured Loss Progression (
metrics.json):
Table with columns: Stage, Validation Loss (64-row), Held-Out Test Loss (64-row), Notes| Stage | Validation Loss (64-row) | Held-Out Test Loss (64-row) | Notes |
|---|
| Source BF16 Baseline | 0.256414 | 0.281261 | Unquantized BF16 reference |
| Initialized NVFP4 (Step 0) | 0.241453 | 0.265642 | MSE + FP8 scale sweep (512 calib samples) |
| Best Scale-QAT (Step 100) | 0.136581 | 0.158151 | Selected best scale checkpoint (selected_initial_scales: false) |
(Note: These are teacher-forced cross-entropy loss measurements on the held-out OpenCodeInstruct splits, not broad external coding benchmark scores.)
4. Precision Exclusions & Exact Scale Export
- Quantized Modules (
NVFP4 W4A4, group_size=16): 400 eligible language model linear projections.
- KV Cache: Quantized to FP8 (
kv_cache_quant_algo: "FP8").
- Higher-Precision Exclusions (Retained in BF16): Sensitive linear-attention gates and convolutions (
conv1d, in_proj_a, in_proj_b), token embeddings (embed_tokens), lm_head, all 27 Vision Tower ViT blocks (visual.*), and the Multi-Token Prediction (mtp.*) draft head (348 retained tensors verified bit-for-bit).
- Exact Scale Export Verification: Export restored all
1,520,435,200 saved scale parameters from step 100 (best_scales.pt). The 90 raw negative learned scale parameters across 6 quantizers derive the exact same clamped FP8-rounded physical scales during export as during fake-quant training, verified via real CUDA regression tests (scale_diagnostics.json).
🚀 Serving with vLLM (1M Context & 512k Output)
Verified on vLLM 0.33.0rc1.dev806+ga2bcb5657.d20261105 (vllm/vllm-openai CUDA 13.4 nightly on NVIDIA GB10) with --quantization modelopt_fp4:
vllm serve KatterMobile/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-NVFP4-QAT \ --served-model-name Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-NVFP4-QAT \ --quantization modelopt_fp4 \ --host 0.0.0.0 \ --port 8000 \ --max-model-len 1048576 \ --gpu-memory-utilization 0.88 \ --max-num-seqs 10 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --max-cudagraph-capture-size 32 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "xhigh", "preserve_thinking": true}' \ --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \ --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0, "max_new_tokens": 524288}'
(For lightweight local testing without 1M YaRN override, omit --hf-overrides to use the native 262,144 context window or pass --max-model-len 32768.)
💻 Quick Start with Pi Code CLI (pi)
Add to ~/.pi/agent/models.json:
{ "providers": { "vllm-home": { "baseUrl": "http://<YOUR_HOST_IP>:8000/v1", "api": "openai-completions", "compat": { "supportsDeveloperRole": false, "supportsReasoningEffort": true, "thinkingFormat": "qwen-chat-template", "chatTemplateKwargs": { "enable_thinking": true, "preserve_thinking": true } }, "models": [ { "id": "Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-NVFP4-QAT", "name": "Huihui Qwen3.8 27B Coder390 EfficientThink NVFP4 QAT", "reasoning": true, "contextWindow": 1048576, "maxTokens": 524288, "samplingParams": { "temperature": 1.0, "top_p": 0.95, "top_k": 20 } } ] } }}
Then run in your terminal:
pi --provider vllm-home --model Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-NVFP4-QAT
📦 Included Reproducibility & Audit Artifacts
To ensure complete transparency, the repository includes the machine-readable verification and provenance records generated during the GX10 Scale-QAT pipeline:
metrics.json — BF16, initialized-quant, and best Scale-QAT (step 100) eval/test losses, runtime, and CUDA memory telemetry.
provenance.json — Exact source SHA-256 digests, NVIDIA/Model-Optimizer commit (09a14b9ae89522363ffca9b0ac2245cbbb073fdd), package versions (torch 2.15.0, transformers 5.18.0, nvidia-modelopt 0.48.0rc1.dev323+g09a14b9ae), and recipe parameters.
data_manifest.json — nvidia/OpenCodeInstruct revision (8f3ba5bafe4d6e8db46082cf7ae6741bc370604d), filter criteria, and sample counts.
quantizers.json & scale_diagnostics.json — Per-layer quantizer configuration and learned scale distribution diagnostics.
vllm_smoke.json & installation.json — Offline vLLM 0.33.0rc1 (modelopt_fp4) artifact integrity and smoke test verification output on NVIDIA GB10.
⚠️ Usage Warnings
- Uncensored / Abliterated Base Model: Because the upstream base model (
huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated) had refusal directions abliterated across layers 2–52, safety filtering is significantly reduced. Users are solely responsible for reviewing outputs and ensuring compliance with applicable laws and safety requirements.
📜 Credits, Attribution & Licensing
Licensed under the Apache 2.0 License.