razarath

Qwen3.8-27B-heretic-ara-GPTQ-W4A16-g64-int8-embed

README

License: apache-2.0

Results (RTX 4090, vLLM 0.30.0 with a small patch, measured 2026-10-10)

Fidelity: KL divergence against the BF16 model

Table with columns: This checkpoint, Same recipe, self-distilled calibration (not published)
	This checkpoint	Same recipe, self-distilled calibration (not published)
KLD mean	0.077	0.088
KLD median	0.0083	0.0094
KLD p90 / p99	0.146 / 1.29	0.147 / 1.54
Top-1 agreement with BF16	91.1%	91.5%
Agentic tool use (Toucan)	0.011	0.012
Coding agent (SWE-smith)	0.0093	0.0135
Code with reasoning (Mixture-of-Thoughts)	0.0067	0.0079
Multilingual (aya)	0.044	0.055
General chat (UltraChat)	0.170	0.189

How these numbers were measured:

64 held-out conversations (not used for calibration), 119k tokens, of which 43k are assistant tokens.
The BF16 reference and the quant both run in vLLM (prompt_logprobs, top-32 plus a tail bucket), so the quant is measured on the kernels it is served with.
Only assistant-turn tokens are scored. Chat models are trained only on those. Inside user, system and tool turns the BF16 model's own predictions are degenerate: on pasted files it predicts <|im_end|> at p≈0.99 mid-line. Quantization noise there dominates an all-token average (0.76 here) and says nothing about what the model writes.
UltraChat is the highest because its answers were written by another model. Even BF16's top-1 prediction matches that text only 56% of the time, against 73–85% for the code and agentic sets. It is high-entropy, off-policy text, where quantization noise shows the most.
Script: scripts/kld_vllm.py.

Serving on one RTX 4090 (24 GB, headless)

vLLM 0.30.0 with the patch below, FP8 KV cache, MTP speculative decoding with 3 tokens, thinking off, greedy.

Table with columns: Long context (1 sequence), Batch (6 sequences)
	Long context (1 sequence)	Batch (6 sequences)
Context	114,688 tokens (112k)	65,536 tokens each
KV pool	117,324 tokens (`--kv-cache-memory-bytes 5.0e9`)	101,395 tokens (`--gpu-memory-utilization 0.92`)
Decode	117–121 tok/s; MTP mean acceptance length 3.24–3.35	121 tok/s for a single stream; 31–32 clips/min for 6 clips of 16 frames each, with curation JSON
Needle recall (8 facts), filled to 97%	8/8 at 110,823 tokens; prefill 62 s (1,780 tok/s)	8/8 on each of 6 concurrent 16k-token prompts (95% of the pool), all done in 38 s

¹ Measured in a 128k configuration (--kv-cache-memory-bytes 5.5e9). That configuration passed at 90% depth, but its allocator was already retrying failed allocations, and at 97% (127k) the engine hung partway through prefill. Hence 112k and 5.0e9 above, which fill to 97% with no retries.

Functional checks on the checkpoint as downloaded from this repo, all passing:

<think> closes properly.
Tool calls parse with qwen3_xml, including a follow-up turn that reads a tool result.
Image: reads a word and a colour.
Video: describes the direction of motion.
Three borderline prompts (a roast, lock picking, a violent fiction scene) were answered without refusal. The refusal rate was not formally re-measured.

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Language-model Linear layers (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
MTP head (`mtp.fc`, `mtp.layers.*`)	INT8 symmetric, group size 128, RTN. It shares the INT8 embeddings and `lm_head`

The layout follows JRamirez-UAB's measured "v4" recipe for the base Qwen3.8-27B (card). This build differs in its calibration:

512 sequences × up to 2048 tokens (879k tokens), every one rendered with the model's own chat template, so the quantizer sees real <think> blocks, Qwen3-Coder XML tool calls and tool results, not plain text:

Table with columns: Share, Source, What it covers
Share	Source	What it covers
27% (140)	Agent-Ark/Toucan-1.5M (`Qwen3`)	multi-turn MCP tool-use trajectories, with tool schemas in the system prompt
20% (100)	SWE-bench/SWE-smith-trajectories (`tool`)	repository-level coding agent runs: bash and file-editor tool calls, long tool outputs
20% (100)	open-r1/Mixture-of-Thoughts

A calibration-only ablation (B) replaced most of this with the model's own answers: about 300 prompts from the same sources, answered by heretic-ara itself, with thinking on, at temperature 1.0. Self-distilled calibration is often recommended for abliterated models. Here it came out slightly worse (see Results), so the build above is the one published. scripts/gen_selfdistill.py is included in case you want to try it.

Tools: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130, on one RTX 4090 with the model held in CPU RAM (sequential onloading).

Requirements: vLLM with a small patch

Stock vLLM (0.29 / 0.30) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Apply the 4-line patch in patches/qwen3_5-embed-quant.patch (from syv-ai, Apache-2.0; see patches/LICENSE-syv-ai.txt). It has two hunks: the main model and the MTP draft module.

bash

VLLM_DIR=$(python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Serving

Long context for agentic coding (1 sequence, 112k, prefix caching on):

bash

vllm serve razarath/Qwen3.8-27B-heretic-ara-GPTQ-W4A16-g64-int8-embed \  --max-model-len 114688 --max-num-seqs 1 --max-num-batched-tokens 4096 \  --kv-cache-dtype fp8 --kv-cache-memory-bytes 5000000000 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --limit-mm-per-prompt '{"image":16,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}'

Batch (6 sequences × 64k, for example image-captioning jobs):

bash

vllm serve razarath/Qwen3.8-27B-heretic-ara-GPTQ-W4A16-g64-int8-embed \  --max-model-len 65536 --max-num-seqs 6 --max-num-batched-tokens 4096 \  --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --limit-mm-per-prompt '{"image":16,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}'

These numbers are for a headless card. If a desktop session also uses the GPU, lower --kv-cache-memory-bytes or --max-model-len by the VRAM it takes.

Notes

FP8 KV cache uses the default scale of 1.0. Do not pass --calculate-kv-scales: on hybrid GatedDeltaNet models it silently corrupts output (vllm#37554).
Size the KV cache with --kv-cache-memory-bytes. On this hybrid model, vLLM's memory profiling under-estimates prefill activations, so a --gpu-memory-utilization percentage can over-allocate.
reasoning_effort accepts xhigh (the default), medium and low, which are prompt instructions from the chat template. Thinking is on by default; send chat_template_kwargs: {"enable_thinking": false} to turn it off.
Tool calls use the Qwen3-Coder XML format: --tool-call-parser qwen3_xml (or qwen3_coder).

Files

model.safetensors + index, model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor)
scripts/: build_calib.py (calibration and eval sets), quantize.py (GPTQ pass), mtp_quant.py (INT8 MTP post-pass), kld_vllm.py (the KLD measurement above), gen_selfdistill.py (the self-distillation ablation), and the Dockerfile for the quantization environment. Together they reproduce this build end to end: build_calib.py → quantize.py (SOURCE=… CALIB=calib_A.jsonl quantize.py OUT 512 2048) → .

Credits

Base model: Qwen/Qwen3.8-27B; abliteration (ARA): heretic-org/Qwen3.8-27B-heretic-ara
Layout and MTP INT8 script: JRamirez-UAB; vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116
Tools: llm-compressor, compressed-tensors, vLLM

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

razarath

Model Tree

Base

heretic-org/Qwen3.8-27B-heretic-ara

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results (RTX 4090, vLLM 0.30.0 with a small patch, measured 2026-10-10)

Fidelity: KL divergence against the BF16 model

Table with columns: This checkpoint, Same recipe, self-distilled calibration (not published)
	This checkpoint	Same recipe, self-distilled calibration (not published)
KLD mean	0.077	0.088
KLD median	0.0083	0.0094
KLD p90 / p99	0.146 / 1.29	0.147 / 1.54
Top-1 agreement with BF16	91.1%	91.5%
Agentic tool use (Toucan)	0.011	0.012
Coding agent (SWE-smith)	0.0093	0.0135
Code with reasoning (Mixture-of-Thoughts)	0.0067	0.0079
Multilingual (aya)	0.044	0.055
General chat (UltraChat)	0.170	0.189

How these numbers were measured:

64 held-out conversations (not used for calibration), 119k tokens, of which 43k are assistant tokens.
The BF16 reference and the quant both run in vLLM (prompt_logprobs, top-32 plus a tail bucket), so the quant is measured on the kernels it is served with.
Only assistant-turn tokens are scored. Chat models are trained only on those. Inside user, system and tool turns the BF16 model's own predictions are degenerate: on pasted files it predicts <|im_end|> at p≈0.99 mid-line. Quantization noise there dominates an all-token average (0.76 here) and says nothing about what the model writes.
UltraChat is the highest because its answers were written by another model. Even BF16's top-1 prediction matches that text only 56% of the time, against 73–85% for the code and agentic sets. It is high-entropy, off-policy text, where quantization noise shows the most.
Script: scripts/kld_vllm.py.

Serving on one RTX 4090 (24 GB, headless)

vLLM 0.30.0 with the patch below, FP8 KV cache, MTP speculative decoding with 3 tokens, thinking off, greedy.

Table with columns: Long context (1 sequence), Batch (6 sequences)
	Long context (1 sequence)	Batch (6 sequences)
Context	114,688 tokens (112k)	65,536 tokens each
KV pool	117,324 tokens (`--kv-cache-memory-bytes 5.0e9`)	101,395 tokens (`--gpu-memory-utilization 0.92`)
Decode	117–121 tok/s; MTP mean acceptance length 3.24–3.35	121 tok/s for a single stream; 31–32 clips/min for 6 clips of 16 frames each, with curation JSON
Needle recall (8 facts), filled to 97%	8/8 at 110,823 tokens; prefill 62 s (1,780 tok/s)	8/8 on each of 6 concurrent 16k-token prompts (95% of the pool), all done in 38 s

Functional checks on the checkpoint as downloaded from this repo, all passing:

<think> closes properly.
Tool calls parse with qwen3_xml, including a follow-up turn that reads a tool result.
Image: reads a word and a colour.
Video: describes the direction of motion.
Three borderline prompts (a roast, lock picking, a violent fiction scene) were answered without refusal. The refusal rate was not formally re-measured.

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Language-model Linear layers (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 64, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT8 symmetric, group size 128, RTN (min-max)
MTP head (`mtp.fc`, `mtp.layers.*`)	INT8 symmetric, group size 128, RTN. It shares the INT8 embeddings and `lm_head`

The layout follows JRamirez-UAB's measured "v4" recipe for the base Qwen3.8-27B (card). This build differs in its calibration:

Table with columns: Share, Source, What it covers
Share	Source	What it covers
27% (140)	Agent-Ark/Toucan-1.5M (`Qwen3`)	multi-turn MCP tool-use trajectories, with tool schemas in the system prompt
20% (100)	SWE-bench/SWE-smith-trajectories (`tool`)	repository-level coding agent runs: bash and file-editor tool calls, long tool outputs
20% (100)	open-r1/Mixture-of-Thoughts

Tools: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130, on one RTX 4090 with the model held in CPU RAM (sequential onloading).

Requirements: vLLM with a small patch

bash

VLLM_DIR=$(python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Serving

Long context for agentic coding (1 sequence, 112k, prefix caching on):

bash

vllm serve razarath/Qwen3.8-27B-heretic-ara-GPTQ-W4A16-g64-int8-embed \  --max-model-len 114688 --max-num-seqs 1 --max-num-batched-tokens 4096 \  --kv-cache-dtype fp8 --kv-cache-memory-bytes 5000000000 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --limit-mm-per-prompt '{"image":16,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}'

Batch (6 sequences × 64k, for example image-captioning jobs):

bash

vllm serve razarath/Qwen3.8-27B-heretic-ara-GPTQ-W4A16-g64-int8-embed \  --max-model-len 65536 --max-num-seqs 6 --max-num-batched-tokens 4096 \  --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \  --limit-mm-per-prompt '{"image":16,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}'

These numbers are for a headless card. If a desktop session also uses the GPU, lower --kv-cache-memory-bytes or --max-model-len by the VRAM it takes.

Notes

FP8 KV cache uses the default scale of 1.0. Do not pass --calculate-kv-scales: on hybrid GatedDeltaNet models it silently corrupts output (vllm#37554).
Size the KV cache with --kv-cache-memory-bytes. On this hybrid model, vLLM's memory profiling under-estimates prefill activations, so a --gpu-memory-utilization percentage can over-allocate.
reasoning_effort accepts xhigh (the default), medium and low, which are prompt instructions from the chat template. Thinking is on by default; send chat_template_kwargs: {"enable_thinking": false} to turn it off.
Tool calls use the Qwen3-Coder XML format: --tool-call-parser qwen3_xml (or qwen3_coder).

Files

model.safetensors + index, model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor)
scripts/: build_calib.py (calibration and eval sets), quantize.py (GPTQ pass), mtp_quant.py (INT8 MTP post-pass), kld_vllm.py (the KLD measurement above), gen_selfdistill.py (the self-distillation ablation), and the Dockerfile for the quantization environment. Together they reproduce this build end to end: build_calib.py → quantize.py (SOURCE=… CALIB=calib_A.jsonl quantize.py OUT 512 2048) → .

Credits

Base model: Qwen/Qwen3.8-27B; abliteration (ARA): heretic-org/Qwen3.8-27B-heretic-ara
Layout and MTP INT8 script: JRamirez-UAB; vLLM patch: syv-ai; quantized-embedding layout first shown by Marcin116
Tools: llm-compressor, compressed-tensors, vLLM