hyssra

Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored-NVFP4-W4A4

README

License: apache-2.0

Quality Verification

All values below were benchmarked under identical conditions: lm-eval 0.4.12 (vLLM backend) in the vllm/vllm-openai:qwen38-x86_64-cu130 image, single RTX PRO 6000 Blackwell (96 GB), --gpu-memory-utilization 0.85, max_model_len 4096, greedy decoding. Prompts are 0-shot raw task prompts — no chat template, no thinking (equivalent to the "Instruct mode" the upstream card benches in). Worst relative degradation: 2.6 % (QC threshold 10 %).

Table with columns: Task, BF16, NVFP4 W4A4, Δ
Task	BF16	NVFP4 W4A4	Δ
arc-challenge	0.6126	0.6399	+2.7 pt
arc-easy	0.8624	0.8699	+0.8 pt
boolq	0.8758	0.9107	+3.5 pt
hellaswag (acc-norm)	0.8587	0.8382	−2.1 pt
openbookqa (acc-norm)	0.4560	0.4480	−0.8 pt
piqa	0.8215	0.8210	0.0 pt
winogrande	0.7830	0.7680	−1.5 pt
gsm8k (strict)	0.9386	0.9212	−1.7 pt
gsm8k (flexible)	0.9371	0.9212	−1.6 pt
truthfulqa_mc2	0.5330	0.5555	+2.3 pt
MMLU overall	0.8670	0.8500	−1.7 pt
— STEM	0.8658	0.8468	−1.9 pt
— Other	0.8822	0.8693	−1.3 pt
— Social sciences	0.9246	0.9126	−1.2 pt
— Humanities	0.8200	0.7983	−2.2 pt
wikitext-2 test PPL (65k tok)	6.2049	6.7095	+8.1 %
lambada PPL (lower is better)	4.1917	3.8602	−7.9 % (improved)

Reading: every task is within ~2.5 pt of the BF16 source; gsm8k keeps 98 % of the source score. The only notable W4A4 cost is the wikitext PPL (+8.1 %), the expected 4-bit language cost on this base. Absolute values differ from third-party harness numbers (e.g. the upstream card's llama.cpp benches); compare against the BF16 column, which was measured under the same conditions.

Chat template (patched in this repo)

This repo ships a hardened chat_template.jinja as the default, and the unmodified upstream template as chat_template-upstream-original.jinja for reference. The exact patch is in template_patch/chat_template.hardened.diff.

Upstream base: the template as of commit 9cdc928b of the upstream repo (2026-09-12, the agentic/tool-calling fix). That fix is folded into this file; everything else in it is upstream.

Potential risks in the upstream template as shipped. It implements an in-message mode switch by scanning messages for a literal {REASON: tag. For many uses this is never a problem, but the design carries several independent sharp edges:

Scope: it scans every message in the history — assistant, tool results, everything — not just what the user typed. Any text that merely quotes the feature (reading a file, fetching a URL, a RAG document, a pasted example) can silently switch the model's reasoning mode, or fail the request entirely.
Hard failure: a value outside the whitelist (
markdown
```
xhigh/medium/low/ einstein/spoon
```
) makes the template raise_exception, aborting the whole request — instead of being ignored.
Self-trigger: the template source itself contains the literal {REASON:, so the moment the file's own content appears in the conversation (e.g. an agent that inspects its own config), the parser extracts a code fragment as the "mode" and every later request in that session fails until a new chat is started.
Vision path: the scan pass renders every message as if it were a system message, so a user message containing an image or video fails the whole request with System message cannot contain images before generation even starts. In other words, served via vLLM, image input was not usable at all with this template. We think the upstream GGUF card's "Vision: tested" refers to the GGUF/mmproj path (llama.cpp), which does not run this jinja scan and never hit this code path — so the issue was not visible from the GGUF side.

The patch closes all four of these with a 7-line change to the parse block; every other line is byte-identical to upstream.

Semantics that differ from the upstream README. The mode switch is now deliberately conservative and stateless:

Table with columns: Behavior, Upstream, This repo
Behavior	Upstream	This repo
Which messages can carry the tag	any role, any position	last user message only, tag at the start of it
Unknown value (e.g. a typo)	request hard-fails	tag left in place, silently ignored
Tag found in tool/assistant text	switches mode / fails	inert (not scanned)
Mode persistence	lasts until changed again (client-side state — which mode you are actually in can depend on the client's history)	per-request only — prepend the tag each turn to keep a mode. The active mode is always either the default or a value you can see in the current message, which is also robust across harnesses and their context-compression implementations
Historical `reasoning_content` /

The five modes themselves and their system-prompt blocks are untouched. Verification: 24/24 robustness/repro checks (including both incident repros on the unmodified template) + byte-for-byte output parity with the original template for all 5 thinking + 5 instruct modes (local jinja2; same engine family as vLLM).

API note. The OpenAI-compatible top-level reasoning_effort field is still validated by vLLM against the standard 7-value ladder, so reasoning_effort="einstein"/"spoon" at the API level is rejected regardless of template. Use

markdown

extra_body={"chat_template_kwargs": {"reasoning_effort": "einstein"}}

, or the in-message tag.

If you prefer the exact upstream template, serve with --chat-template <repo>/chat_template-upstream-original.jinja.

Reproducibility

Built with llm-compressor oneshot + GPTQModifier (recipe.yaml included: block size 128, dampening 0.01, actorder static, MSE observer, group size 16) with a domain-matched calibration blend (long reasoning + general chat + code + GSM8K train math), on a GB10 (sm_121, CUDA 13.0, torch 2.11.0+cu130, transformers 5.14.1, llm-compressor 0.13.0), with the KV-cache calibration pass enabled (kv_cache_scheme: fp8, symmetric, per-tensor, memoryless_minmax) riding on the same one-shot pass.

What the KV calibration adds. 32 scalar FP32 tensors — k_scale/v_scale (one per full-attention layer's k_proj/v_proj; this model has 16 full-attention layers, the rest being Gated DeltaNet linear attention, which keeps a recurrent state and has no paged KV cache) — plus quantization_config.kv_cache_scheme in config.json. vLLM's compressed-tensors path reads these into the attention layers and multiplies the loaded K/V entries by the per-layer scale when quantizing them into the fp8 paged cache.

Post-quantization structural fixes (MTP restoration from the source's model-mtp-restored.safetensors, index.json rebuild, ignore guarantee for mtp.* in config.json, tower split into model-towers.safetensors) are idempotent and re-derivable from the BF16 source. Structural check: PASS (source keys 1199, missing 0). MTP draft head verified working with the speculative config below; vision tower verified structurally (weights intact).

Serve with vLLM

bash
vllm serve <this-repo> \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --kv-cache-dtype fp8 --enable-prefix-caching \
  --max-num-batched-tokens 2048 --max-num-seqs 2 \
  --gpu-memory-utilization 0.44 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

(Flag values are a tested GB10 / 128 GB unified-memory configuration — 256K context works at this setting. On a dedicated GB10 you can raise --gpu-memory-utilization toward ~0.8 for a larger KV pool; avoid ~0.95, which starves the OS on a unified pool.)

Engineering notes

This quantization run (calibration, QC, card, upload) was performed with an LLM agent operating the operator's own toolchain.
The chat template hardening (incident repro, patch, and test suites) was likewise done with the operator's agent; the diff is in template_patch/.

Lineage

Qwen/Qwen3.8-27B → DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU → DavidAU/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored → this. Apache 2.0, unbroken from the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hyssra

Model Tree

Base

DavidAU/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality