protoLabsAI

Qwythos-9B-v2-NVFP4

README

License: apache-2.0

What this is

NVFP4 W4A4 on the 128 transformer linears (attention + MLP). The hybrid DeltaNet (linear_attn) layers, the vision tower, lm_head, and the MTP head are kept BF16 — DeltaNet corrupts under 4-bit, and the vision path stays lossless.
11.2 GB on disk (vs 19.3 GB BF16 source; the BF16-preserved vision tower is most of the remainder).
MTP sidecar included (model-mtp.safetensors) — see the status note below.
Inherits the base wholesale: 1M-token context (YaRN), Qwen3.5 multimodal stack, FTPO looping-fix (greedy loop rate 6.7%→0%), uncensored for research/red-team/bio-chem-clinical work.

Speed

Single RTX PRO 6000 Blackwell (sm120), vLLM 0.24.0, NVFP4 --linear-backend marlin, BF16 KV:

Table with columns: Regime, tok/s, TPOT
Regime	tok/s	TPOT
single-stream (real text)	138	—
chat 1k/1k · C1	152	6.5 ms
chat 1k/1k · C8	1051	7.1 ms
chat 1k/1k · C32	2523	11.4 ms

(Client-side vllm bench serve, random dataset, fixed seed. --ignore-eos. Single trial.)

Serving (vLLM ≥ 0.24.0, Blackwell / sm120)

bash
vllm serve protoLabsAI/Qwythos-9B-v2-NVFP4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --linear-backend marlin \
  --max-model-len 65536 \
  --trust-remote-code

sm120 requires the FlashInfer/CUDA-13 env (or you hit the "no CUDA arch for major 12" crash): VLLM_USE_FLASHINFER_SAMPLER=0, CUDA_HOME=<cu13>, FLASHINFER_CUDA_ARCH_LIST=12.0f, NVCC_APPEND_FLAGS=-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK. --linear-backend marlin is the proven NVFP4 kernel on sm120 (fastest at the single-stream / low-batch regime that matters here).

Recipe

llm-compressor NVFP4, calibrated (512 samples, seq 2048). Ignore list: lm_head, re:.*visual.*, re:.*linear_attn.*, re:.*mtp.*. VL keys canonicalized post-quant. Reproducible from the BF16 source.

Quality gate — NVFP4 vs BF16 base (paired)

Every suite run against both the NVFP4 build and the BF16 source, same prompts/harness, on this rig. Deterministic suites are judge-free (solver-verified reasoning, execution-graded code); claw uses an independent LLM grader. Single trial (coding ×3).

Table with columns: Suite, NVFP4, BF16 base, Δ
Suite	NVFP4	BF16 base	Δ
Function-calling	89% (48/54)	94% (51/54)	−5 pp
Reasoning-v2 (solver-verified)	0.759	0.789	−0.03
Coding (exec-graded, ×3)	0.518	0.553	−0.03
Claw (agentic, 10 tasks)	0.651	0.614	+0.04

Near-parity. The only real cost is a few function-calling cases (single-trial, 54-task suite); reasoning and coding are within noise, agentic is flat-to-better, and there is no quant-rot at long context (verified to 60K). Expected NVFP4 behaviour for a 9B.

Status (2026-07-11)

✅ Serves + coherent output on vLLM 0.24.0/sm120; quality gate passed (table above).
⚠️ MTP sidecar is included but does not accelerate vLLM decoding for this checkpoint (0% draft acceptance). We isolated the cause: the sidecar is byte-identical in layout to a head that drafts at ~67% on the same NVFP4+MTP stack for another Qwen3.5-9B, so this is the head's weights, not our packaging or the runtime — the restored MTP head doesn't predict this FTPO-tuned checkpoint's distribution. The sidecar ships for completeness/llama.cpp; for speculative decoding today use Empero's GGUF-MTP. A distilled-head fix may follow.

Requests

Need a different size or format (GGUF, other quant, longer-context config)? Open a Community discussion — we usually ship within 48h.

License

apache-2.0, inherited from the base. Uncensored — this is Empero's research/red-team model; the quant changes numerics, not alignment. Use responsibly and within the base model's terms.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

protoLabsAI

Model Tree

Base

empero-ai/Qwythos-9B-v2

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities