lyf

Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL

README

License: apache-2.0

Quick start — RTX 5090 / Blackwell

Primary runtime: vLLM's Qwen3.8 CUDA 13 build. The packed NVFP4 weights, multimodal path, and native MTP drafter were validated with the pinned image below. Hugging Face may display an automatic 8-bit badge because two FP4 values are physically packed in each U8 storage element; the quantization is NVFP4 W4A4.

Download

bash
hf download lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --local-dir ./qwen38-nvfp4

Allow roughly 23 GB for the release plus temporary/cache headroom.

Recommended text-only long-context profile — 192K

This is the practical RTX 5090 profile: FP8 KV, one sequence, chunked prefill, prefix caching, native MTP n=3, FlashInfer/CUTLASS NVFP4 GEMM, and the vision graph disabled to preserve KV capacity.

bash
docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp \
  --host 0.0.0.0 --port 8000 \
  --language-model-only \
  --max-model-len 196608 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Aggressive text-only profile — 256K

Use the same command with:

text
--max-model-len 262144
--gpu-memory-utilization 0.96
--max-num-batched-tokens 8192

This is a single-concurrency capacity target. Add --enforce-eager when CUDA Graph capture or workspace overhead prevents startup; eager mode trades some performance for lower graph-memory pressure. Start with 192K for a stable operational profile.

Full image/video profile

The vision tower and encoder cache need additional VRAM. This pinned profile is the verified smoke-test configuration:

bash
docker run --rm --gpus all --ipc=host --network=host \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai@sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 4096 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --enable-prefix-caching \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For multimodal capacity tuning, 32K is the conservative starting point, 64K–96K is the practical target range, and 128K is aggressive on 32 GB. Validate the chosen image/video limits and encoder-cache budget on your runtime before production.

OpenAI-compatible text request

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"qwen38-nvfp4-mtp",
    "messages":[{"role":"user","content":"Explain speculative decoding briefly."}],
    "max_tokens":256,
    "chat_template_kwargs":{"enable_thinking":false}
  }'

Image request

python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="qwen38-nvfp4-mtp-vl",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
            {"type": "text", "text": "Describe this image."},
        ],
    }],
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(r.choices[0].message.content)

RTX 5090 optimization notes

Keep the GPU power limit at the operator-selected 400 W for sustained runs.
flashinfer-cutlass is the preferred NVFP4 GEMM backend on SM120; the validated runtime also logged FlashInferCutlassNvFp4LinearKernel.
FP8 KV is essential for 192K–262K capacity. BF16 KV roughly doubles KV memory.
max_num_seqs=1 reserves capacity for one very long conversation. Increase concurrency only after lowering context or measuring free KV blocks.
Chunked prefill controls peak scheduling pressure for long prompts; prefix caching accelerates repeated prefixes.
expandable_segments:True reduces allocator fragmentation during long-context and multimodal workloads.
Keep MTP at n=3 as the validated speed/acceptance point. Workload-specific acceptance determines the actual gain.
The pinned digest is the compatibility contract. A newer vLLM tag should be revalidated for Qwen3.8, compressed-tensors NVFP4, vision/video, MTP, tool parser, and long-context allocation.
The model card's throughput values come from 4K validation runs. 192K/262K settings are capacity profiles; long-prompt prefill throughput is workload-dependent.

Lineage

Table with columns: Component, Source
Component	Source
Official architecture	Qwen/Qwen3.8-27B
Quantized source	heretic-org/Qwen3.8-27B-heretic-ara @ `2dc9b364104881cbb85e390f00195ba6b9d745e9`
Behavioral lineage	Heretic / ARA / abliterated
Vision/video tower	333 tensors retained from the same upstream checkpoint, BF16
MTP head	15 tensors retained from the same upstream checkpoint, BF16
PTQ	NVIDIA Model Optimizer NVFP4 W4A4

Heretic v1.2.0+custom ARA de-censoring of Qwen/Qwen3.8-27B, published by trohrbaugh and mirrored under heretic-org. The two source repositories were checked and their model artifacts are byte-identical; the organization mirror adds base-model metadata.

What is quantized

Table with columns: Component, Format
Component	Format
Language-model Linear layers	NVFP4 W4A4, group size 16
Vision/video tower (`model.visual.*`, 333 tensors)	BF16
MTP (`mtp.*`, 15 tensors)	BF16
`lm_head` and token embeddings	BF16
Gated DeltaNet `conv1d`	BF16
Norms, biases and small state tensors	BF16/FP32 as exported

Checkpoint metadata:

text
quant_method: compressed-tensors
format: nvfp4-pack-quantized
tensors: 2687
weight files: ~20.56 GB
calibration: 20 samples × 8192 tokens = 163,840 tokens
calibration dataset: abisee/cnn_dailymail, config 3.0.0

Validation

Hardware and runtime:

text
GPU: NVIDIA GeForce RTX 5090 32 GB (SM120)
GPU power limit: 400 W
Runtime: vllm/vllm-openai:qwen38-x86_64-cu130
Runtime digest: sha256:d392f621bb3e372ecc09f0b0cb88099afe9fa05d37a0450de45eeb8c12b6787e
vLLM build: 0.1.dev19754+g3a0914114
PyTorch: 2.13.0+cu130
KV cache: FP8
MTP: n=3

Verified paths:

GET /health → HTTP 200
GET /v1/models → expected served model
OpenAI-compatible text request → HTTP 200
Real PNG request → HTTP 200: A red square and a blue circle are displayed on a white background.
Real MP4 request → HTTP 200: The video shows a red square and a blue circle on a white background.
Native MTP drafter loaded; embeddings and LM head shared with target model

Measured single-stream results from 1024-token generations:

Table with columns: Metric, Result
Metric	Result
Client-observed generation throughput with MTP	120.5–134.7 tok/s
Mean MTP acceptance length	2.83–2.86
Per-position acceptance	`0.777/0.592/0.463 and 0.808/0.599/0.456`
Average draft acceptance	61.1–62.1%
Runtime VRAM during multimodal request	~28,984 MiB

These are local measurements for the stated prompts/configuration. They are reproducibility evidence, not broad quality benchmarks or a guaranteed speedup over every non-MTP setup.

Compact reference command

bash
docker run --rm --gpus all --ipc=host --network=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:qwen38-x86_64-cu130 \
  --model lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL \
  --served-model-name qwen38-nvfp4-mtp-vl \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

For short non-thinking responses, pass this at the top level of the request:

json
{"chat_template_kwargs": {"enable_thinking": false}}

Use the pinned profiles in Quick start for production. The 192K text-only profile is the recommended RTX 5090 long-context setup; 262K is the aggressive single-sequence target. The compact command above remains a readable 32K multimodal example.

Reproduction outline

Load the full BF16 VLM with Qwen3_5ForConditionalGeneration using CPU/disk offload.
Quantize the language model with ModelOpt NVFP4_DEFAULT_CFG.
Calibrate on 20 CNN/DailyMail samples at sequence length 8192.
Keep lm_head, token embeddings, linear_attn.conv1d, visual.*, and mtp.* outside NVFP4.
Export the ModelOpt HF checkpoint.
Retain/graft the 15 MTP tensors from the same upstream source.
Convert ModelOpt tensor conventions to compressed-tensors using ModelOptNvfp4Converter:
- weight →

The converter excludes:

text
lm_head
model.language_model.embed_tokens
re:.*visual.*
re:.*conv1d.*
re:^mtp.*

See BUILD_MANIFEST.json, VALIDATION_REPORT.json, recipe.yaml, and SHA256SUMS in this repository.

Sources and acknowledgements

Qwen/Qwen3.8-27B — official architecture, tokenizer, vision/video stack, and trained MTP head
NVIDIA Model Optimizer — initial NVFP4 W4A4 PTQ/export
compressed-tensors — ModelOptNvfp4Converter and portable nvfp4-pack-quantized packaging
vLLM — Qwen3.8 serving, multimodal API, and MTP speculative decoding
sakamakismile/Qwen3.8-27B-MTP-NVFP4 — working Qwen3.8 compressed-tensors NVFP4+MTP runtime reference
sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP — earlier ModelOpt NVFP4+MTP recipe reference

All upstream behavioral changes belong to their respective source authors. This repository contributes quantization, packaging, and RTX 5090 runtime validation. It does not claim authorship of Qwen3.8, Heretic/ARA, or Blackfrost's model modifications.

Limitations

NVFP4 requires compatible NVIDIA Blackwell kernels/runtime.
The tested full multimodal profile uses most of a 32 GB RTX 5090.
FP8 KV scale warnings may appear when static q/prob scales are absent; the tested runtime uses fallback scale 1.0.
The calibration set is compact. Task-specific quality should be evaluated against the BF16 source before production use.
Abliterated/de-censored behavior comes from the upstream model. Operators are responsible for deployment policy and outputs.

License

Apache-2.0, following Qwen3.8 and the cited upstream checkpoint. The included LICENSE is copied from the upstream release.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

lyf

Model Tree

Base

heretic-org/Qwen3.8-27B-heretic-ara

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality