GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
AEON-7
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
AEON-7
Model Tree
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16
Input Modalities
Output Modalities
Supported Functionality
| Repo | Size | What it is | Target hardware |
|---|---|---|---|
| BF16 master | ~54G | Full-precision uncensored master. Abliterated for coherence, vision + MTP untouched. | H200 / multi-GPU / PRO 6000 when you want the teacher, not the deploy knife |
| This MIXED NVFP4 | ~23.8G (4 shards) | ModelOpt NVFP4 + FP8 + BF16 lattice (last-8 full MLP FP8) + native MTP + vision. | DGX Spark / GB10, RTX 5090, RTX PRO 6000 with AEON vLLM Ultimate images |
Same will. Different seats. The BF16 is the source of truth. This card is what you actually serve day to day.
I do not trust equal-bitwidth recipes on a hybrid GDN / full-attn stack. The first Mix B bake FP8'd only the last-8 down_proj and left gate/up on NVFP4. Coding cratered to 0.694 on AEON Bench unit tests — real syntax and assert fails, not a moody judge. Leftover-KL on those miss prefixes lit up layers 56–63. So the sibling bake did the honest thing: last-8 full MLP (gate + up + down) → FP8. Coding jumped to 0.833.
| Block | Format | Why |
|---|---|---|
| MLP layers 0–55 | NVFP4 W4A4 (MSE + fp8_scale_sweep) | Size + Blackwell NVFP4 throughput |
| MLP layers 56–63 | FP8 full (gate/up/down) | Coding fidelity where residual L2 spiked |
| Softmax attn + GDN writers | FP8 | Writers that corrupt if you force NVFP4 wrong |
| Vision tower + MTP head + embeddings + lm_head + GDN guts | BF16 | Untouched capability organs. MTP grafted from Ultimate BF16 (15 tensors, 0.791G) after ModelOpt export dropped it. |
Calib 1024×2048. Export gate: found=168 missing=0 last8_mlp_fp8=24. Parent BF16 untouched. Post-export: grafted unmodified mtp.* (15 BF16 tensors, 0.791G) from Ultimate BF16 — ModelOpt had dropped the head.
Local AEON Suite v4 run ceaac9e418 (GB10, think 32k / max 65k / retry 131k, c=8, mothership fail-closed):
| Board | This MIXED | Prior Mix B |
|---|---|---|
| Text mean | 0.782 | 0.759 |
| Coding | 0.833 | 0.694 |
| Hermes | 0.918 | ~0.917 |
| OpenClaw | 0.707 | 0.733 |
| OpenCode | 0.818 | 0.793 |
Attested mothership Global (aeon-bench.com, same MIXED tree): overall ~91.9, Perf dial 100, peak_agg 237.67 tok/s (wave-peak, Coding@c16). That peak is the Perf lane below — not the everyday chat recipe.
Use ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-11-v0.29.0-omni on DGX Spark / GB10 for the new engine tip (upstream vLLM 0.29.0, sm_121a bake, digest sha256:2421bb1228a85370c1c50adb31f605c4361acf4d48d65282fcb919e74f34fae7). It keeps the AEON carries (DFlash/DFlash2, reasoning-EOS force-end, NVFP4 paths) and is the cut where dual-Spark TP=2 stays coherent if you set the B0 knobs: VLLM_ALLREDUCE_USE_FLASHINFER=0, fuse_allreduce_rms=false, --disable-custom-all-reduce, and NCCL_IB_HCA=<dev>:1 with a single equals. TP=1 A/B vs 2026-09-07-reasoning-eos held quality and was slightly faster. MIXED still needs the ModelOpt #54367 bind on every rank — in-tree 0.29 ModelOpt alone does not load this lattice. Pin the dated tag until you confirm GHCR :latest shares that digest; rollback remains :2026-09-07-reasoning-eos.
Use ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest on RTX 5090 and RTX PRO 6000 (sm_120; promoted from 2026-09-07-omni-mm = ModelOpt #54367 Spark-parity multimodal/omni stack). Pin dated RTX tags if you want known-good: (or for #54367-only). Do cross Spark and RTX images. Spark is sm_121a / aarch64 UMA. The RTX image is sm_120 / amd64 dedicated VRAM.
Vision / video / audio: RTX :latest / :2026-09-07-omni-mm registers the multimodal processor for Qwen3_5ForConditionalGeneration (same family as Spark omni) and layers vllm-omni==0.27.0rc1 for Omni speech-out models. Keep --limit-mm-per-prompt as in the recipes below. Re-pull: docker pull ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest.
Stale Aug-21 RTX :latest symptom: 'MergedColumnParallelLinear' object has no attribute 'data' during fused gate_up modelopt_mixed load — bind-mount modelopt-54367.py until you pull the 2026-09-07 image.
Gen defaults for all seats (Qwen3.8 card): temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05. Thinking on: chat_template_kwargs={"enable_thinking": true, "reasoning_effort": "medium"}.
| Rule | Detail |
|---|---|
No --quantization | Leave it unset so hf_quant_config.json selects modelopt_mixed. Never pass compressed-tensors, nvfp4, or modelopt_fp4 on this tree. |
| #54367 ModelOpt patch | Required on Spark 2026-09-11-v0.29.0-omni for MIXED (in-tree 0.29 ModelOpt does not load this lattice alone). Folded into Spark 2026-09-07-reasoning-eos, and into RTX 2026-09-07-modelopt / 2026-09-07-omni-mm RTX . Bind-mount -style (GDN + ) over Spark (both TP ranks) or RTX . Full file: on this repo. |
Hugging Face Hub snapshot dirs under ~/.cache/huggingface/hub/models--…/snapshots/<rev> are symlink trees into blobs/. Mounting only the snapshot path into Docker often yields dangling symlinks inside the container. Prefer one of:
-v $HOME/.cache/huggingface:/root/.cache/huggingface:ro then vllm serve AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED --revision <rev> …huggingface-cli download … --local-dir tree.Do not mount only …/snapshots/<rev> unless you have verified every weight path resolves inside the container.
| Seat | max-model-len | seqs | util | Spec | YaRN |
|---|---|---|---|---|---|
| 1× Spark quality | 131072 | 8 | 0.70 | DFlash2 n=7 | off |
| 1× Spark speed | 65536 | 8 | 0.60 | DFlash2 n=7 | off |
| 1× Spark Perf | 262144 |
Native window, seqs 8, util 0.70, TRITON_ATTN, DFlash2 n=7, KV fp8. #54367 folded on :latest / 2026-09-07-reasoning-eos (bind-mount optional/legacy).
bash
MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>DRAFT=/path/to/z-lab__Qwen3.8-27B-DFlash2PATCH=/path/to/modelopt-54367.py # optional on :latest / 2026-09-07-reasoning-eos; required on pre-2026-09-07IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate:latestdocker rm -f aeon-mixed-spark 2>/dev/nulldocker run -d --gpus all --network host \--name aeon-mixed-spark \-e VLLM_USE_V2_MODEL_RUNNER=0 \-v "$MODEL:/model:ro" -v "$DRAFT:/draft:ro" \-v "$PATCH:/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--served-model-name aeon \--host 0.0.0.0 --port 8000 \--gpu-memory-utilization 0.70 \--max-model-len 131072 \--max-num-seqs 8 \--max-num-batched-tokens 8192 \--kv-cache-dtype fp8 \--enable-chunked-prefill \--no-enable-prefix-caching \--tool-call-parser qwen3_coder \--enable-auto-tool-choice \--reasoning-parser qwen3 \--limit-mm-per-prompt '{"image":4,"video":2}' \--attention-backend TRITON_ATTN \--trust-remote-code \--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":7}' \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}'
Ready when /v1/models lists aeon. Smoke: chat completions, model=aeon, short prompt.
Same image, draft, patch, and DFlash2 n=7, but:
--max-model-len 65536--gpu-memory-utilization 0.60VLLM_ALLOW_LONG_MAX_MODEL_LEN, no --hf-overrides)Peak tok/s seat on one GB10. Same 2026-09-07-reasoning-eos / :latest image — not 0.29. Dynamic DFlash2 via num_speculative_tokens_per_batch_size, MRv2 + FULL_AND_PIECEWISE, util 0.85, seqs 16, KV fp8, chunked prefill, no prefix cache, TRITON_ATTN. Everyday chat stays the quality recipe above (DFlash2 n=7 / util 0.70 / seqs 8).
Measured AEON Bench Perf peak overall (wave-peak concurrent window, not decode×conc): 237.67 tok/s Coding@c16.
bash
MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>DRAFT=/path/to/z-lab__Qwen3.8-27B-DFlash2PATCH=/path/to/modelopt-54367.py # optional on :latest / 2026-09-07-reasoning-eos; required on pre-2026-09-07IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-11-v0.29.0-omni # pin dated until :latest digest matches 2421bb…; rollback :2026-09-07-reasoning-eosdocker rm -f aeon-mixed-spark-perf 2>/dev/nulldocker run -d --gpus all --network host \--name aeon-mixed-spark-perf \-e VLLM_USE_V2_MODEL_RUNNER=1 \-e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-v "$MODEL:/model:ro" -v "$DRAFT:/draft:ro" \-v "$PATCH:/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--served-model-name aeon \--host 0.0.0.0 --port 8000 \--gpu-memory-utilization 0.85 \--max-model-len 262144 \--max-num-seqs 16 \--max-num-batched-tokens 16384 \--kv-cache-dtype fp8 \--enable-chunked-prefill \--no-enable-prefix-caching \--tool-call-parser qwen3_coder \--enable-auto-tool-choice \--reasoning-parser qwen3 \--limit-mm-per-prompt '{"image":4,"video":2}' \--attention-backend TRITON_ATTN \--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \--trust-remote-code \--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":10,"num_speculative_tokens_per_batch_size":[[1,1,10],[2,2,10],[3,4,8],[5,8,7],[9,10,6],[11,12,5],[13,14,4],[15,16,3]],"attention_backend":"TRITON_ATTN"}' \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}'
Runtime K by batch: c1–2→10, c3–4→8, c5–8→7, c9–10→6, c11–12→5, c13–14→4, c15–16→3.
Use only when two Sparks share a fast link (InfiniBand or RoCE). Each box needs the same $MODEL, $DRAFT, $PATCH, and image. Start rank 1 (headless) first, then rank 0 (API). Clients use http://$MASTER_IP:8000.
bash
MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>DRAFT=/path/to/z-lab__Qwen3.8-27B-DFlash2PATCH=/path/to/modelopt-54367.pyIMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate:latestMASTER_IP=10.10.10.1IFACE=enp1s0f0np0IB_HCA==rocep1s0f0:1HF_OVERRIDES='{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}'
Change MASTER_IP, IFACE, and IB_HCA to this pair (ip -br link). The == in IB_HCA is required (NCCL exact match). No InfiniBand? Drop --device /dev/infiniband, -e NCCL_IB_HCA=..., -e NCCL_IB_GID_INDEX=3, and --cap-add IPC_LOCK; keep NCCL_SOCKET_IFNAME on the fast nic.
Do not pass --quantization. Bind-mount #54367. Set VLLM_USE_V2_MODEL_RUNNER=0.
On the second Spark:
bash
NODE_IP=10.10.10.2RANK=1NAME=tp2-node1docker rm -f "$NAME" 2>/dev/nulldocker run -d --name "$NAME" --gpus all --ipc=host --shm-size=16g --net=host \-e VLLM_HOST_IP="$NODE_IP" \-e NCCL_SOCKET_IFNAME="$IFACE" -e GLOO_SOCKET_IFNAME="$IFACE" \-e NCCL_IB_HCA="$IB_HCA" -e NCCL_IB_GID_INDEX=3 -e NCCL_DEBUG=WARN \--device /dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1:-1 \-e VLLM_USE_FLASHINFER_SAMPLER=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-e VLLM_USE_V2_MODEL_RUNNER=0 \-v "$MODEL":/model:ro -v "$DRAFT":/draft:ro \-v "$PATCH:/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--headless \--tensor-parallel-size 2 --nnodes 2 --node-rank 1 \--master-addr "$MASTER_IP" --master-port 29501 \--kv-cache-dtype fp8_e4m3 --attention-backend TRITON_ATTN \--max-model-len 1000000 --max-num-seqs 16 --max-num-batched-tokens 16384 \--gpu-memory-utilization 0.70 \--disable-custom-all-reduce \--enable-chunked-prefill --no-enable-prefix-caching \--mamba-cache-mode align \--limit-mm-per-prompt '{"image":4,"video":2}' \--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":7}' \--hf-overrides "$HF_OVERRIDES" \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}' \--trust-remote-code
Rank 1 must have --headless. Do not give it --host or --port.
On the first Spark:
bash
NODE_IP=10.10.10.1RANK=0NAME=tp2-node0docker rm -f "$NAME" 2>/dev/nulldocker run -d --name "$NAME" --gpus all --ipc=host --shm-size=16g --net=host \-e VLLM_HOST_IP="$NODE_IP" \-e NCCL_SOCKET_IFNAME="$IFACE" -e GLOO_SOCKET_IFNAME="$IFACE" \-e NCCL_IB_HCA="$IB_HCA" -e NCCL_IB_GID_INDEX=3 -e NCCL_DEBUG=WARN \--device /dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1:-1 \-e VLLM_USE_FLASHINFER_SAMPLER=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-e VLLM_USE_V2_MODEL_RUNNER=0 \-v "$MODEL":/model:ro -v "$DRAFT":/draft:ro \-v "$PATCH:/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--served-model-name aeon --host 0.0.0.0 --port 8000 \--tensor-parallel-size 2 --nnodes 2 --node-rank 0 \--master-addr "$MASTER_IP" --master-port 29501 \--kv-cache-dtype fp8_e4m3 --attention-backend TRITON_ATTN \--max-model-len 1000000 --max-num-seqs 16 --max-num-batched-tokens 16384 \--gpu-memory-utilization 0.70 \--disable-custom-all-reduce \--enable-chunked-prefill --no-enable-prefix-caching \--mamba-cache-mode align \--limit-mm-per-prompt '{"image":4,"video":2}' \--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":7}' \--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \--hf-overrides "$HF_OVERRIDES" \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}' \--trust-remote-code
bash
docker logs -f tp2-node0curl -s http://127.0.0.1:8000/health
1M on TP=2 is the long-context recipe. It is slower than 64k.
Same run as above, but:
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1--hf-overrides--max-model-len 65536--gpu-memory-utilization 0.60--quantizationaeon-vllm-ultimate-rtx:latest32 GB dedicated. Long-context chat seat: TRITON_ATTN + fp8_e4m3 KV (FlashInfer + fp8 KV can garbage on sm_120 at long ctx). Seqs 4, util 0.92–0.95, 131k, no YaRN. Native MTP n=3 is in the checkpoint (~1% KV cost).
aeon-vllm-ultimate-rtx:latest / 2026-09-07-omni-mm folds #54367 and Spark-parity vllm-omni (vision/video/audio + Omni speech-out). 2026-09-07-modelopt is the #54367-only intermediate. On older Aug-21 images, set PATCH and bind-mount as below (harmless no-op if the file matches the folded copy).
bash
# Prefer HF cache mount + repo-id (avoids snapshot symlink dangling links):# -v $HOME/.cache/huggingface:/root/.cache/huggingface:ro# serve AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED --revision <rev>MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>PATCH=/path/to/modelopt-54367.py # optional on :latest / 2026-09-07-omni-mm; required on Aug-21IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest # or :2026-09-07-omni-mmdocker run -d --name aeon-mixed-5090 --gpus all --ipc=host --shm-size=8g -p 8000:8000 \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-v "$MODEL:/model:ro" \-v "$PATCH:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--served-model-name aeon --host 0.0.0.0 --port 8000 \--max-model-len 131072 --max-num-seqs 4 --max-num-batched-tokens 8192 \--gpu-memory-utilization 0.92 --kv-cache-dtype fp8_e4m3 \--mamba-ssm-cache-dtype bfloat16 --attention-backend TRITON_ATTN \--enable-chunked-prefill --no-enable-prefix-caching \--limit-mm-per-prompt '{"image":2,"video":1}' \--speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}' \--trust-remote-code
If KV will not allocate at 131k, try --gpu-memory-utilization 0.95 then drop --max-num-seqs to 2. Do not add a fat DFlash drafter on 32 GB for the long seat.
aeon-vllm-ultimate-rtx:latestSame image family as the 5090. More VRAM → raise concurrency and context headroom. Pin TRITON_ATTN for long context. If you use MTP, put "attention_backend":"TRITON_ATTN" inside --speculative-config (top-level flag does not always propagate to the drafter).
Same rule as the 5090: :latest / 2026-09-07-omni-mm folds #54367 + omni/MM; bind-mount modelopt-54367.py on Aug-21. Prefer HF cache + repo-id over snapshot-only mounts.
bash
MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>PATCH=/path/to/modelopt-54367.py # optional on :latest / 2026-09-07-omni-mm; required on Aug-21IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest # or :2026-09-07-omni-mmdocker run -d --name aeon-mixed-pro6000 --gpus all --ipc=host --shm-size=16g -p 8000:8000 \-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \-v "$MODEL:/model:ro" \-v "$PATCH:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py:ro" \--entrypoint vllm "$IMAGE" serve /model \--served-model-name aeon --host 0.0.0.0 --port 8000 \--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 \--gpu-memory-utilization 0.80 --kv-cache-dtype fp8_e4m3 \--mamba-ssm-cache-dtype bfloat16 --attention-backend TRITON_ATTN \--enable-chunked-prefill --no-enable-prefix-caching \--limit-mm-per-prompt '{"image":4,"video":2}' \--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}' \--trust-remote-code
Native MTP (preferred on this seat):
bash
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'
For Qwen long think, prefer a per-request thinking_token_budget over blindly raising max_tokens — otherwise the model can spend the whole budget inside <think>.
markdown
AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16↓NVIDIA ModelOpt 0.46 · MSE + fp8_scale_sweep · 1024×2048↓Mix B (last-8 down_proj FP8 only) → Coding 0.694↓leftover-KL on coding unit-test misses↓this sibling: last-8 FULL MLP FP8↓AEON Bench Coding 0.833
I figure if you own the hardware and the weights, you should own exactly how they are used. You still own the ramifications. This model will write what the base model would refuse. Intended for research, red-team, creative work without a hall monitor — and a threat if the prompt is. A disclaimer plus the payload is not a refusal. Over-abliteration into gibberish defeats the reason you wanted the unlock.
By accessing, downloading, using, running inference on, fine-tuning, merging, quantizing, distributing, integrating, or otherwise interacting with this model, you acknowledge and agree to the following:
This model is a tool with no opinions of its own. You supply the judgement. The outputs carry your fingerprints.
Apache 2.0 (inherited from Qwen/Qwen3.8-27B).
vllm-omni2026-09-07-omni-mm2026-09-07-modelopt:latestmodelopt.pyin_proj_qkvzFP8_PER_CHANNEL_PER_TOKEN/usr/local/lib/python3.12/site-packages/.../modelopt.py/usr/local/lib/python3.12/dist-packages/.../modelopt.py| TRITON_ATTN | Always --attention-backend TRITON_ATTN on Spark and RTX long seats. |
| No prefix cache | Always --no-enable-prefix-caching. |
| Spark speculative | Everyday chat: external DFlash2 n=7 (z-lab/Qwen3.8-27B-DFlash2). Perf / peak: Dynamic DFlash2 via num_speculative_tokens_per_batch_size. |
| RTX speculative | In-checkpoint MTP n=3. Do not hang a fat DFlash drafter on 32 GB. |
| Never MTP + DFlash | Pick one speculative path. Never both. |
| V2 runner | Everyday Spark chat recipes set VLLM_USE_V2_MODEL_RUNNER=0. The Perf / peak lane is the exception: MRv2 + FULL_AND_PIECEWISE. |
| 16 |
| 0.85 |
| Dynamic DFlash2 |
| off |
| TP=2 quality | 1000000 | 16 | 0.70 | DFlash2 n=7 | on (official factor-4 hf-overrides) |
| TP=2 speed | 65536 | 16 | 0.60 | DFlash2 n=7 | off |
| RTX 5090 | 131072 | 4 | 0.92–0.95 | MTP n=3 | off |
| RTX PRO 6000 | 262144 | 8 | 0.80 | MTP n=3 (optional) | off |