GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
AEON-7
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
AEON-7
Model Tree
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16
Input Modalities
Output Modalities
Supported Functionality
On DGX Spark, run this body with a DFlash drafter (not native MTP): it lands at parity speed with the smaller -MTP-XS sibling while scoring higher on quality-eval benchmarks — the recommended daily-driver body when you have the VRAM. (Native qwen3_5_mtp decoding stays a dedicated-VRAM-Blackwell path — see the routing table below.)
bash
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest # body (this repo) + z-lab DFlash drafterGIT_LFS_SKIP_SMUDGE=1 git clone \ https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP /models/mm-mtp( cd /models/mm-mtp && git lfs pull )GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/z-lab/Qwen3.6-27B-DFlash /models/dflash( cd /models/dflash && git lfs pull ) # ENTRYPOINT is /bin/bash — pass --entrypoint vllmdocker run -d --name aeon-vllm --gpus all --ipc=host --shm-size=16g --net=host \ -e VLLM_USE_FLASHINFER_SAMPLER=1 \ -v /models/mm-mtp:/model:ro -v /models/dflash:/drafter:ro \ --entrypoint vllm ghcr.io/aeon-7/aeon-vllm-ultimate:latest \ serve /model --served-model-name aeon \ --quantization modelopt --kv-cache-dtype fp8_e4m3 \ --attention-backend TRITON_ATTN \ --max-model-len 229376 --max-num-seqs 16 --max-num-batched-tokens 32768 \ --gpu-memory-utilization 0.60 \ --enable-chunked-prefill --enable-prefix-caching \ --generation-config vllm \ --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --mm-encoder-tp-mode data \ --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":12,"attention_backend":"TRITON_ATTN"}' \ --trust-remote-codeOn GB10 unified memory keep
--gpu-memory-utilizationat0.6–0.7; above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box (even0.85stalls). Use0.60when ASR/TTS/embedding sidecars share the Spark, or with high concurrency, fp16 KV, or DFlash. The recommended Spark sidecar profile uses--max-model-len 229376,--max-num-seqs 16, and--max-num-batched-tokens 32768: one near-full-context session can run, while smaller agent sessions still share the pooled FP8 KV budget dynamically.vLLM 0.24.0 DFlash note: set the attention backend in both places.
--attention-backend TRITON_ATTNselects the target-model backend, but vLLM does not inherit that into the speculative drafter; the DFlash JSON must also include"attention_backend":"TRITON_ATTN". Leave--mamba-block-sizeunset and let vLLM derive the page/block geometry for the hybrid GDN stack. Full recipe matrix (NVFP4-KV capacity, TurboQuant, dedicated-VRAM MTP): container README.
| Format | Size | Use case |
|---|---|---|
| BF16 | 51 GB | Full-precision reference weights (A100/H100 80 GB, RTX PRO 6000 96 GB, multi-GPU, fine-tuning) |
| NVFP4 (compressed-tensors + DFlash) | 26 GB | DGX Spark / GB10 — production validated with DFlash speculative decoding. Patched vllm-aeon-ultimate-dflash container. |
| Multimodal-NVFP4-MTP (this repo) | 27 GB |
This is the modelopt-format NVFP4 variant with MTP speculative decoding, multimodal-preserved, of AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16 — the lossless abliteration of Qwen 3.6 27B (KL 0.000492 vs base, 0/100 refusals, multimodal preserved, hybrid GDN-aware quantization).
Specifically:
nvidia-modelopt 0.43.0 with NVFP4_DEFAULT_CFG. This is the modelopt compressed-tensors format that vLLM serves through --quantization modelopt (different code path from the -NVFP4 sibling release which uses --quantization compressed-tensors).*linear_attn.conv1d* ignore plus our explicit *linear_attn* exclude keeps these intact.Qwen/Qwen3.6-27B checkpoint (15 tensors, BF16). The base contains MTP heads but Qwen3_5ForConditionalGeneration.from_pretrained drops them during loading; the lna-lab pipeline pattern (which this build follows) explicitly grafts them back into the quantized output, giving vLLM a working drafter for .Multi-Token Prediction (MTP) lets the model predict multiple future tokens per forward pass via the trained mtp.* head, enabling speculative decoding without a separate drafter model. The acceptance rate is high because the drafter is the model itself — same architecture, same weights, same distribution.
| Hardware | Median tok/s | Peak tok/s | Spec-decode acceptance |
|---|---|---|---|
| RTX PRO 6000 Blackwell (96 GB dedicated VRAM) | ~92 (this variant) / 111.4 (XS sibling) | 124.7 (XS sibling) | 67.7 % regular / 69.2 % XS |
| DGX Spark / GB10 (unified memory) — MTP method | 24.1 (XS sibling) | 27.5 | 66.3 % |
| DGX Spark / GB10 — DFlash method on this body 🏆 | 38.5 tok/s thinking-on / 38.1 thinking-off | 71.3 tok/s thinking-on / 68.4 off | DFlash v2 |
| RTX 5090, B100 / B200 |
n=3: ~132 tok/sOn RTX PRO 6000 the XS sibling beats DFlash territory (~111 tok/s vs DFlash-class ~85 we'd expect there). On DGX Spark, DFlash beats MTP by 26 % median / 52 % peak — the unified-memory bandwidth caps how much MTP's high acceptance can translate to throughput. So: MTP is a dedicated-VRAM-Blackwell variant, not a universal upgrade. Full bench data: GitHub repo Performance section.
The right speculative-decode method depends on memory architecture:
| Hardware tier | Recommended variant | Why |
|---|---|---|
| DGX Spark / GB10 (sm_121a, unified memory) | This body — with a DFlash drafter ✅ (recommended daily-driver) | Run this body + z-lab DFlash drafter (see Quickstart above): parity speed with the XS sibling, higher quality-eval scores. Use DFlash, not native qwen3_5_mtp, on Spark — DFlash beats the MTP method by +26 % median / +52 % peak here (unified-memory bandwidth doesn't reward MTP's high acceptance). |
| RTX PRO 6000 Blackwell (sm_120, 96 GB dedicated VRAM) | This variant (Multimodal-NVFP4-MTP) ✅ if you need vision; Text if text-only | MTP wins on dedicated VRAM. ~92 tok/s median measured with GDN BF16; dedicated-VRAM bandwidth lets the MTP head's high acceptance rate translate to throughput. |
| RTX 5090 (sm_120, 32 GB dedicated VRAM) | if you use vision; if text-only |
Full bench numbers: GitHub repo Performance section.
bash
# One-time: pull this repo locallyhf download AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP \ --local-dir ./aeon-ultimate-multimodal-nvfp4-mtp # Serveexport VLLM_USE_FLASHINFER_SAMPLER=1 # v0.24.0 removed VLLM_NVFP4_GEMM_BACKEND / VLLM_USE_FLASHINFER_MOE_* — use the KernelConfig flagsvllm serve ./aeon-ultimate-multimodal-nvfp4-mtp \ --quantization modelopt \ --linear-backend flashinfer_cutlass --moe-backend cutlass \ --mamba-cache-dtype float32 \ --trust-remote-code \ --max-model-len 262144 \ --max-num-seqs 32 \ --max-num-batched-tokens 32768 \ --gpu-memory-utilization 0.94 \ --enable-chunked-prefill \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'num_speculative_tokens=3 is the canonical setting for qwen3_5_mtp. Higher values diverge the drafter further from the target distribution and acceptance falls.
--quantization modelopt is required (not compressed-tensors — different format).--speculative-config '{"method":"qwen3_5_mtp", ...}' activates the grafted MTP head as the spec-decode drafter. No external drafter download needed — the head is in the safetensors of this repo.--gpu-memory-utilization 0.94 is the validated cap on RTX PRO 6000; 0.95 causes the FlashInfer NVFP4 GEMM autotuner to OOM on first boot. See the GitHub repo's RTX PRO 6000 page for the same OOM behavior under DFlash.nvidia-modelopt 0.43.0 with NVFP4_DEFAULT_CFGQwen3_5ForConditionalGeneration.from_pretrained (multimodal-preserved class)neuralmagic/calibration LLM split, 20 samples × 8192 tokenslm_head, proj_out.*, *router*, *mlp.gate.* (NVFP4_DEFAULT_CFG)*linear_attn.conv1d*, *mixer.conv1d* (NVFP4_DEFAULT_CFG)AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16. See that card for the full abliteration pipeline.docs/MTP_GRAFT_RECIPE.md)sakamakismile/Qwen3.6-27B-Text-NVFP4-MTPnvidia-modelopt 0.43.0)Qwen/Qwen3.6-27BApache 2.0, inherited from Qwen/Qwen3.6-27B. This is an uncensored model. Read the full User Responsibility & Arbitration Clause on the BF16 source card before deploying. Summary: you implement downstream safety layers (input validation, output filtering, content moderation, audit logging, rate limiting, access controls, human-in-the-loop for high-risk workflows). The model has no opinions of its own — you supply the opinions, the judgment, and the ethics.
AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:
Become a member on Patreon → patreon.com/cw/AeonForge7/membership
Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.
If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.
Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.
High-bandwidth dedicated GPUs (RTX 5090, RTX PRO 6000, B100/B200) with MTP speculative decoding via the model's native mtp.* head. modelopt format, --quantization modelopt. Vision tower preserved. |
| Text-NVFP4-MTP | 20 GB | Same as this repo but with vision tower stripped. Smaller footprint for text-only deployments on tighter VRAM. |
--speculative-config '{"method":"qwen3_5_mtp",...}'| not yet measured by us — community welcome |
| XS variants fit comfortably in 32 GB. 111.4 tok/s median measured on RTX PRO 6000; RTX 5090 should land near or above that. |
| A100 / H100 (no native FP4) | BF16 | NVFP4 dequantizes to BF16 on Ampere/Hopper — no benefit. |
| B100 / B200 (sm_100, dedicated FP4) | This variant (Multimodal) or Text variant | Native FP4 + dedicated VRAM = MTP territory. |
*linear_attn* (added — full GDN preservation)*visual* (added — vision tower preservation)*mtp* (added — MTP head preservation)*output_layer*, output.*Qwen/Qwen3.6-27B after modelopt export (AutoModelForCausalLM.from_pretrained drops them; explicit graft restores)