GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Capicua25x
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Capicua25x
Model Tree
deepreinforce-ai/Ornith-1.0-35B
Input Modalities
Output Modalities
Supported Functionality
2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM 0.19.1 (image below).
qwen3_xml), and reasoning split (qwen3) all confirmed working.| concurrent | short prompt | ~6k prompt |
|---|---|---|
| 1 | 130.9 / 131 | 100.9 / 101 |
| 16 | 58.9 / 889 | 34.2 / 523 |
| 32 | 40.2 / 1219 | 20.3 / 617 |
| 64 | 29.9 / 1822 | 12.3 / 723 |
| 96 | 23.4 / 1584 | 9.2 / 717 |
| 128 | 21.4 / 1779 | 8.2 / 761 |
Usable concurrency ceiling (per-user ≥ 20 tok/s): ~128 at short context, ~32 at 6k context. Single-stream decode is 130.9 tok/s short / 100.9 tok/s at 6k — the A3B sparsity (~3 B active) makes it faster single-stream than a dense 9B at short context, and comparable at long context, while carrying far more total knowledge.
Uses the prebuilt RDNA4 vLLM image
capicua25x/vllm-rocm-rdna4 (tag 0.19.1):
bash
docker run --rm --network=host \--device=/dev/kfd --device=/dev/dri \--group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \capicua25x/vllm-rocm-rdna4:0.19.1 \--model Capicua25x/Ornith-1.0-35B-MXFP4-Vision-MTP \--served-model-name ornith --trust-remote-code \--tensor-parallel-size 2 \--gpu-memory-utilization 0.92 \--max-model-len 262144 \--attention-backend TRITON_ATTN \--enable-prefix-caching --max-num-seqs 64 \--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--attention-backend TRITON_ATTN — required on gfx1201; the 0.19.1 default (ROCM_ATTN) collapses
spec-decode throughput at concurrency.--speculative-config '{"method":"mtp","num_speculative_tokens":3}' — enables the grafted MTP head.
n=3 maximizes throughput; n=1–2 maximize per-token acceptance. Tune per workload.--tool-call-parser qwen3_xml --reasoning-parser qwen3 — Qwen-family tool-calling + reasoning split.--trust-remote-code — the Qwen MoE vision architecture.--gpu-memory-utilization 0.92,
--max-num-seqs 64). The 35 B does not fit a single 32 GB GPU at this context; use TP2.compressed-tensors (4-bit float, group 32, symmetric;
lm_head / embed_tokens / norms / vision tower left BF16).mtp.* MoE head (785 BF16 tensors — including the per-expert
mtp.layers.0.mlp.experts.N.* projections) from the donor checkpoint into a new
model-mtp.safetensors shard and patch model.safetensors.index.json.mtp.* Linear module (the fc, the attention projections,
and all per-expert MLP projections) to quantization_config.ignore, so vLLM's compressed-tensors
loader keeps the BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one
mixed-precision gotcha; for a MoE head it means hundreds of ignore entries (derived
automatically from the head's rank-2 tensors).Step 1 (the MXFP4 quantize) is scripted in quantize_mxfp4.py (weight-only MXFP4A16, group 32, data-free — deterministic/byte-reproducible). Steps 2–3 are scripted in recipe_graft_mxfp4.py — it copies every
mtp.* tensor from the donor and derives the ignore list from the rank-2 (matmul) weights, so the same
script handles both a dense head and this MoE head.
Ornith-1.0-35B, the base/trunk model (MIT).Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill, which trained the MoE MTP head grafted here.tcclaviger) — the RDNA4 vLLM base image that made gfx1201 serving possible. The image this model runs on is a forward-port of his work — without it, none of this runs.Apache-2.0. This combines the Ornith-1.0-35B trunk (MIT, DeepReinforce) with a grafted MTP head (Apache-2.0). MIT is one-way compatible with Apache-2.0, so the combined derivative is distributed under Apache-2.0; both upstream notices are preserved above in Credits.
capicua25x/vllm-rocm-rdna4