GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
JRamirez-UAB
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
JRamirez-UAB
Model Tree
ornith-ai/Ornith-1.5-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
| Revision | bpw | Weights | KLD internal · public | GPU | Context | Decode with MTP, 1 / 2 sessions (tok/s) | Prefill 62k → 237k (tok/s) | Needle | Vision | Tools A · B · C (of 30) | Reasoning xhigh (48 problems) | Really free at peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
main | 4.00 | 19.0 GB | 0.0163 · 0.0168 | RTX 3090 | 262,144 | 113.8 / 195.7 | 2,441 → 1,255 | 8/8 at 237k | 15/15 | 30 · 30 · 30 | 24,289 (47/48) | 1,063 MiB |
4.50bpw | 4.50 | 21.1 GB | 0.0126 · 0.0128 | L40S 48 GB | 262,144 | 119.2 / 233.2 | 3,524 → 2,649 | 8/8 at 237k | 15/15 | 30 · 30 · 29 | 25,651 (45/48) | 13,988 MiB (KV 10e9) |
5.00bpw | 5.00 | 23.2 GB | 0.0095 · 0.0095 | L40S 48 GB | 262,144 | 118.9 / 220.9 | 3,457 → 2,657 | 8/8 at 237k | 14/15 | 30 · 30 · 30 | 26,417 (48/48) | 12,006 MiB (KV 10e9) |
How to read it:
qbench on an internal sysadmin/agent evaluation
set (not published) and on turboderp's public trace (qbench_prompts_gen.json). Lower is better.xhigh at the model's default temperature, B = xhigh at T=0.3,
C = thinking off at T=0.3. Reasoning: total reasoning tokens over 6 checkable problems × 8 tries (T=0.6), and how many were correct.opencode run, 4 sysadmin tasks × 2 with real tool execution): 6/8 (4.00).--max-model-len 262144, fp8 KV cache, MTP 1, 2 sequences, prefix caching, 1 image per request (≤1 MP). RTX 3090 rows: KV 3.5e9 B (287,575-token pool).File names: like other Ornith models it sometimes translates file names written in Spanish in the request (e.g. writes errors.txt when asked for errores.txt); both failed agent tasks here were that, with the right content. On Ornith-1.5-9B, writing the names between backticks fixed it (16/16). The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70). The 4.50/5.00 branches do not fit a 24 GB GPU with 262k + vision + MTP (see the branch notes in Quick start for what does fit on an RTX 3090). On a 24 GB card, measured on the 4.50 branch: the vision tower costs ≈45k tokens of context and the MTP head ≈112k (its weights are only 0.4 GB, but it adds its own KV cache, ≈16 % more per token, and ≈0.9 GB of buffers).
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers and
INT8 embed_tokens / packed lm_head for qwen3_5 models.qwen38_exl3_vision 0.2.0 (ours, in plugin/qwen38_exl3_vision/): EXL3 MoE experts (moe.py: the per-expert trellis tensors are
stacked and run with exllamav3's fused MoE kernel, CUDA-graph friendly), the EXL3 vision tower, padding to multiples of 128 and dense fp16 layers converted once to the
model dtype.Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:
bash
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Ornith-1.5-35B-A3B-EXL3-4.00bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionWe checked these exact steps on a fresh machine (A40): plugin 0.2.0 loads our dense Swift-1.5 27B EXL3 checkpoint (downloaded from Hugging Face) and Qwen3.6-35B-A3B 4.00 (MoE), with the same results as our own build.
bash
export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Ornith-1.5-35B-A3B-EXL3-4.00bpw-24GB \ --served-model-name ornith-1.5-35b-a3b \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 262144 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 3500000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \ --compilation-config '{"max_cudagraph_capture_size":16}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder4.50bpw branch (21.1 GB of weights): does not fit a 24 GB GPU with 262k + vision + MTP. Validated at 262,144 on an L40S 48 GB with --revision 4.50bpw and --kv-cache-memory-bytes 10000000000 (other flags the same). On a 24 GB RTX 3090 it fits with 171,008 tokens with vision + MTP (--kv-cache-memory-bytes 2173912576 --max-model-len 171008; needle 8/8 at 154k, vision 15/15, 323 MiB really free at peak), 215,040 text-only with MTP (--language-model-only, KV 2685617664) or the full 262,144 with vision and no MTP (KV 3063329408).5.00bpw branch (23.2 GB of weights): does not fit a 24 GB GPU with 262k + vision + MTP. Validated at 262,144 on an L40S 48 GB with --revision 5.00bpw and --kv-cache-memory-bytes 10000000000 (other flags the same). On a 24 GB RTX 3090 it does not fit with vision + MTP; text-only (--language-model-only) it fits with 26,624 tokens with MTP (KV 565396992) or 124,928 without MTP (KV 1456910976, no --speculative-config); needle 8/8 at 90 % of each.On 48 GB GPUs raise --kv-cache-memory-bytes (10e9 B gives an 821k-token pool: room for 8 sessions; add --max-num-seqs 8).
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required. The fused exllamav3 MoE kernel synchronises its blocks through global counters and needs the whole GPU;
vLLM 0.30 otherwise runs the shared expert on a parallel CUDA stream and decoding hangs after a few seconds. The plugin sets it, but export it yourself too.max_cudagraph_capture_size 16): the fused kernel handles up to 128-256 rows per expert; larger batches (prefill) take a
slower path that reconstructs the expert weights.nvidia-smi --query-gpu=memory.free. Size the KV cache
with --kv-cache-memory-bytes, not --gpu-memory-utilization. The KV cache of these models is small (≈13 KB/token in fp8: only 10 of the 40 layers are
full attention, with 2 KV heads), which is why 262k fits next to the 19 GB of weights.quantization_config.bits not being an integer is expected (EXL3 is an average).sc_measure --shaped, --max-sys 24) on a 196 × 2048-token calibration trace,
bit allocation with sc_optimize, conversion with the same trace. The trace was generated by Qwen3.8-27B (same tokenizer) on prompts from exllamav3's SC set
plus internal Linux sysadmin/agent conversations (not published); with a trace generated by the model itself the KLD was not better (Ornith 4.00: 0.0171 vs 0.0163).lm_head 6 bits, vision tower 6 bits, MTP head 4 bits, codebook mul1, out_scales: always.scripts/embed_int8.py from the converted checkpoint (≈−0.5 GB of VRAM, no measurable change).ornith-ai/Ornith-1.5-35B-A3B revision 10fbf86f. Chat template and tokenizer are the base model's.Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
qbench evaluation, public trace from turboderp's repos);
OrcaSAQ2-kernel (EXL3 in vLLM); vLLM.bits--tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts xhigh, medium and low; they are prompt instructions, not limits.