GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
abhaymin
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
abhaymin
Model Tree
0xKitkat/Ornith-1.5-35B-A3B-Uncensored
Input Modalities
Output Modalities
Supported Functionality
Stated per Apache-2.0 §4(b):
r=16, alpha=16, use_rslora=true
(scaling alpha/sqrt(r) = 4.0). Targets attention and shared-expert projections plus, via PEFT
target_parameters, the fused MoE expert tensors mlp.experts.gate_up_proj and
mlp.experts.down_proj. Training focus: coding, market/trading analysis, content classification,
and tool/function calling.config.json
(rope_type: "yarn", factor: 4.0, original_max_position_embeddings: 262144).The vision tower was not fine-tuned — every adapter tensor lives under language_model.*, and
model.visual.* is bit-identical to the base after merging. Vision/video ability is inherited.
Tested on 2 x RTX 5060 Ti (16 GB each, 31.7 GB total), CUDA 13.1, 59 GB RAM, vLLM 0.27.1. Loads and generates correctly.
bash
vllm serve <this-repo> \--tensor-parallel-size 2 \--max-model-len 4096 \--gpu-memory-utilization 0.92 \--cpu-offload-gb 22 \--enforce-eager
--cpu-offload-gb is required on 32 GB of VRAM: the weights are ~67 GB, so roughly 44 GB spills
to system RAM. Expect about 1.8 tok/s in that configuration — it is offload-bound, not
compute-bound. Drop --cpu-offload-gb entirely once you have ~80 GB of VRAM (e.g. 2xA100 80GB,
4xL40S), and raise --max-model-len toward 1048576 as KV memory allows.
With enough VRAM, vLLM can also use the multi-token-prediction head for speculative decoding —
it registers Qwen3_5MoeMTP, a head that llama.cpp discards:
bash
vllm serve <this-repo> --tensor-parallel-size 4 --max-model-len 262144 \--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
The architecture is supported upstream (Qwen3_5MoeForConditionalGeneration); upstream Ornith
recommends SGLang 0.5.9+. Not tested on this hardware — 32 GB of VRAM is below what a
67 GB BF16 model needs, and SGLang has no CPU-offload equivalent to vLLM's --cpu-offload-gb.
Treat this as a starting point on a larger machine, not a verified command:
bash
python -m sglang.launch_server \--model-path <this-repo> \--tp 2 \--context-length 262144 \--mem-fraction-static 0.90 \--port 30000
python
from transformers import AutoModelForImageTextToText, AutoProcessorimport torchmodel_id = "<this-repo>"processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto",)
Requires Transformers 5.8.1+ (Qwen3_5MoeForConditionalGeneration does not exist in 4.x).
config.json carries max_position_embeddings: 1048576 with:
json
"rope_parameters": { "rope_type": "yarn", "factor": 4.0,"original_max_position_embeddings": 262144, ... }
vLLM 0.27.1 reads the transformers-5 rope_parameters key (not the legacy rope_scaling), so the
scaling applies automatically. --max-model-len still bounds what you actually allocate.
It is extrapolation, not training. The base is natively 262,144 with rope_type: default.
YaRN interpolates RoPE frequencies at inference; nothing here was trained beyond 262,144, and no
long-context evaluation was run. Treat 1M as an upper bound. For results you can trust, cap at the
native length:
bash
vllm serve <this-repo> --max-model-len 262144 ...
Memory. Qwen3.5-MoE is hybrid — only 10 of 40 layers use full attention, the rest are
linear/SSM with constant-size state — so KV is about 20 KiB/token at f16, roughly a quarter of
a comparable dense model. At 1,048,576 that is still ~21 GiB of KV on top of ~67 GiB of BF16
weights, so full 1M in BF16 wants ~90 GiB of VRAM. On the 31.7 GB test rig this is not reachable;
--max-model-len 4096 was used for verification.
If you are training on these weights, set the context back to native first. Training under YaRN while your sequences are a few thousand tokens applies the scaling to short positions, which is not how the base was trained:
json
"max_position_embeddings": 262144,"rope_parameters": { "rope_type": "default", ... } // drop factor + original_max_position_embeddings
Produced with soup-cli — the LoRA merge, the YaRN context config, and the GGUF sibling build all ran through it. The rope block was generated by soup's own helper rather than hand-written:
python
from soup_cli.utils.long_context import get_rope_scaling_configget_rope_scaling_config("yarn", 1_048_576, 262_144)# {'type': 'yarn', 'factor': 4.0, 'original_max_position_embeddings': 262144}
soup can serve and batch-run these weights directly, wrapping vLLM or SGLang:
bash
soup serve -m <this-repo> --backend vllm --tp 2 --max-model-len 262144soup serve -m <this-repo> --backend sglang --tp 2 --max-model-len 262144soup infer -m <this-repo> -i prompts.jsonl -o out.jsonlsoup chat -m <this-repo>
Note soup's backends are transformers, vllm, sglang and mii — there is no llama.cpp serving
backend, so use llama-server directly for the GGUF sibling.
max_position_embeddings back to 262144 and rope_type to default first —
training under YaRN on short sequences is not how the base was trained.<|audio_start|>-style tokens in the
tokenizer are vestigial shared-vocabulary entries. Use an external ASR front-end.Upstream recommended sampling: temperature=0.6, top_p=0.95, top_k=20. This is a reasoning
model and emits <think>...</think> before its answer.
This is a derivative several steps down a chain. Credit belongs to, in order:
Licensed under Apache-2.0, inherited from the base model. Upstream components carry their own licenses as listed above.