GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
maci0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
maci0
Model Tree
deepreinforce-ai/Ornith-1.0-35B
Input Modalities
Output Modalities
Supported Functionality
Near-lossless versus the bf16 source, 20 GB vs 70 GB bf16 (~29%), at wikitext-2 perplexity 7.33. GPTQ error compensation and an MSE observer keep the drop from bf16 minimal; the header lists the full characteristics and Quantization covers the recipe.
NVFP4 is auto-detected from config.json (compressed-tensors); no quantization flag
needed.
bash
vllm serve maci0/Ornith-1.0-35B-abliterated-NVFP4 \--served-model-name ornith-35b-abliterated-nvfp4 \--max-model-len 131072 \--gpu-memory-utilization 0.90 \--kv-cache-dtype fp8 \--reasoning-parser qwen3 \--enable-auto-tool-choice --tool-call-parser qwen3_coder
--language-model-only to skip the vision tower and free KV cache for text use.python
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="x")r = client.chat.completions.create(model="ornith-35b-abliterated-nvfp4",messages=[{"role": "user", "content": "Add a retry with exponential backoff to this HTTP client and explain the change."}],)print(r.choices[0].message.content)
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "ornith-35b-abliterated-nvfp4","messages": [{"role": "user", "content": "Add a retry with exponential backoff to this HTTP client and explain the change."}]}'
Ornith-1.0 is a self-improving family of open agentic-coding models from Deep Reinforce. The 35B member is a Mixture-of-Experts Qwen3.5-MoE-family vision-language model with thinking-mode reasoning and a 256K context.
max_position_embeddings 262144).This is the notable part of this build. Ornith-1.0-35B is a fused-expert MoE: all 256
experts of a layer are packed into a single fused tensor (Qwen3_5MoeExperts.down_proj)
rather than 256 iterable Linear modules. Standard abliteration tooling (including
Heretic) walks the module tree looking for Linear
layers, so it silently skips every expert and can only touch the attention projections.
On this model that leaves a hard refusal floor (roughly 88/100 harmful prompts still
refused), because the refusal behaviour lives in the experts.
To reach the experts, this model was abliterated with a custom MoE-aware method: for each
layer, compute the refusal direction, then orthogonalize the fused experts.down_proj
output per expert against that direction. That is 10240 expert slices (256 experts x 40
layers), applied in two rounds (a second pass removes the residual refusal signal that
remains after the first). Refusal directions are estimated from
mlabonne/harmless_alpaca (good) vs mlabonne/harmful_behaviors (bad).
| Stage | Result |
|---|---|
| Standard tools (Heretic) | attention only, experts skipped, refusal floor ~88/100 |
| MoE-aware round 1 | per-expert orthogonalization of the fused experts.down_proj |
| MoE-aware round 2 | residual refusal direction removed |
| Held-out probe | 0/8 harmful prompts it previously refused |
Honest validation caveat. The result was checked on a small held-out probe: the model no longer refuses 8 harmful prompts it previously refused (0/8). This is a sanity check, not a rigorous refusals-at-fixed-KL benchmark like the smaller dense Ornith build. Treat the decensoring as demonstrated on a small probe, not exhaustively measured.
| Scheme | NVFP4, W4A4 |
| Weight rounding | GPTQ (Hessian-based error compensation), MSE observer |
| Weights | FP4 (E2M1), group_size=16, tensor_group, FP8 (E4M3) group scales, shared across fused layers |
| Activations | FP4, dynamic per-group, FP8 (E4M3) scales |
| Quantized | all language-model Linear layers, including the fused MoE expert tensors |
| MoE calibration | moe_calibrate_all_experts=True (every one of the 256 experts must be routed during calibration, or unrouted experts produce garbage) |
GPTQ is a quantization-time cost only; inference speed and format are identical to plain round-to-nearest NVFP4, but it chooses better 4-bit values.
Calibration: 512 domain-matched samples (long reasoning + general chat + code),
max_seq_len=2048, text-only path through the VL model, with
moe_calibrate_all_experts=True so all 256 experts per layer receive calibration traffic.
Toolchain: llmcompressor==0.12.0, compressed-tensors==0.17.1, transformers==5.12.1,
torch==2.11.0+cu130, on an NVIDIA GB10 (Blackwell, sm_121). The routers (mlp.gate,
shared_expert_gate) are left in bf16 so routing stays exact.
Thinking mode is the default.
temperature=0.6, top_p=0.95, top_k=20temperature=1.0, top_p=0.95, top_k=20temperature=0.7, top_p=0.80, top_k=20{%- set enable_thinking = false %} in the chat template, or
pass extra_body={"chat_template_kwargs": {"enable_thinking": false}}.--reasoning-parser and --tool-call-parser are not auto-detected; pass them explicitly.chat_template_kwargs.Apache-2.0, following the base model. Intended use and all responsibility for use follow the base model.
experts.down_proj), because standard Linear-walking tools cannot reach fused MoE experts| Kept in bf16 | routers (mlp.gate, shared_expert_gate), vision tower (model.visual.*), lm_head |
| Untouched | gated delta-net Conv1d and SSM params (A_log, dt_bias), never Linear |